With all of these discussions of scalability on the G+ dominated servers, would it make sense for us to try a scaled infrastructure somewhere to confirm that it works or is there a pod doing this already? I’m thinking specifically something like:
Load balancer in front of multiple Diaspora servers
Static content served by an S3 (or comparable) object storage system
DB layer with multiple nodes rather than a single node
Has anyone tried this or run something like that in production? Is that an overly ambitious thing to try to setup? As I work on the API Spec I was thinking that could be something where we could build that out and then use test servers to fill it up with a substantial amount of activity.
I’ve talked to the Pluspora podmins about this last week, as they were finding the limits of their server start to be tested. I suggested various ways that they could alleviate this, and help people moving from G+ to understand better how diaspora* works. They have, I think, posted in G+ about decentralisation and the pods, and some instructions to people about how to choose a pod to sign up to, so that not all G+ exiles are on a single pod. If they find that their server can’t take much more load, they can close sign-ups temporarily. Since then I’ve seen far fewer sign-ups to Pluspora and a big spike in sign-ups to pods across the network, so I guess that this is what they’ve done.
That is the primary answer to scalability issues for pods; explaining to prospective users that there is no need, nor any benefit, to being on the same pod as people with whom you might be likely to interact, and helping them to make a choice about which pod to make their home. This is something that the d* founders failed to do in the project’s early days, which led to joindiaspora.com becoming horribly over-loaded and performing very poorly as a result. It’s something that can be avoided by educating people before they sign up to d*. There is still a lot that the project can do to improve how it does this.
Doesn’t this undermine the distributed philosophy of diaspora*?
I don’t know where the breaking points on pods are so I don’t know. Pods with 10K or 100K users doesn’t sound unreasonable to me. When I look at the topology of Mastodon and what the resources are behind their larger pods (mastodon.social the one I know anything about because of a Changelog podcast with the project lead developer) this is how they are addressing that and maintaining performance. We should be able to do the same thing if you look at our technologies but has anyone tried and seen the results? Mastodon.social has about 250K registered users and about 30K active weekly users on that pod, which again doesn’t sound like an inappropriately large/active user base for a pod. There are limits to this sort of scalability anyway so it’s not like this would foster the creation of megapods with millions or tens of millions of users. Proving how to maintain performance under a higher user load on a reasonable but larger pod is the nature of my question.
