It seems like we are throwing an awful lot of feature proposals on the back-burner due to a need to improve the Federation protocol. In contrast, I have seen little to no discussion on actually improving Federation. I started this discussion hoping to change that.
Perhaps we could start by:
Identifying all the problems with Federation that are currently holding us back
Combine these into an easy-to-read document/wiki article so new developers can get a better a sense of what is going on (and probably be more able to contribute! ;)) and more experienced developers can have a single document to reference while brainstorming. (If there’s not somebody on the “Documentation Editors” team that wants to do this, I will happily volunteer.)
Establish an open team of developers who would like to collaborate and solve these complex problems. (Count me in!)
Please chime in with your thoughts. The above course of action is merely a suggestion, so if you have a better idea on how to approach this, please share!
Note: This discussion was imported from Loomio. Click here to view the original discussion.
I think this is a really good starting point. We could set something up on the wiki that specifically dives into the problems we have, and I think we could probably get a much better map of how we want to address things.
Of course, I think that it’s also important to consider existing and developing standards for federation. There is a larger problem than just our own platform here, and it involves every decentralized social network’s ability to interoperate. Everyone has a difference piece to the puzzle,
Ideally, it would be wonderful to have an open standard that all social platforms could communicate through, but OStatus is slow in development and not really a one-size-fits-all solution, Tent is still in its infancy, and Zot2 is still largely only used in the experimental development of Friendica Red. There are also some platforms out there that use XMPP purely for broadcasting federation, and probably several more unused standards that no one’s even heard about yet.
So that leaves us with the following general questions:
How exactly do we want to “fix” federation?
We have a lot of people on pods currently. How can we make sure federation scales out to support a large network of people, both on shared pods and their own lone ones?
What standards can we build on that will provide us proper structure for the features we want? (Example: If we want Groups to work with federation, what’s the easiest, sanest way to implement that?)
What are the most awful problems currently plaguing how our existing system works?
The first step is to separate the federation to the rest of the code. After that, we can try different implementation for federation (our own one we currently have, Tent, XMPP…) make some stress-tests, do what we want, and choose the best solution at the end.
We can probably work with the Tent team when the federation will be separate. But we need to do it first. @florianstaudacher@jonnehass who are knowing enough the base code to do that? Is it possible to organize a meeting? I’m sure that 3/4 persons working together can do the big part of the job in one week-end.
Public posts not visible between pods (go on a profile of someone from your pod, and then by entering hispod.tld/u/hishandle and you’ll see that there is often more posts there)
Delete post / comment are not federated
Images are not federated (but this is maybe fixed by #3940)
Profile is not always up to date (but this is maybe fixed by #3976)
Not a really a bug, but you will not find a post containing a tag by searching this tag if it is not already in your pod. Solving that would be awesome.
For us newbs, could one of the more experienced devs give at least a high level explanation on how federation is currently co-mingled into the app? Was there ever a plan made for putting federation into it’s own layer or did that only get as far as deciding it should be done?
For individual problems that are not really problems due to the big design there is an easy answer = they need to be fixed just like any other bug. I don’t think we need much discussion on how to fix these and they certainly don’t need to wait for federation being put in to a layer.
But larger changes should really be talked about carefully before implementing anything. We must have a plan for federation breaking changes and preferably do those in stages with a clear roadmap.
Personally I would like the small issues to be fixed as far as they can and then start working with separating the federation.
Maybe a public post on DiasporaHQ and the website with simple explanations for users about what federation is, the problems and a small roadmap will be good for people to follow what is going oh here…
I believe indeed that pod communications (ie federation), which is decentralization, and account migration from one pod to another are what make D* special and really needed. This all about the first fundraising at kickstarter and its success. As long as they are not implemented and working, D* goals are not achieved and people will not see the point of such a social network among so many others.
This why communication toward users and newby devs should be more clear about these two points (decentralization and migration).
Public posts not visible between pods (go on a profile of
someone from your pod, and then by entering
hispod.tld/u/hishandle and you’ll see that there is often
more posts there)
This is an interesting one. We should definitely fix this problem. Is there an issue for this already @flaburgan or a test case I can observe?
Delete post / comment are not federated
We’re kind-of discussing that API in https://www.loomio.org/discussions/4093. I think we should see how that discussion and hopefully proposals turn out before making any decisions on deleting content through federation. I do feel like that is a necessary feature. Everybody makes mistakes.
The only major, horrible problem I can really see with federation is that it’s very difficult for people who have no idea how Diaspora works to learn about how we federate content and build upon that. What little documentation exists is mostly concerned with how we securely authenticate content, which does matter but could really be encapsulated in a library (leaving that documentation unnecessary for most hackers).
@tomscott I’m sure there is an issue about that, I’ll take a look.
But about the federation, there is two possible reasons that the post was not federated : a problem in the protocol or a problem in the processing of the queue by Resque (now Sidekiq). Sometimes it failed, and if it’s not retried, the post will never be federated. So I don’t know what caused this problem, we have to make test with small pods to precisely monitor what happens.
I would like to propose that we move forward in improving federation with a first step of separating federation into it’s own layer. This would involve :
Forming a working group of developers/collaborators who would like to volunteer their time to this problem. I think it’s important to have a defined (though very much open) group dedicated to this problem, because simply leaving it open for an individual to pick up and run with leaves me with the feeling that it’s simply going to never get done.
That group would come up with detailed plan for separating out federation and any potential problems that might arise. Communication can be done through IRC Meeting/Email Chain/Loomio/whatever works best for the people involved. The goal here is to produce an organized document that can be easily understood by newcomers who want to jump in and help out as well as a reference for those involved to ensure everybody is on the same page with what exactly needs to be done.
Plan could be verified/approved by other developers/collaborators through Loomio. (Maybe this step isn’t necessary, but I think it’s always a good idea to get outside opinions before implementing a plan.)
Plan is broken down into manageable tasks and put into Github where developers can pick them up and begin making code changes.
Outcome: N/A
Votes:
Yes: 13
Abstain: 0
No: 0
Block: 0
Note: This proposal was imported from Loomio. Vote details, some comments and metadata were not imported. Click here to view the proposal with all details on Loomio.
@seantilleycommunit :: Good points, but not necessarily show stoppers. If we could even get 2-3 devs working on this to start out, I think we could take it a long way. I’m one! =) It would be nice if we had a developer who knows the code well to help out, but in the event that doesn’t happen, I’ll happily be the one to tear through the code until I understand what’s going on. I would love to see some of the newer devs jump in to help figure it out too! I know the task is a daunting one, but it has to be done by somebody if we want our app to succede. This is one of the core pieces of our network and it’s really holding us back on the features we can implement. I really hate telling users, “We can’t do that because the distributed part of our distributed network sucks.” I’d much rather start moving forward on this with limited manpower than continue waiting for a magical influx of dev power.
