Changes to the diaspora protocol

Federation
1

Some changes have recently been made to the diaspora protocol while extracting it outside the main application.

While this is great work and changes do need to be made, in this case changes were made without raising awareness of those changes to all the stakeholders in the protocol.

Please, the protocol is more than diaspora. Even if it is “YOLO code”, coming from one developer, that doesn’t mean it doesn’t have to be supported, maintained and more importantly, not broken for ALL stakeholders.

Change is fine, change without announcement is not fine. I’m guessing this was just an oversight, not realising that asking the other server software stacks to verify and test would be a good idea.

To me, and probably the person writing the actual code, this is clear. But I just had a discussion with some other core developers who don’t think consulting stakeholders is necessary.

So, to raise the point and to take an official decision, we need to vote on the matter.

Note: This discussion was imported from Loomio. Click here to view the original discussion.

2

Proposal: Changes to the protocol must be pre-communicated with ALL stakeholders

With all stakeholders, here we mean:

  • Diaspora core developers
  • Friendica core developers
  • RedMatrix/Hubzilla core developers

Pre-communicated means:

  • Deliver a plan of changes being done to stakeholders when those are planned or latest when work begins on them (cannot work without having a plan)
  • Do not merge in changes before listening to all the stakeholders and noting their concerns with implementation and timetable

Outcome: The diaspora core developers do not want to communicate changes.

Votes:

  • Yes: 5
  • Abstain: 4
  • No: 5
  • Block: 0

Note: This proposal was imported from Loomio. Vote details, some comments and metadata were not imported. Click here to view the proposal with all details on Loomio.

3

As you should be aware of if you want to take place in this discussion, diaspora never had or has a fixed, well defined protocol. Because of that, stuff is sometimes broken and writing actual documentation is impossible. In the past, other networks started implementing our protocol by basically reverse engineering our stuff. While this is fine, we obviously are not able to support this in any way. In fact, our wiki page about the protocols used clearly states

Implementers of this protocol should be advised, though, that Diaspora is in Alpha, and as such, this document is not authoritative, and may lag behind the reference implementation

This is a pretty clear disclaimer. Because of that, no core member of the project ever stated the communication with other networks is a thing we support and that we can/will assure this stuff does not break. In fact, I am not even aware of any of the third party networks devs (except Michael Vogel for some debugging) ever talked to members of the core team to discuss about best practices here.

In our ongoing endeavor to move parts of the federation code inside the new gem, some changes to the protocol have to be made. At the moment, all changes were made to get closer to the originating standards like Salmon to ensure even more compatibility in the future. Even though those changes have been made, we (as in Benjamin, Jonne and myself) ensured that no older diaspora pods communication breaks. The newer 0.5.0.0 releases are much closer to the originating standards but still compatible with very old diaspora releases.

Third party networks have implemented old diaspora federation parts in the past, but they have done a pretty bad job at it. The stuff Friendica sent in never was even close to match the stuff diaspora sends, but it has worked with some hacks on both sides.

This discussion is getting kinda funny here because you are talking about a “diaspora protocol” as if this is something that even exists. diaspora never had a fixed protocol and even the third party networks never did the same we did. So apparently, not even third party implementers were interested in keeping the same standards. Also, at this state, I would like to point out that YOU, Jason, even liked our plans which we are realizing at the moment three years ago. The first discussions about moving the federation inside a separate module were started 2 years ago and even some code was written by Florian. Think about that for a second. I consider three years a pretty good time.

Yes, there were changes made in the new Gem which is actually already included in the diaspora core. You even commented on some of Benjamins pull request and never raised issues and now you act like you never had the chance to participate. I am a bit confused now.

All changes made to the current gem are made in consideration of keeping old diaspora instances alive and at the moment, no one actively involved in those steps has plans on removing said compatibility, because that would be pretty much stupid. In addition, every third party network implementing exactly those old implementations on old pods will still work fine.

If someone added dirty hacks (that were never used by us) to their protocol and now it breaks? Well. Too bad. We keep our old stuff but we are not able to support hacks made by everyone. This would be pretty insane.

Next, the term “stakeholders” which actually drives me insane. Maybe you should read the definition first. We are the development team of diaspora and this is our project. The diaspora users are the stakeholders and not some third party projects developers who are not involved in our development. This is not the W3C and this is not the W3C Social WG. If you want to create a protocol that can be used with all systems, you have to take everyones needs into account and that cannot and will never be done by a single project. You are part of the Social WG and you should know that.

I am a bit angry right now. What you are just trying to do is basically blocking our work on a more stable federation inside the diaspora network and bringing our stuff back to already defined and widely used standards. We have discussed about the importance of the federation in the past and you were on the same side and now you’re basically saying we should not be able to make changes without asking everyone on this planet.

So, just to summarize: We (as in the people who have actively participated in the development of the federation modules in the past) will continue to move the stuff into an own gem while keeping compatibility with old diaspora pods. Some time in the future (and this is going to take quite some time), we will gradually deprecate old federation implementations, thus forcing administrators of old diaspora pods to update. This will be done after a long pre-announcement, but again, we are not even close to think about that.

I would like to add something personal. Right now, we are moving from a pretty crappy implementation to a well-defined and well-developed ecosystem. If we decide to move to just another randomized implementation because this is what the community decides, fine, but then the project has to progress without my support, I fear.

4

“YOLO code”

ROFL.

I agree. Maybe it could be time to write down a formal protocol not tied to diaspora and invite other participants at the table to work with us.

5

As you should be aware of if you want to take place in this discussion, diaspora never had or has a fixed, well defined protocol. Because of that, stuff is sometimes broken and writing actual documentation is impossible.

I’m sorry to say, but that’s the worst excuse I’ve ever read for not writing a documentation. The actual lack of documentation is probably the reason why there has been so few people working on the project for 3 years. This really is a problem and this really is the main recurrent criticism I’ve heard since I’ve been contributing. And I find it pretty accurate.

The fact that de federation protocol has flaws and is not guarantied to work is not to my eyes a good reason for not documenting, not communicated and not trying to be more open towards other projects. Otherwise, I don’t think diaspora can be called an open-source project anymore.

I’d be less strong on the Do not merge in changes before listening to all the stakeholders and noting their concerns with implementation and timetable part than Jason. But stil, that doesn’t mean we can’t communicate. We are not autists, are we!?

6

@jasonrobinson Before I’m able to vote you will have to define “changes” and “diaspora protocol” first.

Since there is no formal definition of the protocol right now, for me the protocol is defined by our own implementation. Do you have a different definition? Would you define the protocol as the “union” of all protocols used by social network implementations that claim to be compatible with diaspora? (friendica, redmatrix, hubzilla, pyaspora, …) Would you define the protocol as the “union” of all protocols that are accepted (but not automatically spoken) by the current federation implementation of diaspora?

To define “changes” let’s take the example you mentioned. Benjamin started to fix some bugs in the federation code (like #2440) but kept the federation code compatible with the old one. Pods using some old d* code are still able to communicate with pods using the latest code. He extended the protocol to allow communication between pods which is closer to the standards we use. Is the extension of the protocol already a protocol change or do we change to protocol whenever some old implementation becomes incompatible?

Benjamin also added some validations that were fine for our own implementation but not for others. Depending on the definition of “diaspora protocol” he made the protocol incompatible with other protocols or with other “flavors” of the d* protocol which were different from the one we use.

7

@dennisschubert

Also, at this state, I would like to point out that YOU, Jason, even liked our plans which we are realizing at the moment three years ago.

Sure, exciting stuff, still support it. All I’m asking is to respect other adopters and talk with them - not do one sided work with a “fix yours if you want to” attitude. This is the attitude which still you can see as negative towards the whole diaspora project that was caused by the original founders not promising to respect any other software stacks.

You even commented on some of Benjamins pull request and never raised issues and now you act like you never had the chance to participate. I am a bit confused now.

I don’t implement the protocol. As stakeholders here I am talking of the other software stacks.

Maybe you should read the definition first.

‘an individual, group, or organization, who may affect, be affected by, or perceive itself to be affected by a decision, activity, or outcome of a project

Sounds like Friendica and Redmatrix.

This is not the W3C and this is not the W3C Social WG. If you want to create a protocol that can be used with all systems, you have to take everyones needs into account and that cannot and will never be done by a single project. You are part of the Social WG and you should know that.

Yes, being a part of the SocialWG is exactly why I feel that diaspora is the real hope of decentralizing the social web. Tbh, I’m not too sure what the result of the SocialWG will be. Really, the diaspora protocol, or “not protocol” as you want it, is already unifying three different projects and with improvements hey maybe it could be a real unifying thing. I heard even the XMPP networks are planning on federating over some gateway using the diaspora protocol (no, sorry, not a protocol, a “thing”). I know some people from GNU Social have had thoughts at least on this. I know a wip python library that a certain image hosting site was planning to use to federate profiles on their platform.

None of this can happen if we don’t collaborate. You guys, the three who are working on improving the protocol, are in the position to do awesome stuff - but instead today I hear that the plan is to just move thinking only of the diaspora Rails based software stack with no care about any other implementers.

You find this justified so you can iterate faster and talk with less people. I find this the death of the dream of growing the decentralized social web around diaspora. It’s the death of The Federation.

What you are just trying to do is basically blocking our work on a more stable federation inside the diaspora network and bringing our stuff back to already defined and widely used standards.

No, I’m saying respect is needed and dialogue.

Right now, we are moving from a pretty crappy implementation to a well-defined and well-developed ecosystem.

Great, why not do it right then?

@steffenvanbergerem

No, the protocol (or non-protocol thing) is not so well set in stone that one can look at a change in code and say for sure “this will or wont break something”.

But you’re not seeing my point. I learnt today that the 3 people working on the improvements will not consult, hear or care about changes that affect software outside of the Rails based Diaspora software stack. There will be no thought on whether Friendica or Redmatrix federation breaks because “hey we didn’t declare support for it”.

I never said in the OP and sorry if it sounded like that that the changes recently done by Benjamin would have even been possible to see whether they break the integration. The fellows from Redmatrix didn’t come blaming the diaspora developers and quickly Benjamin also submitted a patch to revert some of the changes. Even not doing a hotfix to restore breakage is ok I guess because only a part of the pods are affected.

But the idea that in the future, changes will be made to redesign parts of the diaspora non-protocol thingy with only diaspora in mind and not even slightly giving a damn thought whether someone else might be interested is just disgusting.

Are you @jhass and @dennisschubert really going to go so far as to say yes to the above statement, that you have absolutely no desire for collaboration with other networks, that your only interest is diaspora and only diaspora?

If that is so, I think this project is not for me. I don’t spend as much time on diaspora as at least Jonne does, but I spend enough that I want to make sure it is worth it. I care a lot for diaspora, but I care more about working towards collaborating and networking together. That is the real future not some single software stack. Alone diaspora is not worth the effort.

8

I do not want to spend time helping other projects adopt our broken “protocol”. Instead I want to move to a protocol that’s worth of being adopted by other implementations, asap. And then help other projects adopt it. The current one simply isn’t, and I’m sure anyone doing it is simply looking to reach our userbase, not because it’s in any way neatly designed, standards compliant, reliable, flexible or full featured. So yes, in the current state I do not want to collaborate with other networks in the aspect of helping them to support our current implementation, I think it’d be toxic for them and wasting both our time.

If you remember NodeInfo I very well wanted to collaborate on things that are actually properly designed. But all I got was interest after there was an implementation to copy, that’s not really collaboration and certainly not the kind of collaboration your proposal demands from us before moving forward.

9

If you remember NodeInfo I very well wanted to collaborate on things that are actually properly designed. But all I got was interest after there was an implementation to copy, that’s not really collaboration and certainly not the kind of collaboration your proposal demands from us before moving forward.

Yeah and once you actually got feedback all you did was complain that “it’s too late you should have given suggestions earlier”. Sorry, if you can’t take feedback then you shouldn’t work on stuff that you want others to implement. The same is I guess going to be with the future diaspora protocol then?

Well, my faith in diaspora is dying fast. If you really want to exchange your technical excellence quest to kill the user base it would be great if you did it with some other project than diaspora. If you really feel this then I fear we’ve fallen too far apart. Well, at least it means you don’t consider my contribution to be worth much to even consider changing your opinion.

10

I got angry not because of the feedback, but because I asked long long before for feedback, got none, then spend time on an implementation and then got feedback that would require me to spend time to at least half rewrite my implementation whereas if I had gotten that feedback earlier I wouldn’t have to spend that extra time. In short I got my time wasted, and yes that makes me angry. If you lack the empathy to understand that then we maybe really should just say goodbye.

11

This is not a new problem or one that is even limited to Diaspora. Communication protocols (of many kinds) get deployed and improved, and even radically altered all the time. The way we traditionally do this is to provide backward compatibility for a period of time, while providing some kind of indicator within the protocol that shows what revision or expectations this server supports. It’s unrealistic in any decentralised platform to require all stakeholders to upgrade en masse. This argument would be valid in either a homogenous or heterogenous network. Even Diaspora has outlying nodes that would like to upgrade on their own schedule - not yours. “Be conservative in what you generate and liberal in what you accept” and with mechanisms for backward compatibility to the extent that this is possible is the way good software has always been written.

As an example I recall the transition from SMTP to MIME and to ESMTP. We’ve gone through several flavours of IMAP and LDAP and TLS and HTML and yadda yadda. All went smoothly. Some protocols have even gone through practically “overnight” change, like SMTP and XMPP. I’m just suggesting there are ways of managing these changes. Just quietly changing something in an incompatible way and without notifying other stakeholders has never been particularly well received.

As one of the mentioned stakeholders I will not participate in this vote.

I’m fine with Diaspora changing their protocol. There are a number of things that need work. You certainly don’t need my approval. A warning and a basic idea of what things are changing would be nice so we know what to look for. Otherwise it’s just rude to wake up one day with all your members screaming at you that everything is broken - because of some decision another project implemented and didn’t even warn you about.

I had no idea five years ago that Diaspora would want to be a walled garden even though there have been plenty of indications and warning signs. You can have your walls if you want them. My sincere apologies for trying to break down the darn things.

12

I can’t add much here. We have currently no “protocol”. With my work at the federation-gem I try to change this and move to a more defined protocol with better documentation. And all this without breaking changes.

But I can’t continue my work if I need to test against (and support) 4 crappy implementations instead of only one. And if I would, the result would be crap again, because I need to add even more hacks to support 3 more implementations (which we currently don’t support). At the moment, the federation does not work really well and I want to change that as soon as possible (it was broken long enough). Or do you think it is more important to “try” to support friendica, redmatrix and hubzilla than a better federation for all diaspora users? And I think at the end also friendica, redmatrix and hubzilla will profit from this.

The federation-gem will be (and currently is) backward compatible, but only with the old diaspora-implementation. I can’t guarantee that friendica, redmatrix or hubzilla doesn’t break if they do something different than diaspora.

The only planned changes are not new: #2440 and #5963. And as I said, these will be added backward compatible. I will notify friendica, redmatrix and hubzilla before I remove the old implementation of these issues.

After we have a real definition, better implementation and documentation of the protocol, there will be a better basis to discuss changes or a new protocol. And I can’t wait 8 months for every little change, only to get no feedback … (see nodeinfo).

13

TL;DR: I am OK-ish with “Changes to the protocol must be pre-communicated with ALL stakeholders” (if communicate means “notify them, before we remove something old”) and the currently planned changes are already communicated on GitHub (#2440 and #5963). So I don’t understand why this proposal is needed right now? But I am not OK with this from the OP “Even if it is “YOLO code”, coming from one developer, that doesn’t mean it doesn’t have to be supported, maintained and more importantly, not broken for ALL stakeholders.”. I can only guarantee that old diaspora pods don’t break (and not even that with 100%, because the old implementation is already broken). Otherwise, I can no longer work at the federation.

14

@dennisschubert The current implementation of the Diaspora protocol in Friendica was done in cooperation with Ilya Zhitomirskiy and @mikemacgirvin. I never even knew that there were made some changes on the Diaspora side to support Friendica. I thought the implementation in Friendica was a (perfect) copy of the part in Diaspora.

For me this “We are the development team of diaspora and this is our project. The diaspora users are the stakeholders and not some third party projects developers who are not involved in our development.” sounds like:

“We will change stuff without any pre-warning and we won’t even tell this other persons. Maybe they will be aware of the changes when the federations breaks. But even then we won’t tell them what we changed because we really don’t care about anyone else. Maybe we lose some thousand contacts but we don’t mind as long as they aren’t using our software.”

All I’m asking for is a short notification like what is done in the issue 6405 where there was told what was changed and how to cope with that.

This really shouldn’t be a blocker for a further development of the protocol. I really would like to do all necessary changes on the Friendica side - I only need to know what to change. Please give me some time and some piece of information.

@jhass sorry that I replied so late to your nodeinfo proposal. I was really busy in the last months (not only because of Friendica but also at work) so I wasn’t able to do everything in time.

15

To be honest, it looks like neither @augier nor @mikemacgirvin and @michaelvogel have actually read the comments and the original proposal, which makes me angry as hell. I wrote that long comment for a reason. Thanks for voting and commenting without knowing what is actually going on.

I think we made pretty clear that we will be compatible with older diaspora pods (and thus with your old implementation, if it is 100% compatible). I also made clear that we of course will issue announcements if we ever will remove old compatibility layers, which - again - we do not have actual plans right now.

Jason basically proposed that we should not be able to do any changes without discussion it with third parties first. This is pretty clear by stating:

Do not merge in changes before listening to all the stakeholders and noting their concerns with implementation and timetable

And I am absolutely not fine with that. We are all very busy and I have no interest in working on a project where we need to wait 8 months just to make a little change. That’s killing everyone.

Again, we will stay compatible with older diaspora versions but as much as I like that other networks have implemented the current state, if something is not able to keep up with the enhancements because the implementations differ, then too bad. We’re simply not able to keep everything in mind.

16

@michaelvogel: I think you mix two different things here:

  1. The planned changes which are still backward compatible (the guid and public-key: #2440) and which I planned to tell you, before the backward compatibility will be removed and after I wrote a proper documentation.

  2. #6405: the stuff that broke here was not planned, so I couldn’t tell you before it broke. I added some more validations, and didn’t know that friendica and redmatrix wasn’t valid against them. I removed some constraints now (because diaspora doesn’t use these fields at the moment) to make friendica and redmatrix compatible again.

17

There’s nothing wrong with informing other projects about changes in the federation, but when the devs have to wait for confirmation from other projects, it becomes a problem. It slows down development, and it may hinder improvements because other projects don’t have the time to adapt to the changes.

Last but not least. Either you officially support third parties or you don’t. Keeping crappy implementations just

This isn’t even about compatibility within Diaspora but about compatibility to third parties; however, when Diaspora developers don’t want to wait for approval from other projects they might start a fork, and then we’ll have the problem of making different forks compatible to each other. Friendica support is nice, but it’s not essential; and without a nicely working federation, it’s not particularly useful.

All I’m asking for is a short notification like what is done in the issue 6405 where there was told what was changed and how to cope with that.

I’m pretty sure we could all agree that a notification is nice. In that case, a meaningful commit message and well commented code should be enough to do that. Also (I’m not sure if Diaspora already has that) building documentation from comments may be the perfect way to do this. That way, third parties would have docs they could consult.

Last but not least: Either you officially support third parties or you don’t. Keeping crappy code because you don’t want to break compatibility and don’t want to deal with actually helping those third parties adapt to the new code doesn’t help anyone. That’s called being lazy, and it’s a disservice to third parties and Diaspora itself.

If someone really wants to make sure that third parties can support Diaspora, he or she should help document changes made to Diaspora and help implement them in third party clients.

18

@dennisschubert I read all comments here. You wrote “we will be compatible with older diaspora pods (and thus with your old implementation, if it is 100% compatible)” but you also wrote “Third party networks have implemented old diaspora federation parts in the past, but they have done a pretty bad job at it. The stuff Friendica sent in never was even close to match the stuff diaspora sends, but it has worked with some hacks on both sides.”

So it seems that Friendica isn’t 100% compatible. Like I said: The only thing I’m asking for, is a notice when a part is removed that only concerned Friendica. I really would like to make Friendica 100% compatible. But since I don’t know Ruby or the project structure, I will not be able to find all quirks on my own.

I do not want to veto on changes. I only want a fair chance to make a change before stuff breaks.

19

We do not have exceptions for friendica but we were pretty bad at validating inputs and actually reading and using the standards we use in the past, thus, we are simply not able to tell when stuff breaks. This is why, for example, #6405 happened. This stuff is totally unpredictable. Requiring us to provide notices for unpredictable stuff is nuts and you should understand that.

I do not want to veto on changes. I only want a fair chance to make a change before stuff breaks.

So you disagree with the proposal then. Why did you agree?

Edit: Just to make sure everyone read what I wrote in my first comment:

Some time in the future (and this is going to take quite some time), we will gradually deprecate old federation implementations, thus forcing administrators of old diaspora pods to update. This will be done after a long pre-announcement, but again, we are not even close to think about that.

This is as clear as it gets.

20

I do not want to veto on changes. I only want a fair chance to make a change before stuff breaks.

Well, but that is what the proposal suggests.

Do not merge in changes before listening to all the stakeholders and noting their concerns with implementation and timetable