Questions to ask ourselves

Federation
1

Carrying on from the discussion that formed this group: http://loom.io/discussions/766

Some things worth asking ourselves:

What pieces tend to fail the most during federation issues?

What fundamental flaws do we have in not only our protocol, but how the pod deals with federation?

If we take a modular approach, which way should we consider taking? Put federation into its own layer? Allow for multiple federation backends? Use some kind of translater to turn one protocol into another?

How can we improve how our system that handles federation affects a person with many contacts, rather than just a few pods with many people?

What methods can we use to provide a good middle-ground for developers of other federated socnets to talk to us?

Can we find ways to simplify the system that deals with federation in the future?

Note: This discussion was imported from Loomio. Click here to view the original discussion.

2

Carrying on from the discussion that formed this group: http://loom.io/discussions/766

Some things worth asking ourselves:

  1. What pieces tend to fail the most during federation issues?

  2. What fundamental flaws do we have in not only our protocol, but how the pod deals with federation?

  3. If we take a modular approach, which way should we consider taking? Put federation into its own layer? Allow for multiple federation backends? Use some kind of translater to turn one protocol into another?

  4. How can we improve how our system that handles federation affects a person with many contacts, rather than just a few pods with many people?

  5. What methods can we use to provide a good middle-ground for developers of other federated socnets to talk to us?

  6. Can we find ways to simplify the system that deals with federation in the future?

3
  1. The two major issues seem to be missing roots for relayables and either GUID collisions or stuff that got send twice. The real issue behind this two is IMO some state in the protocol, like letting the receiver answer “Oh, I’ve got this already, thanks” and “Oh, hm I never got the root for this, can you send it again please?”

2.) Carrying out the feedback of failures to the user, especially for remote user discovery we should let them know why it failed, not only that.

3.) http://loom.io/discussions/612

4.) We need some “pod is permanently offline” mechanism IMO. Maybe kind of a not reached in x tries in y days where the x/y would need to be told from experience.

5.) On a protocol or on a people level?

6.) Yep, with 3.)

OT: Asking that many questions, which is roughly equal with the same amount of topics, kinda defeats the purpose of creating a dedicated group IMO :wink:

4

First of all, thank you for supporting and creating this group!

Personally I would like to first get clarity how many people know current implementation of federation in Diaspora well and together document it with nice diagrams and references to code/tests.

To my current knowledge we use protocols from OStatus stack but not staying compatible with OStatus itself. One of protocols PUbSubHubbub currently moves to version 0.4 which tries to answer issues of privacy. ActivityStreams have moved quite a lot in direction of JSON (not sure how it looks in Dia codebase). Webfinger ATM gets some solid feedback and rework on apps-discuss@ietf.org with attempt to not duplicate discovery with SWD worked on by folks from OpenID Connect etc.
Also meantime people from https://rstat.us have developed OStatus compliant platform with RoR, where I hope we can share some gems!

So again I look forward to work in a team here with people who want to dedicate their focus to all federation related challenges :slight_smile:

5

I have a few more questions:
How active is OStatus as a protocol? The main OStatus site has been sitting in a hacked state for a while now, the last news item from 2010. The ostatus mailing list has only a handful of posts in 2012. It doesn’t instill a lot of confidence in the protocol.
Next, let’s say hypothetically diaspora was magically rewritten over night to use the ostatus set of protocols instead of the current diaspora specific protocol. What features would need to be dropped for that to happen (assuming you didn’t extend the protocol)?
Next, what sites can actually interoperate via “compliant ostatus” requests?

I think the end-game of a fully distributed social networking world is a standard federation and one or more API protocols (the distributed social networking equivalent of SMTP and (IMAP|POP) respectfully). It would seem that OStatus would fill in the SMTP gap, but if the standard doesn’t have any traction, we either need to take over and reignite interest in the project, or else propose a new standard. Either way, we need to work REALLY HARD to reach out to other similar projects and get other dis. soc. sites to buy in.

6

Unfortunately, OStatus is very limited in what it can do as a protocol. To my knowledge, it still technically doesn’t have support for private status messages out of the box.

I think that having OStatus compliance would be really great for network compatibility. We already have partial support for it (identi.ca users can subscribe to Diaspora ones), but I don’t think it would necessarily take a rewrite of our protocol spec or handlers. Evan Prodamau of StatusNet had expressed interest in working with any interested community devs to get this compatibility to happen.

7

I would caution against focusing too much time and resources only on diaspora specific protocols - instead I would prefer to use the unique needs of diaspora to help drive forward a shared standard.

Since OStatus is currently one of the few existing standards that exist, I think we might do well to start with it as a base, and fill in the gaps as needed, while working with evan and other networks to drive OStatus forward as a shared protocol.

Concurrently with that process, and assuming we also make our federation protocol more modular and versioned, we could continue to support both the existing, though limited, OStatus protocol in addition to our existing diaspora protocol. Some of the lessons learned from that process could actually inform the development of OStatus v.next.

8

Hi I’m Simon from buddycloud. This was our thinking on federation - perhaps it’s useful for you.

  • we worked out we wanted a unique identifier that looked like a user’s email address: (name@domain.com not http://name.domain.com)
  • we also wanted to have messaging between users and ideally hook into existing IM clients and libraries.
  • we weren’t particuarly keen on inventing any federation systems ourselves, or spend a lot of time working through edge cases. (For example it’s taken the XMPP project 10 years to work through federation)
  • we wanted private posts and public posts.
  • we wanted to use activity streams for representing content.
  • we are very keen on encryption between nodes (pods).
  • oStatus seemed like a good idea but doesn’t support private posts - that’s really important for us (a private channel in buddycloud).
  • were there good existing libraries (server is in node and we had some ruby bits).

In the end it seemed like a nobrainer (at least for our team) to go with XMPP and simply pass any messages to the XMPP server and have that “just work” for us. This also bought us user to user messaging for free and a bunch of existing chat clients that help draw users into the full experience.

Happy to help if there are any questions.

9

Hi, I’m rekado from Libertree. We’re currently transitioning from our bare TCP/IP protocol to XMPP. In later iterations we will be using activitystreams as a representation of user activity as well.

What’s good about XMPP is that we can outsource everything that relates to message distribution to other software projects. Distribution can be delegated to an XMPP server by running a simple XMPP server component that is connected to the server via TCP socket.

Using XMPP as a transport layer, however, doesn’t mean that we’d automatically be compatible with other networks. (That would be similar to saying that all of the HTTP based protocols are compatible because they are built on top of HTTP.)

Personally, I think that XMPP is a better fit for distributing messages than stateless HTTP.

What I’m interested in is this: how can we make our networks transparently work together at a higher level than just cross-posting and polling remote feeds?

10

Hi, I’m one of the architects of the Tent protocol.

Our primary goal is to build a generic, extensible framework for data storage and communication that is developer friendly. This protocol encompasses both server to server communication, as well as app to server communication.

Here are some of the requirements we started with:

  • All features must work if every entity is on a different server.
  • The protocol must be usable by client-side webapps (Javascript) as well as native mobile applications.
  • Developers must be able to create new content types without modifying the protocol or servers, and they should be able to adopt these new content types organically.
  • Data and relationships must be portable. This means the protocol must handle changing identifiers and data export/import.
  • Users need comprehensive, granular privacy controls.
  • Data should be pushed between servers, not polled.
  • If a server is down, all content that was previously sent by that server should be cached.
  • Content must be easily linked and referenced.
  • Reposts of content are references, not copies.
  • Users must be able to use many applications with a single server, and applications do not act as servers.
  • Apps should only talk to a user’s server, the server is responsible for talking to other servers.
  • Server to server, and application to server communication must be encrypted.

What we ended up with is a protocol built on HTTP, TLS, and JSON and based around the concept of posts.

A post has metadata, JSON content, and optional binary attachments. The metadata includes a post type, timestamps, permissions, and version information. Each post has one or more versions. These versions build a graph of modifications, similar to how git and other distributed version control systems work.

Apps create posts on the server, and then the server handles sending these posts out to other entities and apps via webhooks (and in future versions, a streaming connection).

Servers render feeds of posts to apps based on query parameters that allow filtering based on metadata such as the post type, publishing entity, etc.

We evaluated just about every related protocol and specification when designing Tent, and chose building blocks that did not limit our goals.

We did not use OStatus because it did not meet our requirements. OStatus is designed for microblogging, not generic content storage and communication. It also does not support private posts, and does not specify an application API. In short, it doesn’t really meet any of our design goals.

We did not use XMPP because it requires persistent non-HTTP connections, making it unusable for Javascript and mobile applications. In order to keep the protocol simple, we use the same building blocks for application and server to server communication.

We also evaluated other existing prototype federation and decentralized social networking systems, but found that their designs were limited to specific interactions common to “social media” and did not meet our requirements.

As we’ve mentioned previously, Diaspora would make a great Tent app. It has a usable UI for rich social communication, and shouldn’t need to be responsible for robust federation, which is solved by Tent. Diaspora would be a Tent app, and relies on a user’s Tent server for reliable federation and data storage.