Public post federation

(Mark Williams) #61

Glad to see there’s still interest in this topic. I’ve made some progress on a prototype solution for public post federation using Pastry; I’ll update this thread when it’s ready for testing on real pods.

In case this needs more clarification: the local database for a pod, regardless of whether it’s a relational database or a key-value store (and hard experience taught the D* team that a key-value store is not the appropriate choice), is completely distinct from the DHT-based distributed network layer (Pastry, in this case) I’m proposing to use to implement federated public post features. This is a “third way” which avoids the type of naive network search that puts a large burden on the network, while also avoiding the need to introduce central hubs.

IMO central hubs would weaken D* by creating a dependency on special, well-connected, resource-intensive nodes, which would be too expensive for the average person to run. This would erode much of the benefit of a decentralized D* network; it would make it much less democratic, more susceptible to interference and obstruction. It would also be a less robust solution than distributing responsibility for doing federated processing across many/all D* pods.

(Jason Robinson) #62

@markwilliams2 looking forward to seeing your update!

(Trolli Schmittlauch) #63

I guess @melroyvandenberg 's point is that if we want to federate all public posts, currently this would require every pod to receive and save all public posts locally.
Saving posts locally is ok as long as only the posts from contacts in aspects of users of the pod are saved. This isn’t much data, at least on small pods. But saving all public posts requires a lot of more space and resources, doesn’t it?

@melroyvandenberg 's idea seems to be dynamically getting the posts from other pods only if requested by the user. I don’t know whether this can work, but this is my understanding of the issue.

(raven24) #64

I’m a little skeptical to introduce a whole new concept like DHT. It may seem appropriate, but I also detect faint amounts of “if all you know is a hammer, everything looks like a nail” … :wink:

So, what about the pseudo PubSub stuff we have going for relaying interactions on posts to/from different pods? I suppose that could be extended into a full-blown PubSub system where users could actually subscribe to contents on other pods…

(Jason Robinson) #65

@florianstaudacher the requirement IMHO is that everything is transparent. If a user follows a tag - posts should be in an ideal situation seen from all around the network - just like on Twitter. That is at least something I would see that would be the only way it would work - normal users will just be confused if they have to do additional work to “enable” content from other pods.

(Melroy van den Berg) #66

@jasonrobinson Exactly my point.

(Maciek Łoziński) #67

@melroyvandenberg, do you suggest that pods should be connected on a database level (Cassandra) instead/in addition to the protocol level?

(Melroy van den Berg) #68

I think that would be indeed a wonderful solution. Except that you should think about privacy in transporting this data between the pods, similar to the way that privacy is also important of the current data which is sent using the current protocol.

This way we can see other posts from every pod in the world, meaning we are fully finally connected between the pods (just like a big network). Which is your main goal after all, right?

(Jason Robinson) #69

Started thinking of a hacky non-core code requiring solution that would allow users to participate via pod looking relay servers. Terrible in terms of privacy compared to the original proposal, but it would be easier to implement :stuck_out_tongue:

If it worked and would be wanted, then core code could be introduced to make it transparent.

(aj) #70

i kind of like the way diaspora forms communities and groups in sort of an organic way, if there were a common aggregate of all public posts like a global pubsubhubbub or whatever it would maybe kind of change the way of it…

(Jason Robinson) #71

That works fine for medium to large pods, but single user and small pods are lonely places until they share with lots of contacts. One shouldn’t be required to follow hundreds of users just to see public posts.

(aj) #72

ya starting my pod i more or less had to find contacts on jd and then search for the same contact from my pod to add it, a real pain… would be great if a new pod could get a feed of public posts from one of the larger pods, at least for few weeks after being added to the network

(Jason Robinson) #73

Updated proposal specifications - and now with some PoC code

See here:

I’ll continue working on the code part and hopefully aim to submit a PR to diaspora core for at least post relaying within my summer holiday (so under 2 weeks).

(Jason Robinson) #74

hey @jhass @dennisschubert and others. What do you think about the proposed pod settings this stuff would need? I’m kinda ready to implement the last part of the relay ie querying single pods and pushing posts out to them. So I could also do the PR towards diaspora - for the inbound configuration part first, then second the outbound configuration.

(Jonne Haß) #75

statistics.json/NodeInfo is about metadata, not protocol extensions, that is protocols shouldn’t make decisions based on its output.

I’d say just add a .well-known route, /.well-known/x-diaspora-relay or something.

(Jason Robinson) #76

@jhass Fair enough, that might make sense. I’m supposing it should be constructed by the diaspora-federation gem?

But in general, you or other core members don’t object to the extra configuration for diaspora as proposed, assuming the change is that it is reflected in .well-known, not nodeinfo?

(Jonne Haß) #77

Well, it’s needed for the feature to work, right?

(Jason Robinson) #78

@jhass as written currently, yes. The other option is each podmin configures their web server to serve a manually maintained file :stuck_out_tongue:

But really, if the configuration would not be accepted to diaspora, as a fall-back I would centralize the idea to and add to it a form for podmins to register their subscription preferences. The send part is more difficult, that would require a patch commit that podmins could pull in if they wanted.

I personally don’t see any harm in including the configuration in diaspora itself. I believe this is a good way forward to attempt to fix some of the issues caused by the federation model ie small pods not able to receive enough public posts for it to make sense to set up a single user pod. Also, this would allow pods to customize their scope to specific interest areas, if the solution gets wider adoption within the network.

(Jason Robinson) #79

I’ll make it .well-known/diaspora-relay. RFC5785 doesn’t state prefixing names with x- and it doesn’t seem common looking at the registry. I’ll submit a registration request also for this .well-known.

Thinking about it, we should really put version and protocol generic information in .well-known/diaspora, not in headers/statistics as currently is done. But not in scope of this :slight_smile:

(Jason Robinson) #80

Or actually, it should probably be .well-known/social-relay, not diaspora-relay. Nothing in the relay concept the way I think about it is diaspora specific, except the initial implementation is geared towards diaspora.