Backup and restore – account migration

Features and Ideas
41

but AFAIK there is no formal specification document for the federation protocol.

True, and at this point, there is no point in writing one. After the federation-gem efforts are done, I intent to formalize what we have at that point so we have a reference documentation for further usage and other networks as well.

I’d like to add something for Jason here. You are trying to write a spec, which is much appreciated. “Spec” is short for specification, but I get the feeling that you are actually trying to be as vague as somehow possible. I see why you’re trying to do this, but if you want to come out with something that people might actually use, you have to do the opposite. Be as specific as possible. Otherwise, people will never be able to adopt the spec, everyone is going to maintain their own set interpretation errors and nothing will ever be compatible.

Example: Defining terminology.

Bad:

username contains a string.

Good:

username contains an identifier that uniquely identifies an user across the network.

Example: Defining formats.

Bad:

groups contains an array of contact groups.

Good:

groups contains an array of objects specifying groups. Group objects should contain a title attribute as well as a members array. title should be the human readable identifier. members should be an array of usernames.

Do you understand what I’m trying to say?

42

Regarding diaspora protocol. Yes, the initial version in the wiki that I wrote in October had the idea that the whole backup/restore would be implemented with minimal changes by hooking up to the current federation endpoints, signing, etc. This is definitely the easy route, I and to be honest I wouldn’t of imagined any other way.

However, it is clear that there is heavy work going into the protocol itself and I’m not at all sure it makes sense to smash a concept like this that has absolutely nothing to do with social content in. This would weaken not only the protocol but also make the backup/restore feature unstable due to breaking changes in the protocol itself.

As such, I began looking at it from outside of diaspora. As a separate feature, not conflicting or requiring the diaspora, or any other, protocol, it would become much more stable to implement. The downsides are of course clear; more time spent on defining the spec (or specification, thanks Dennis) and more time spent on development (signing methods, endpoints, etc). The end result however would be cleaner and more stable not just for the backup/restore feature, but also the diaspora federation protocol.

I know it is hard for people here to see the federated web outside the context of diaspora, but that is exactly what I would like to do here.

So, there are two ways forward:

  1. Implement using the diaspora protocol. Some of the code would then have to be implemented in the diaspora federation gem, making it a permanent feature there.
  2. Continue writing a more generic specification and implement using that, with no changes at all to the diaspora federation gem.

I’m not too interested in 1) to be honest, but if that is chosen, then imho implementation can start whenever a basic flow is agreed upon, and it’s already there written waiting for comments. There is no need to write fancy specifications because implementing it would require understanding something that has no specification and that needs reverse engineering. If anyone is able to do that they should be able to follow this half done spec already.

I’m really interested in 2) and that doesn’t really even depend on diaspora implementing it. For that, there is much work to do and many comments needed, from outside of diaspora developers. I’ll do what I can to get some comments.

Btw, regarding 2), I’d like to make use of as much of existing or upcoming standards as possible.

Using ActivityStreams2 for the JSON messages for example would enable using existing and upcoming parser libraries - the spec is currently a W3C working draft and is moving on nicely. Backup/restore would use extensions to it where needed as per AS2 spec.

For signatures, there are a few that have been mentioned within the W3C SocialWG group. The most important thing would be to use something that has support on implementation or standardization level. It is possible signatures will be a part of ActivityPub, the federation spec from the SocialWG, but this is not guaranteed unfortunately.

So, how should it be for diaspora - backup/restore on top of the diaspora protocol or backup/restore implemented as a feature nothing to do with the diaspora protocol?

Should I create a proposal now, seems a good thing to decide before moving on with specifics?

43

I still don’t see the connection between a spec describing how data should be exported and a spec describing how servers should communicate with each other.

44

I still don’t see the connection between a spec describing how data should be exported and a spec describing how servers should communicate with each other.

You’re looking at only the export/import part - which is manual. I only added that by request from @jhass . The main idea of the spec is to make backups automatic, using the whole network as a way to protect user identities (= one pod goes down, only some backups are lost temporarily).

To make this kind of automated system, servers need to understand each other. I’m not sure how that can be done without defining how servers talk to each other in the spec.

45

I see. As usual, I like well-defined standards more than self-hacked stuff. If you feel like AS is your way of exchanging stuff, sure, go ahead.

But how does ActivityStreams solve your issue? Sure, you got a way to exchange json objects, but ActivityStreams is rather incomplete at defining the fields (which, obviously, is done on purpose to keep AS extendable). You still have to very carefully design the actual fields to export? Isn’t that the main issue?

46

You still have to very carefully design the actual fields to export? Isn’t that the main issue?

Yes that is one large issue. It’s certainly not the main issue imho, I’d say the signing is very critical.

Anyway, assuming diaspora as a project would want a backup/restore feature, and let’s assume for the sake of this discussion that the answer is yes, a decision needs to be made I think whether to implement it on top of the current federation layer or implement it as a separate feature not related to federation layer.

And if you want to discuss content further, I’d say implementing on top of the current diaspora protocol we should just go with the current fields as per export schema (more basic set for the automated backup archives, but from the same set). A clearly defined schema is needed however for 2).

Btw, it’s nice you didn’t even read the document before commenting on it :wink:

47

Oh, don’t worry, I have read it. I stopped at the point you tried to specify how the user has to sign in and what buttons he has to click as well as the point where you started to assume rules about how implementations should store their stuff in the database, which is why I wrote the initial comment. Nice try, though. Maybe it’ll work next time.

Feel free to open a proposal.

48

Thanks Dennis, as polite to other people as always :slight_smile:

49

Proposal: If diaspora* implements the backup/restore specification, should it base it on the diaspora federation protocol?

Assuming diaspora* might want to implement a backup/restore process (which enables migrating across pods), should it build this support into the diaspora protocol or base it on a separate specification, not modifying the diaspora protocol to support this feature?

Note! The backup/restore spec COULD be one like https://the-federation.info/specs/backup-restore/, but this proposal is NOT about whether to implement this specification or not.

Votes:

  • YES - any backup/export type specification should be built on top of the diaspora federation protocol
  • NO - any backup/export type specification should not be implemented on top of the diaspora federation protocol.

For clarifications, see for example comment https://www.loomio.org/d/qsVJ2K1t/backup-and-restore#comment-923083

Outcome: N/A

Votes:

  • Yes: 2
  • Abstain: 3
  • No: 0
  • Block: 0

Note: This proposal was imported from Loomio. Vote details, some comments and metadata were not imported. Click here to view the proposal with all details on Loomio.

50

Well, I like both ideas. The first option I like for its simplicity of implementation. The second is good as it is a completely separate piece of functionality, and the generalization would make it possible to use the spec even in some applications where we don’t have social media at all. Like integrating backup/restore in some completely different platform, like, for example, Loomio. It is a big, complex and interesting task. The main question is whether it makes much sense to implement this spec somewhere outside the diaspora* federation world? If one could name a few decent applications of the spec outside the world of the software which supports the diaspora* federation protocol, I would probably support the latter proposition.

I believe that the issues with diaspora* federation protocol instability is not a big deal for the topic, because there will be transition periods which will guarantee backward compatibility for sensible periods of time, so it’ll be completely transparent for the feature because we operate on the upper level and the transport level is provided to us.

51

@renatozippert “disagree” does not mean the data will not be sent over the network. Apparently, that’s not even a point open for discussion here. The question is if the spec should be based on diasporas protocol or something different like ActivityStreams.

52

Thanks @dennisschubert… I’ll abstain for now then and read more about it. I need to understand this better.

53

If it helps, we can first vote whether to work on this feature at all - if that is required to continue. I feel it would be better to decide how to do it first and then vote to approve it. It would not be fair to vote on approving something that isn’t finished as a plan.

@renatozippert the whole backup/restore idea here is about automatic encrypted backups over the network. If you are wary of this please vote disagree when there is a vote to approve the feature for implementation - as said the current proposal is not approving the feature for implementation.

@dennisschubert - ActivityStreams2 is not a protocol, it is a content serialization specification. I doubt it can be used to federate anything…

54

The problem of basing this on the protocol would be having to change the protocol that is related to other projects?
Right now I tend to agree with basing on the protocol, as creating the backups and moving profiles would be much more elegantly implemented with protocol support, not as something “external”.

55

Ultimately I would love to see our federation protocol getting used for all kinds of stuff, but right now I think it wouldn’t make sense to take it much further than “social media” content.
For me, an account backup-replication protocol is out of scope of the original federation. I’d imagine in a far away future you could sync your account backup to some not-even-related-to-diaspora service (e.g. owncloud?) and that shouldn’t depend on implementing anything else other than the backup endpoint, imho.

56

Ok, how about the following idea.

We agree on the backup/restore message set and implement it with both federation and non-federation based transport protocol? This is redundant work, but not that big, and it will make happy everyone. After a while, we’ll see, which version got integrated to the third party services (that’s why we standartize it at all) and drop the one, that is redundant. I read that this is how decisions were made in Linux kernel sometimes - by including two competing technologies.

So, as for the specification, it must then be made agnostic to the transport protocol used.

P.S. Does XMPP have anything on the topic?

57

Hi ! I do not understand what is technically going on here, but it is great to hear that this fundamental feature is on its way. Thanks !

58

I don’t consider myself competent to vote on this, so will leave it up to those of you with the technical knowledge required.

59

Proposal: Shall we encypt the backup archive when a user exports it manually?

Currently we have the archive export feature and it is being exported unencrypted. It is convenient since a user can browse the contents of the archive himself. On the other hand, if the user by his mistake (or some malicious software) passes the archive to a third party, it would lead to bad consequences. We might encrypt the archive with a password like it was proposed for the automatic backup feature. (Here is a possible rough implementation of the archive encyprtion presented.) This way it gonna be more safe, but unreadable without some extra tools (Here is a snippet for decrypting archives produced by the modified export feature).

Outcome: N/A

Votes:

  • Yes: 2
  • Abstain: 3
  • No: 8
  • Block: 0

Note: This proposal was imported from Loomio. Vote details, some comments and metadata were not imported. Click here to view the proposal with all details on Loomio.

60

I would recommend not encrypting the data; it adds additional complexity to the system that the developer and user ultimately needs to keep track of, introduces additional failure points, while not providing much in the way of actual benefit to the user.

Any user can encrypt the data him/herself using a variety of tools once its downloaded.