The last I heard about backup and restore hasnt been given much interest. Any reason for that?
https://github.com/diaspora/diaspora/issues/5343 has some progress.
Note: This discussion was imported from Loomio. Click here to view the original discussion.
I hope I can participate in it in a while, after some present jobs are done.
The latest automatic back up / restore spec proposal includes the import parts in this issue. Would be fancy to get more comments on it, split it into smaller chunks and dump a lot of bounty money on the issues - maybe get something to happen 
Anyone wanting to give their insight, please check the proposal and comment in issue #908 . If no comments are received after some time I could just create a proposal to accept the proposal as a guideline for development.
Please don’t start working on any import thingy before at least considering that proposal 
Its a very detailed proposal. Isn’t there a simple way like mysql dump to csv/json and import csv/json ? Pod to pod communication sounds very complex.
Dumping database would not work for ordinary users very well - and certainly would not work in the situation where the pod just disappears. And, importing raw SQL dumps into other pods would probably not be a very nice thing to do
Regarding JSON, yes we already allow exporting data. The proposal details how this data could be to not only restore the data to a new pod (profile, etc), but also migrate the identity, and thus take control of previously shared posts etc.
Regarding JSON, yes we already allow exporting data.
Yes now we need an import function but your proposal looks too big and lot of stuff needs to be implemented. Agreed its the optimal way to go forward.
but also migrate the identity, and thus take control of previously shared posts etc.
Can’t we have something just for exporting all the posts and its comments which are imported back under new account. " Taking Control" sounds complex.
Can’t we have something just for exporting all the posts and its comments which are imported back under new account. “ Taking Control” sounds complex.
That would create duplicates and there would be no interaction possibility. No comments to the old ones would arrive to your new identity, for example.
Sure, you can have (almost) anything (sane) - just find a coder to do it The issue has been there for a while waiting for someone to grab it. This proposal aims to solve the disappearing pod problem and clean migration. That doesn’t stop anyone doing partial or full imports before a “proper” migrate solution is done.
Can’t we have something just for exporting all the posts and its comments which are imported back under new account. “ Taking Control” sounds complex.
So you would like to share all your previous posts again from your new profile?
So we have User and Person models. On migration we must create a new user but do we preserve the same Person model which was known for that person? I suppose we should.
Profile data export package contains also posts and comments which can be ignored during the restore process.
I don’t think posts and comments can be ignored. It is possible that the new pod doesn’t contain some posts (especially private). I believe it must not be dropped and we should fetch them on restore also.
https://wiki.diasporafoundation.org/Account_Backup_And_Restore#Backup_delivery_message
“backup” is signed and encrypted using the user private key
Where do we store signature? Don’t we have to have a separate field in the schema for that?
Pods should schedule backups of user data once a week to the backup pods.
I think it is not frequent enough. Some people post frequently, and one week would be a huge loss for them. How about making that adjustable according to person’s activity level? Or simply backup every day if there were changes. If there weren’t, don’t backup at all.
I also beleive we must support some kind of limitation for pods backup capacity. For example a pod wants to store backups, but no more than for 100 people. Then it shows that it doesn’t receive backups anymore, but previous still work.
So we have User and Person models. On migration we must create a new user but do we preserve the same Person model which was known for that person? I suppose we should.
Yes, we should keep the same Person as it is the same identity, just moved local instead of remote.
I don’t think posts and comments can be ignored. It is possible that the new pod doesn’t contain some posts (especially private). I believe it must not be dropped and we should fetch them on restore also.
Well, it’s not feasible to import posts or comments imho. What purpose would that even solve? Your contacts would not see any of the uploaded private posts unless you also push them out to them - which would be bad because they already have them from before.
The main point is to save the identity, not every crumb of data related to it.
Where do we store signature? Don’t we have to have a separate field in the schema for that?
This would be the same signing method we use for delivering content - ie the receiver only accepts it after verifying the signature in the payload against the public key of the person. Whether this should live in the diaspora federation gem where the code is readily available is up to question. I’d say no, but of course it would be easier to implement that way. I don’t consider this part part of the protocol as such, but then that is not in any way under specification anyway.
I think it is not frequent enough. Some people post frequently, and one week would be a huge loss for them. How about making that adjustable according to person’s activity level? Or simply backup every day if there were changes. If there weren’t, don’t backup at all.
I believe this can be left as detail when the thing actually works. If we leave out posts and comments, which I think is the only real way to do it, then there is absolutely no sense in sending backups daily.
I also beleive we must support some kind of limitation for pods backup capacity. For example a pod wants to store backups, but no more than for 100 people. Then it shows that it doesn’t receive backups anymore, but previous still work.
Again, if posts and comments are not part of the archive, it will always be very small. Additionally, backup pods would be randomized, guaranteeing some balancing of load between pods. Of course we can introduce a setting like this, but personally it doesn’t sound super useful, considering how much data pods store normally compared to how much this would add.
One interesting case is what happens if the pod has turned off sign-ups in the event that someone wants to restore? I think we should bypass the sign-up not being on and let the user restore. This would speak towards the setting you mention, so that pods don’t gather too many possible identities that could activate suddenly.
Great comments, thanks!
Well, private posts may contain some valueable information for the user herself. I don’t have extra copies of texts I posted, and I don’t like to lose these texts. Moreover it would be definitely nice to preserve a possibility to continue conversation on some private post that was going on before the move. I don’t see why we should push them again. A guid of the post is preserved, so all new comments will be federated well to the right place.
Not to say about public posts which might be wanted to be shared with some new contacts.
So I think posts, the content is extremely important part of the network. That’s why people are in it. And the restore feature without posts restore would look unfinished.
Maybe if we push posts to the restore pod as if there were some subscriber on it could make restore easier, since we won’t need frequent backup then and any special posts restore feature. At least we could do that for public posts, since for private posts that would imply trust for restore server even before a password was entered. That is not really acceptable.
https://wiki.diasporafoundation.org/Account_Backup_And_Restore#Backup_delivery_message
“backup” is signed and encrypted using the user private key. Once opened, it should contain the following schema…
It’s not very clear the structure of the backup field before we “open” it. It must be some data array containing signature and encrypted data, right?
BTW, does any other software in the federation (friendica, redmatrix) do some sort of backup restore?
So here are my changes to the spec according to what we’ve discussed
https://wiki.diasporafoundation.org/index.php?title=Account_Backup_And_Restore&diff=4454&oldid=4418
@comradesenya unfortunately I’ll unlikely have much time for going through these before friday - I’ll reply then. Until then, it would be nice if the wiki could be left alone before accepting changes here in Loomio. I don’t agree (still) to some things like including posts and comments in the backup archive and I’m not prepared to change my opinion unless some others join in and support that idea and also give valid technical ways to do it sanely which at the moment is missing.
Anyway, lets continue discussion on that, for a few days I’ll have to pass on comments. I’ll clean up the wiki page to reflect those items that have been mutually agreed upon then.
Once we have a clear understanding we can do a proposal.