Personal Data Import/Export

@goob sorry haven’t touched it since I’ve involved myself in way too many things and since I myself don’t need a pod migration script any more, I’ve not needed to touch it yet. I plan to at least package it properly and then that will be it.

This script is just a hack and not usable for permanent code. The proper way should be to allow the user to 1) export contacts as CSV and 2) import them on the other pod - that would do exactly the same thing.

No problem, and no need to apologise. I just came to this discussion to bump it, and saw the end of it, I thought I’d ask where you’d got to.

Anyone got any suggestions about how we can get this moving? What would people need in order to create such a facility? And how can we enthuse people to work on this? We have agreement that creating a manual export/import as an interim measure is desirable, but what steps do we need to take now to try to get things moving.

Wish I could help with code!

A manual export is not an interim solution - we need that anyway, so would be a good place to start. I guess we need some blueprint on the data export format, contents, etc.

@jasonrobinson how would you suggest we proceed to get this blueprint and so on?

@goob someone needs to write something up and propose I guess :stuck_out_tongue: On my todo but I don’t think I’ll be having time to tackle that any time soon :frowning:

Maybe a good idea would be not to try and cover everything at once - instead just some basics like contacts which imho is the most important thing to start with.

You suggest exporting in CSV. The old system was XML.

Does anyone else have suggestions for the best format to use for export? As suggested by Jason, we’ll take a vote just on contacts export first, but it might be necessary to take into account other things that will be exported in the future (posts, photos, etc) in deciding which format will be best.

In fact, the ‘old’ exporting in XML still works - I just tried it. I thought it had been broken at some point. Is there any reason we can’t retain this export as it is, and look to build an import for at least some of the contents?

What about exporting everything as JSON?

I agree with @jasonrobinson; exporting and re-importing contacts is the only feature I need to migrate to another pod and definitely abandon joindiaspora. Jason, I tried your hack script but it fails at login on source pod, returns 406.
No preference on format since I don’t develop on this project t–I’d just say, on the projects I work on, whenever I can I dump XML for other lightweight solutions - csv mostly.

@goob I’m pretty sure I meant to write JSON :slight_smile:

I’m not sure we really need to vote whether we want to export contacts? Surely we do not want to be worse than Facebook - which allows you to export just about all the content you post :wink:

But of course if someone wants a vote we can vote. Still, someone needs to write some code for it, I think that is the real reason the export still sucks - not that we don’t need or want it.

I agree we don’t need a vote on whether or not to create an export facility - it’s essential, and in any case the vote has already been taken - I meant that we should vote on which format to use for export. My previous comment wasn’t very clear about that.

Worth taking a vote on using JSON as the export format?

@goob yeah sure, we can do that, then make a blueprint for the actual contents in the wiki

Proposal: Use JSON as format for account export

We need to have a means of exporting account data (contacts, posts, etc) for migration to be possible. It has been suggested that we use JSON as the export format. Do you agree?

Outcome: We will use JSON as the format for exporting account data.


  • Yes: 20
  • Abstain: 9
  • No: 1
  • Block: 0

Note: This proposal was imported from Loomio. Vote details, some comments and metadata were not imported. Click here to view the proposal with all details on Loomio.

@goob about the old export btw - it doesn’t really work. It exports a huge amount of stuff that a lot of is totally irrelevant to a user. Did you look at it? :slight_smile: Also the XML doesn’t necessarily even come out valid - bug

@jasonrobinson I’ve not looked at it in detail, just saw that it contains contact information, post information and one or two other things (I think, can’t remember now). When I said ‘it works’, I simply meant that the export button does still provide you with a dump - not necessarily that it was in any form which would be useful for importation elsewhere.

@rich1, now you’re in the swing with JSON, how about creating an export feature…? :wink:

I could write one in .NET - not sure it’d be of much use to anyone though?? Unless it’s externally hosted perhaps… Hmm… I’ll have a think about that :slight_smile:

@rich1 well it needs to be part of the d* core code - but if you have good ideas on the structure, feel free to create some specs on the content :wink:

Hey, new version of diaspora-tools out, Python script to migrate contacts across pods - a little more reliable now, uses fresh diaspy and also works out of the box with SNI SSL pods with Python 2.x (like my pod :P).

As written down in my explaination: I’d go for JSON as it comes to contacts/aspects, since those are simply data.

On the other hand, posts are embedded in context, which XML can better map, I think.

When it is zipped anyway, why not split those information?