Backup and restore – account migration

[tclaus (Thorsten Claus)]

The labels are complete.
The #8274 pr includes (now) all migration stuff- separated by commits.
The reason simply is that all of them are needed to let it work.

[flaburgan]

I asked for framasphere users who wanted to be beta testers and I’m running now their archive import locally with your latest code Thorsten.

I just had a rake aborted, apparently because another pod: https://gist.githubusercontent.com/Flaburgan/add91d40b252726337d3ebf23e0aa2e9/raw/1036c0268e5bac2725d4a8bf01d8889b2c7f54f3/Import%20test

I contacted the podmin to have more info about it: Hello @Nate Bohman, while testing the import feature (see this conv... is pod is running the next-minor branch so should be up to date.

But if the problem is really coming from his pod, I guess we need to be resilient and not fail because of that.

Note that when I launched the migration again, it worked: Migration finished took 105.102874114 seconds. (Photos might still be processed)

[flaburgan]

So in the end the import went well, and in the end posts, contacts and pictures are there. The pictures which were profile pictures are not imported but I guess this is intended?

[flaburgan]

Still importing locally accounts of beta testers. I saw this “ROLLBACK” in the logs, not sure what it corresponds to so I wanted to mention it:

[2021-09-13T12:30:46] DEBUG PID-19627 TID-62280 ActiveRecord::Base:   ActsAsTaggableOn::Tagging Create (0.3ms)  INSERT INTO "taggings" ("tag_id", "taggable_id", "taggable_type", "context", "created_at") VALUES ($1, $2, $3, $4, $5) RETURNING "id"  [["tag_id", 365], ["taggable_id", 3175], ["taggable_type", "Post"], ["context", "tags"], ["created_at", "2021-09-13 10:30:46.140125"]]
[2021-09-13T12:30:46] DEBUG PID-19627 TID-62280 ActiveRecord::Base:   ActsAsTaggableOn::Tag Update All (0.3ms)  UPDATE "tags" SET "taggings_count" = COALESCE("taggings_count", 0) + 1 WHERE "tags"."id" = $1  [["id", 365]]
[2021-09-13T12:30:46] DEBUG PID-19627 TID-62280 ActiveRecord::Base:    (2.7ms)  COMMIT
[2021-09-13T12:30:46] DEBUG PID-19627 TID-62280 ActiveRecord::Base:   Mention Load (0.3ms)  SELECT "mentions".* FROM "mentions" WHERE "mentions"."mentions_container_id" = $1 AND "mentions"."mentions_container_type" = $2  [["mentions_container_id", 3175], ["mentions_container_type", "Post"]]
[2021-09-13T12:30:46] DEBUG PID-19627 TID-62280 ActiveRecord::Base:   Post Load (0.4ms)  SELECT  "posts".* FROM "posts" WHERE "posts"."id" = $1 LIMIT $2  [["id", 3175], ["LIMIT", 1]]
[2021-09-13T12:30:46] DEBUG PID-19627 TID-62280 ActiveRecord::Base:   OpenGraphCache Load (0.4ms)  SELECT  "open_graph_caches".* FROM "open_graph_caches" WHERE "open_graph_caches"."url" = $1 LIMIT $2  [["url", "turnoff.us"], ["LIMIT", 1]]
[2021-09-13T12:30:46] DEBUG PID-19627 TID-62280 Rails: ETHON: performed EASY effective_url=http://turnoff.us/ response_code=200 return_code=ok total_time=0.060295
[2021-09-13T12:30:46] DEBUG PID-19627 TID-62280 Rails: ETHON: performed EASY effective_url=http://turnoff.us/ response_code=200 return_code=ok total_time=0.018708
[2021-09-13T12:30:46] DEBUG PID-19627 TID-62280 ActiveRecord::Base:    (0.5ms)  BEGIN
/var/lib/gems/2.7.0/gems/activemodel-5.2.6/lib/active_model/naming.rb:190: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/var/lib/gems/2.7.0/gems/i18n-1.8.10/lib/i18n.rb:196: warning: The called method `translate' is defined here
/var/lib/gems/2.7.0/gems/activemodel-5.2.6/lib/active_model/translation.rb:67: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/var/lib/gems/2.7.0/gems/i18n-1.8.10/lib/i18n.rb:196: warning: The called method `translate' is defined here
/var/lib/gems/2.7.0/gems/activemodel-5.2.6/lib/active_model/errors.rb:430: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/var/lib/gems/2.7.0/gems/i18n-1.8.10/lib/i18n.rb:196: warning: The called method `translate' is defined here
[2021-09-13T12:30:46] DEBUG PID-19627 TID-62280 ActiveRecord::Base:    (0.6ms)  ROLLBACK
[2021-09-13T12:30:46] DEBUG PID-19627 TID-62280 ActiveRecord::Base:    (0.4ms)  BEGIN
[2021-09-13T12:30:46] DEBUG PID-19627 TID-62280 ActiveRecord::Base:   ActsAsTaggableOn::Tagging Load (0.7ms)  SELECT "taggings".* FROM "taggings" WHERE "taggings"."taggable_id" = $1 AND "taggings"."taggable_type" = $2  [["taggable_id", 3175], ["taggable_type", "Post"]]
[2021-09-13T12:30:46] DEBUG PID-19627 TID-62280 ActiveRecord::Base:   ActsAsTaggableOn::Tag Load (1.5ms)  SELECT "tags".* FROM "tags" INNER JOIN "taggings" ON "tags"."id" = "taggings"."tag_id" WHERE "taggings"."taggable_id" = $1 AND "taggings"."taggable_type" = $2 AND (taggings.context = 'tags' AND taggings.tagger_id IS NULL)  [["taggable_id", 3175], ["taggable_type", "Post"]]
[2021-09-13T12:30:46] DEBUG PID-19627 TID-62280 ActiveRecord::Base:   Post Exists (0.6ms)  SELECT  1 AS one FROM "posts" WHERE "posts"."guid" = $1 AND "posts"."id" != $2 LIMIT $3  [["guid", "b91cef80e6fa0135d3a1543d7ed6cc36"], ["id", 3175], ["LIMIT", 1]]
[2021-09-13T12:30:46] DEBUG PID-19627 TID-62280 ActiveRecord::Base:   Person Load (0.3ms)  SELECT  "people".* FROM "people" WHERE "people"."id" = $1 LIMIT $2  [["id", 3027], ["LIMIT", 1]]
[2021-09-13T12:30:46] DEBUG PID-19627 TID-62280 ActiveRecord::Base:   Post Exists (0.5ms)  SELECT  1 AS one FROM "posts" WHERE "posts"."id" = $1 AND "posts"."id" != $2 LIMIT $3  [["id", 3175], ["id", 3175], ["LIMIT", 1]]
[2021-09-13T12:30:46] DEBUG PID-19627 TID-62280 ActiveRecord::Base:   ActsAsTaggableOn::Tag Load (0.6ms)  SELECT "tags".* FROM "tags" WHERE (LOWER(name) = LOWER('turnoff') OR LOWER(name) = LOWER('turnoffus') OR LOWER(name) = LOWER('comic') OR LOWER(name) = LOWER('humour') OR LOWER(name) = LOWER('humor') OR LOWER(name) = LOWER('devhumour') OR LOWER(name) = LOWER('devhumor') OR LOWER(name) = LOWER('geek'))
[2021-09-13T12:30:46] DEBUG PID-19627 TID-62280 ActiveRecord::Base:   ActsAsTaggableOn::Tag Load (0.8ms)  SELECT "tags".* FROM "tags" INNER JOIN "taggings" ON "tags"."id" = "taggings"."tag_id" WHERE "taggings"."taggable_id" = $1 AND "taggings"."taggable_type" = $2 AND (taggings.context = 'tags' AND taggings.tagger_id IS NULL)  [["taggable_id", 3175], ["taggable_type", "Post"]]
[2021-09-13T12:30:46] DEBUG PID-19627 TID-62280 ActiveRecord::Base:   ActsAsTaggableOn::Tag Load (0.6ms)  SELECT "tags".* FROM "tags" WHERE (LOWER(name) = LOWER('turnoff') OR LOWER(name) = LOWER('turnoffus') OR LOWER(name) = LOWER('comic') OR LOWER(name) = LOWER('humour') OR LOWER(name) = LOWER('humor') OR LOWER(name) = LOWER('devhumour') OR LOWER(name) = LOWER('devhumor') OR LOWER(name) = LOWER('geek'))
[2021-09-13T12:30:46] DEBUG PID-19627 TID-62280 ActiveRecord::Base:   ActsAsTaggableOn::Tag Load (0.8ms)  SELECT "tags".* FROM "tags" INNER JOIN "taggings" ON "tags"."id" = "taggings"."tag_id" WHERE "taggings"."taggable_id" = $1 AND "taggings"."taggable_type" = $2 AND (taggings.context = 'tags' AND taggings.tagger_id IS NULL)  [["taggable_id", 3175], ["taggable_type", "Post"]]
[2021-09-13T12:30:46] DEBUG PID-19627 TID-62280 ActiveRecord::Base:    (0.3ms)  COMMIT

I also still have errors for reshare sometimes, most of the time it is NotFetchable but here I have a nil:

[2021-09-13T12:34:07] DEBUG PID-19627 TID-62280 Rails: ETHON: performed MULTI
[2021-09-13T12:34:07] DEBUG PID-19627 TID-62280 ActiveRecord::Base:   Reshare Load (1.2ms)  SELECT  "posts".* FROM "posts" WHERE "posts"."type" IN ('Reshare') AND "posts"."id" = $1 LIMIT $2  [["id", 3239], ["LIMIT", 1]]
[2021-09-13T12:34:07] ERROR PID-19627 TID-62280 DiasporaFederation::Federation::Receiver::Public: failed to receive Reshare:7b83c190a644013633f12a0000053625:
[2021-09-13T12:34:07] ERROR PID-19627 TID-62280 DiasporaFederation::Federation::Receiver: failed to receive public message: NoMethodError: undefined method `author' for nil:NilClass
[2021-09-13T12:34:07] DEBUG PID-19627 TID-62280 DiasporaFederation::Federation::Receiver: received data:
<?xml version="1.0" encoding="UTF-8"?>
<me:env xmlns:me="http://salmon-protocol.org/ns/magic-env">
  <me:data type="application/xml">PHJlc2hhcmU-CiAgPGF1dGhvcj50aGVqZWplMjBAZnJhbWFzcGhlcmUub3JnPC9hdXRob3I-CiAgPGd1aWQ-N2I4M2MxOTBhNjQ0MDEzNjMzZjEyYTAwMDAwNTM2MjU8L2d1aWQ-CiAgPGNyZWF0ZWRfYXQ-MjAxOC0wOS0yOVQxODozNjoyMFo8L2NyZWF0ZWRfYXQ-CjwvcmVzaGFyZT4=</me:data>
  <me:encoding>base64url</me:encoding>
  <me:alg>RSA-SHA256</me:alg>
  <me:sig key_id="dGhlamVqZTIwQGZyYW1hc3BoZXJlLm9yZw==">EPJCUr7cYxjpVOI_odl6CzbilO31usD2Bb5xOvnVGI22eU5OOROXQocl7emeRm8KzhoX0GBrhvUL_LadvEVTwD_gb47c4gpJY-sDs2ZmwNhIUVM3wxT2mNU-LJwYpbB-DxX4tPeLcuAq7Ws13Ese_fqMx4-8HMeIjv_1V6433bZ6scwml50BUkD_np7pMefpUw0Z5475w0U-9Q329SvEPDazJcAklYV8knEjGhY2FdB8lY-_A5MtYbfap7ik0WqPZY0he0FUudAVVfEQWimj0ooz6t3u2YGDES4HisrE6ed1O3rhHJOkL2RtvOlYKVA4FzlVkYpkZ3q7RZC8JuzB-xZ2PmrKwi98SM9W_zpS_xBDJLuXnB9gA3h3JF9oUjC2YxXqFbvs_6U2iHlLKfa8EmTqPJBQ-lR0PGx0FceR8H2Ngsd_hFB-XoTuKj04ccyMxZqcHuDAY7E2KEq-HBrKnjE7i1FuvNt8lMefw1ZH1ajBgO153cNRFb19kbC2ByjapCgHb0qjDF7HdHa5UEngaPhhd_gYpSTqm_tg4C-nwKXEk5CwxW3KE-QDQ5vz94v1TUA2-zAzDpQL-h8pKwm9b8YZagFRQ1zlss5o7czHUfbSrYKRuudDz0_1cLVg902OKZzVGYAKZZYUXsF45PKH27RF58A2iJT4IydwhCFzKsg=</me:sig>
</me:env>

[flaburgan]

We may have a (bad) answer to that question:

Here, I’m importing a user who has in his posts a post by a user who I imported before him. That created an invalid GUID and an non catch error. Well, it may be only because obviously the import was not run for real (framasphere is still answering when we ask about the first user, and locally we have something different) so maybe that won’t be a problem in production. I don’t know.

[2021-09-13T15:35:15] DEBUG PID-31419 TID-62280 ActiveRecord::Base:   StatusMessage Update All (1.2ms)  UPDATE "posts" SET "comments_count" = 2 WHERE "posts"."type" IN ('StatusMessage') AND "posts"."id" = $1  [["id", 3104]]
[2021-09-13T15:35:15] DEBUG PID-31419 TID-62280 ActiveRecord::Base:    (0.2ms)  BEGIN
[2021-09-13T15:35:15] DEBUG PID-31419 TID-62280 ActiveRecord::Base:   StatusMessage Update (0.4ms)  UPDATE "posts" SET "updated_at" = $1, "interacted_at" = $2 WHERE "posts"."id" = $3  [["updated_at", "2021-09-13 13:35:15.958933"], ["interacted_at", "2021-09-13 13:35:15.958933"], ["id", 3104]]
[2021-09-13T15:35:15] DEBUG PID-31419 TID-62280 ActiveRecord::Base:    (0.9ms)  COMMIT
[2021-09-13T15:35:15] DEBUG PID-31419 TID-62280 ActiveRecord::Base:   Post Load (0.2ms)  SELECT  "posts".* FROM "posts" WHERE "posts"."guid" = $1 LIMIT $2  [["guid", "ce97fb906c3b0134b85e2a0000053625"], ["LIMIT", 1]]
[2021-09-13T15:35:15] DEBUG PID-31419 TID-62280 ActiveRecord::Base:   Person Load (0.2ms)  SELECT  "people".* FROM "people" WHERE "people"."id" = $1 LIMIT $2  [["id", 1127], ["LIMIT", 1]]
[2021-09-13T15:35:15] DEBUG PID-31419 TID-62280 DiasporaFederation::Entities::RelatedEntity: create entity DiasporaFederation::Entities::RelatedEntity with data: {:author=>"les_dessins_de_peha@framasphere.org", :local=>false, :public=>true, :parent=>nil}
[2021-09-13T15:35:15] DEBUG PID-31419 TID-62280 DiasporaFederation::Entities::Comment: create entity DiasporaFederation::Entities::Comment with data: {:author=>"hucste@framasphere.org", :guid=>"0a625a106c430134b85e2a0000053625", :parent_guid=>"ce97fb906c3b0134b85e2a0000053625", :author_signature=>"M4txNEhDoY+1tkNiMTVZxuwCvkiHH9UTkMPdKaeiyOIZ9ngrdXpdKLLY6rzPhQ6HHCgyWBld0ungPcx01jQf0vqvq6KKKug9MIXr4ucetjRm0w2B6dsRyJn3Ef+AJ2M4jAxZ7U8yhvyQLZBujIqdLG/pWodh+2SP8GbXdySXcmAydn+WnQgNK8gBboQotpWFymlOCYL3l5OPAUfcIi3zQ948ktC7ZGmBWmBjRLo4wJyyJc8kOL1ODQEj2dwoGg8JB7xD2zgOX+km7OUTz7tsum6PCuEthN94tjcKb0TX1dR7KDDLCYTfVSss9k05EyU0ik2btbcrRMq5LPTw/Xi5etgAQhAc/bXqcf9se2yK7HKnUoHco4lryCxwvetr3EEzbxIcgwL0+KX095cXBoORau/xBlRhrUhXnPrPQpDAXhzjZVaE9x+Wc+Y2JdbXlZlIkv1MovxhLO7cxFHGA8HSHCID5ogzCZTPyqh6s2jydEV7dEUgwe3ojyoBEVzA50dqrExhY1G9oGsOGgiOzd9FT9mfNjmu0lPjh8MNCs1JJCF09RABhnzPeNy46j1J2Z23j9pUr0JHhiUpm3RP9D3hUkAy2poJTo4ApCJ8wZnKE7OZ5HrMhCz/ftaaKC6HMagN3pHUckrmgs/Y1Dvuvq5zDpgcWDf7uA3/bXDDoGM2ISo=", :text=>"Trop facile la \"note\" :p", :created_at=>2016-10-04 09:29:58 UTC, :parent=>#<DiasporaFederation::Entities::RelatedEntity:0x000055eddef789e8 @public=true, @author="les_dessins_de_peha@framasphere.org", @local=false>}
[2021-09-13T15:35:15] DEBUG PID-31419 TID-62280 ActiveRecord::Base:   Person Load (0.2ms)  SELECT  "people".* FROM "people" WHERE "people"."diaspora_handle" = $1 LIMIT $2  [["diaspora_handle", "hucste@framasphere.org"], ["LIMIT", 1]]
[2021-09-13T15:35:15] DEBUG PID-31419 TID-62280 ActiveRecord::Base:   Profile Load (0.2ms)  SELECT  "profiles".* FROM "profiles" WHERE "profiles"."person_id" = $1 LIMIT $2  [["person_id", 2973], ["LIMIT", 1]]
[2021-09-13T15:35:15] INFO  PID-31419 TID-62280 DiasporaFederation::Entities::Comment: event=verify_signature signature=author_signature status=valid obj=Comment:0a625a106c430134b85e2a0000053625:ce97fb906c3b0134b85e2a0000053625
[2021-09-13T15:35:15] DEBUG PID-31419 TID-62280 ActiveRecord::Base:   Person Load (0.2ms)  SELECT  "people".* FROM "people" WHERE "people"."diaspora_handle" = $1 LIMIT $2  [["diaspora_handle", "hucste@framasphere.org"], ["LIMIT", 1]]
[2021-09-13T15:35:15] DEBUG PID-31419 TID-62280 ActiveRecord::Base:   Comment Exists (0.3ms)  SELECT  1 AS one FROM "comments" WHERE "comments"."guid" = $1 AND "comments"."author_id" = $2 LIMIT $3  [["guid", "0a625a106c430134b85e2a0000053625"], ["author_id", 2973], ["LIMIT", 1]]
[2021-09-13T15:35:15] DEBUG PID-31419 TID-62280 ActiveRecord::Base:   Person Load (0.2ms)  SELECT  "people".* FROM "people" WHERE "people"."diaspora_handle" = $1 LIMIT $2  [["diaspora_handle", "hucste@framasphere.org"], ["LIMIT", 1]]
[2021-09-13T15:35:15] DEBUG PID-31419 TID-62280 ActiveRecord::Base:   Post Load (0.2ms)  SELECT  "posts".* FROM "posts" WHERE "posts"."guid" = $1 LIMIT $2  [["guid", "ce97fb906c3b0134b85e2a0000053625"], ["LIMIT", 1]]
[2021-09-13T15:35:15] DEBUG PID-31419 TID-62280 ActiveRecord::Base:   Person Load (0.2ms)  SELECT  "people".* FROM "people" WHERE "people"."id" = $1 LIMIT $2  [["id", 1127], ["LIMIT", 1]]
[2021-09-13T15:35:15] DEBUG PID-31419 TID-62280 ActiveRecord::Base:    (0.1ms)  BEGIN
[2021-09-13T15:35:15] DEBUG PID-31419 TID-62280 ActiveRecord::Base:   Comment Exists (0.3ms)  SELECT  1 AS one FROM "comments" WHERE "comments"."guid" = $1 LIMIT $2  [["guid", "0a625a106c430134b85e2a0000053625"], ["LIMIT", 1]]
[2021-09-13T15:35:15] DEBUG PID-31419 TID-62280 ActiveRecord::Base:   ActsAsTaggableOn::Tagging Load (0.2ms)  SELECT "taggings".* FROM "taggings" WHERE "taggings"."taggable_id" = $1 AND "taggings"."taggable_type" = $2  [["taggable_id", 2938], ["taggable_type", "Post"]]
[2021-09-13T15:35:15] DEBUG PID-31419 TID-62280 ActiveRecord::Base:   ActsAsTaggableOn::Tag Load (0.5ms)  SELECT "tags".* FROM "tags" INNER JOIN "taggings" ON "tags"."id" = "taggings"."tag_id" WHERE "taggings"."taggable_id" = $1 AND "taggings"."taggable_type" = $2 AND (taggings.context = 'tags' AND taggings.tagger_id IS NULL)  [["taggable_id", 2938], ["taggable_type", "Post"]]
[2021-09-13T15:35:15] DEBUG PID-31419 TID-62280 ActiveRecord::Base:   Post Exists (0.3ms)  SELECT  1 AS one FROM "posts" WHERE "posts"."guid" = $1 AND "posts"."id" != $2 LIMIT $3  [["guid", "ce97fb906c3b0134b85e2a0000053625"], ["id", 2938], ["LIMIT", 1]]
[2021-09-13T15:35:15] DEBUG PID-31419 TID-62280 ActiveRecord::Base:   Post Exists (0.3ms)  SELECT  1 AS one FROM "posts" WHERE "posts"."id" = $1 AND "posts"."id" != $2 LIMIT $3  [["id", 2938], ["id", 2938], ["LIMIT", 1]]
[2021-09-13T15:35:15] DEBUG PID-31419 TID-62280 ActiveRecord::Base:    (0.2ms)  ROLLBACK
[2021-09-13T15:35:15] DEBUG PID-31419 TID-62280 ActiveRecord::Base:   Comment Load (0.3ms)  SELECT  "comments".* FROM "comments" WHERE "comments"."guid" = $1 LIMIT $2  [["guid", "0a625a106c430134b85e2a0000053625"], ["LIMIT", 1]]
rake aborted!
Diaspora::Federation::InvalidAuthor: Comment:0a625a106c430134b85e2a0000053625: hucste@framasphere.org
/home/fla/diaspora/diaspora/lib/diaspora/federation/receive.rb:339:in `block in load_from_database'
/home/fla/diaspora/diaspora/lib/diaspora/federation/receive.rb:337:in `tap'
/home/fla/diaspora/diaspora/lib/diaspora/federation/receive.rb:337:in `load_from_database'
/home/fla/diaspora/diaspora/lib/diaspora/federation/receive.rb:315:in `rescue in ignore_existing_guid'
/home/fla/diaspora/diaspora/lib/diaspora/federation/receive.rb:312:in `ignore_existing_guid'
/home/fla/diaspora/diaspora/lib/diaspora/federation/receive.rb:266:in `save_relayable'
/home/fla/diaspora/diaspora/lib/diaspora/federation/receive.rb:261:in `receive_relayable'
/home/fla/diaspora/diaspora/lib/diaspora/federation/receive.rb:32:in `comment'
/home/fla/diaspora/diaspora/lib/diaspora/federation/receive.rb:9:in `public_send'
/home/fla/diaspora/diaspora/lib/diaspora/federation/receive.rb:9:in `perform'
/home/fla/diaspora/diaspora/lib/archive_importer/entity_importer.rb:14:in `import'
/home/fla/diaspora/diaspora/lib/archive_importer.rb:97:in `block in import_collection'
/home/fla/diaspora/diaspora/lib/archive_importer.rb:96:in `each'
/home/fla/diaspora/diaspora/lib/archive_importer.rb:96:in `import_collection'
/home/fla/diaspora/diaspora/lib/archive_importer.rb:88:in `import_others_relayables'
/home/fla/diaspora/diaspora/lib/archive_importer.rb:20:in `import'
/home/fla/diaspora/diaspora/app/services/migration_service.rb:55:in `import_archive'
/home/fla/diaspora/diaspora/app/services/migration_service.rb:23:in `perform!'
/home/fla/diaspora/diaspora/app/services/import_profile_service.rb:35:in `import_user_profile'
/home/fla/diaspora/diaspora/app/services/import_profile_service.rb:13:in `import_by_files'
/home/fla/diaspora/diaspora/lib/tasks/accounts.rake:13:in `block (2 levels) in <top (required)>'

Caused by:
ActiveRecord::RecordInvalid: Validation failed: Guid has already been taken
/var/lib/gems/2.7.0/gems/activerecord-5.2.6/lib/active_record/validations.rb:80:in `raise_validation_error'
/var/lib/gems/2.7.0/gems/activerecord-5.2.6/lib/active_record/validations.rb:52:in `save!'
/var/lib/gems/2.7.0/gems/activerecord-5.2.6/lib/active_record/transactions.rb:315:in `block in save!'
/var/lib/gems/2.7.0/gems/activerecord-5.2.6/lib/active_record/transactions.rb:387:in `block in with_transaction_returning_status'
/var/lib/gems/2.7.0/gems/activerecord-5.2.6/lib/active_record/connection_adapters/abstract/database_statements.rb:267:in `block in transaction'
/var/lib/gems/2.7.0/gems/activerecord-5.2.6/lib/active_record/connection_adapters/abstract/transaction.rb:239:in `block in within_new_transaction'
/var/lib/gems/2.7.0/gems/activerecord-5.2.6/lib/active_record/connection_adapters/abstract/transaction.rb:236:in `synchronize'
/var/lib/gems/2.7.0/gems/activerecord-5.2.6/lib/active_record/connection_adapters/abstract/transaction.rb:236:in `within_new_transaction'
/var/lib/gems/2.7.0/gems/activerecord-5.2.6/lib/active_record/connection_adapters/abstract/database_statements.rb:267:in `transaction'
/var/lib/gems/2.7.0/gems/activerecord-5.2.6/lib/active_record/transactions.rb:212:in `transaction'
/var/lib/gems/2.7.0/gems/activerecord-5.2.6/lib/active_record/transactions.rb:385:in `with_transaction_returning_status'
/var/lib/gems/2.7.0/gems/activerecord-5.2.6/lib/active_record/transactions.rb:315:in `save!'
/var/lib/gems/2.7.0/gems/activerecord-5.2.6/lib/active_record/suppressor.rb:48:in `save!'
/home/fla/diaspora/diaspora/lib/diaspora/federation/receive.rb:271:in `block (2 levels) in save_relayable'
/home/fla/diaspora/diaspora/lib/diaspora/federation/receive.rb:267:in `tap'
/home/fla/diaspora/diaspora/lib/diaspora/federation/receive.rb:267:in `block in save_relayable'
/home/fla/diaspora/diaspora/lib/diaspora/federation/receive.rb:313:in `ignore_existing_guid'
/home/fla/diaspora/diaspora/lib/diaspora/federation/receive.rb:266:in `save_relayable'
/home/fla/diaspora/diaspora/lib/diaspora/federation/receive.rb:261:in `receive_relayable'
/home/fla/diaspora/diaspora/lib/diaspora/federation/receive.rb:32:in `comment'
/home/fla/diaspora/diaspora/lib/diaspora/federation/receive.rb:9:in `public_send'
/home/fla/diaspora/diaspora/lib/diaspora/federation/receive.rb:9:in `perform'
/home/fla/diaspora/diaspora/lib/archive_importer/entity_importer.rb:14:in `import'
/home/fla/diaspora/diaspora/lib/archive_importer.rb:97:in `block in import_collection'
/home/fla/diaspora/diaspora/lib/archive_importer.rb:96:in `each'
/home/fla/diaspora/diaspora/lib/archive_importer.rb:96:in `import_collection'
/home/fla/diaspora/diaspora/lib/archive_importer.rb:88:in `import_others_relayables'
/home/fla/diaspora/diaspora/lib/archive_importer.rb:20:in `import'
/home/fla/diaspora/diaspora/app/services/migration_service.rb:55:in `import_archive'
/home/fla/diaspora/diaspora/app/services/migration_service.rb:23:in `perform!'
/home/fla/diaspora/diaspora/app/services/import_profile_service.rb:35:in `import_user_profile'
/home/fla/diaspora/diaspora/app/services/import_profile_service.rb:13:in `import_by_files'
/home/fla/diaspora/diaspora/lib/tasks/accounts.rake:13:in `block (2 levels) in <top (required)>'
Tasks: TOP => accounts:migration
(See full trace by running task with --trace)

[tclaus (Thorsten Claus)]

Profile picture import seems not to be implemented.

[tclaus (Thorsten Claus)]

Thanks for analyzing this - I’ll check this. Hope, that I can help…

[tclaus (Thorsten Claus)]

IMHO all code exists that a import is promoted to all other pods and locally all references change. (Done in account_migration.rb)
To be honest: I don’t know if this behavior changes in production.
Fla: If you can make a test on your prod-server and submit the logs from that time.?

[flaburgan]

Well I don’t want to put that code in production in d-fr right now, but we have your pod so this is enough.
To test it we need to create two accounts in diaspora-fr, make them share with each other, then export both account. Then import one, and then import the second (with the export which is older than the first import), and we will see.

[flaburgan]

So to sum up the remarks about the import backend (I may retry to import the different archives I have with the code which is now merged but I think results will be the same):

• A cert error, that should probably not make the import fails
• A decision to take about the email being already token in the pod where the import is done.
• Improving the rake task so it’s clear if the account is a merge or a creation
• A question by goob, what if the pod of the export is down during the migration, but come back up after?
• A rake aborted because of MaxRetriesReached
• Some errors which may be ignored, or not?

Should I create github issues for those points?

[tclaus (Thorsten Claus)]

I think git issues are better trackable than bullet points here

[flaburgan]

I created issues https://github.com/diaspora/diaspora/issues/8300 and https://github.com/diaspora/diaspora/issues/8301 for the cert error and the MaxRetries, I think we need to discuss the other points here.

[supertux88 (Benjamin Neff)]

I finally had the time to go through some of the messages, mentions and questions in this thread.

@goob

One thing I would like as part of a migration feature is the ability to be able to decide which contacts are migrated. Obviously with people I am sharing with, I can unshare before migrating. But it is harder with people who are sharing with me but with whom I’m not sharing. One reason for someone to migrate is to escape stalking or other harassment, so some means of doing this would be good.

This isn’t possible, both from the backend and from federation side. You can’t make someone unfollow you and the migration also isn’t a solution for that. The migration message is federated as public message with pods as recipients, and all the pods do is basically “rename” the old account to the new one (at least that’s how it looks to the users on other pods). So neither unfollowing nor make somebody unfollow you is possible during the migration. If you want to cleanup your aspects/contacts you either need to do it before or after the migration. Because the migration messages don’t contain any information about contacts.

@goob

One other question: is it possible that a pod running an older version of the software will have the export buttons for photos and data but not have the code to accept a ‘close account’ request from a remote pod when those data are migrated? Again, that’s a back-end question, not a UI question, but it occurred to me when I tested the process. If so, there’s a danger of duplicate and/or orphan accounts from people who ‘migrate’ but don’t realise that their old pod can’t process the close account request.

If a pod can’t understand the migration message yet, then the account is not migrated/closed, and the new account will just look and work like a new account, but I think most pods should have migration logic for profiles.
Photos on the other hand are a completely different topic, none of the pods handle photo migration yet, neither do other pods then link photos to the new pod, nor does the old pod cleanup the photos of the old account. And if the old pod is going offline, all images break. Fixing that will need another 0.7.x release including that logic and then most pods updating to that release before a lot of migrations are done, all accounts that are migrated before that will have broken photos if the old pods goes away.

@goob

The one contact who was ‘only sharing with me’ was not imported. I don’t know whether or not this is intentional. I had ‘ignored’ that account before exporting my data, so it might be that this makes a difference to whether or not the contact is imported. I am happy in my personal case for ‘only sharing with me’ contacts not to be transferred, as I have no interest in having followers, but for some people, who post a lot publicly and/or use diaspora like a blog, it would be a big disincentive to migrate if they would lose all their followers in the process.

I had in mind that there was a bug regarding contacts migration, but I thought that was fixed, so this would need to be checked if it’s because of the block or if it’s still a problem. Stuff like this ends up in inconsistencies, because the other people still think they follow you, but you think they don’t follow you anymore.

It is also a problem that the export uses the share relationship exactly the wrong way around (which is a completely different thing to fix, it works at the moment, but the code is very hard to understand and very easy to make mistakes because everything is flipped … but since it’s working probably not a blocker).

@goob

It would also be good to have the option whether to import ‘only sharing with me contacts’. But if that is not supported in the back-end migration code, that’ll be outside the scope of this work. It could possibly added in a second iteration.

As said, this shouldn’t be an option, it should always import the contacts exactly the way it was on the old pod, because the other pods do the same. If the importing pod does anything different they have a different state.

@goob

It would be worth looking into this to check whether or not a second account closure request is sent to the originating pod when this is done. It would be worth also suppressing the flash message in this case.

If you try to migrate the same account, it wouldn’t do anything (it would fail with an error, that the account already was migrated). I’m not sure if any errors are shown to the user already, and the flash message shouldn’t appear a second time (and it shouldn’t be scheduled a second time, if that’s really happening, even if it immediately fails anyway).

@goob

The imported account and all its linked data will be deleted from the previous pod.

The old pod will only delete what’s still remaining from the old account, after the migration is complete, so profile information, contacts and stuff attached to the old user. It will not delete posts/comments, they will just stay there as they were federated to the old pod, because there are maybe still users on the old pod wanting to see these posts.

@goob

If your old pod is offline when you migrate, your contacts and posts will still be imported, but your contacts will receive a notification that you have started sharing with them from this account.

This is also not true, if the old pod is offline, the old pod will receive the migration when it’s up again (if before the final retrie). But all other pods will just do the migration, the old pod isn’t needed for that. If any of the recipients is down for the migration, and also during the retries, then they won’t know about the migration but they also don’t receive any additional “started sharing” notifications.

@flaburgan

But telling to other pod that a user migrated isn’t instantaneous. What will happen if two users sharing with each other are migrating at the same time? With messages about migration crossing each other…

That’s a good question. Most things should work just normally, but I guess one of them will have a wrong contact in the end? But would need to have a closer look to be absolutely sure and maybe fix that.

@goob

We’d need to make it clear that this is not an option for new users to choose so that they don’t miss out on getting started hints and get confused by the migration section in the setting page, but is this a good approach?

Getting started hints are disabled during the import anyway, so it could just be a link to the import section and then if the user decides they are wrong there, they can still go back to the stream and have getting started hints, and if the user migrates/imports an account, hints will be disabled.

@flaburgan

The validation went well, but the actual migration failed, is that just because I’m running on localhost

@flaburgan

Actually if I launch it again it works, so I don’t know if the initial cert errors were only about some old pods…

That’s weird, and shouldn’t happen. A pod being down or having an invalid cert shouldn’t fail an import, this needs to be checked.

@goob

When I do this, will it:
a) fail with an error;
b) duplicate the content that was part of my earlier import;
c) recognise a previously imported account from the key and refuse to proceed with the migration;
d) recognise previously imported content and correctly skip that content?

a) It will fail, since you can’t import the same account twice. But even if your archive isn’t the most up to date, the first import should already be a lot better than you might expect. If the new pod had a connection to your old account and was receiving pods, it would also already migrate newer posts to your new accounts, even if they aren’t in the archive. The only thing you probably miss out are, if you added new contacts to your old account after the backup, you would need to add them again manually. Maybe it can be added to update contacts from a newer export in the future, but at the moment only one import from an account is supported.

@tclaus

A Close message is send out to your old pod. This signings are repeated several times for some days (weeks?) So if your old pod comes back, chances are high that it will receive the close-message.

Exactly.

@flaburgan

But if the problem is really coming from his pod, I guess we need to be resilient and not fail because of that.

You are also running things locally in dev-mode again, without a sidekiq running, where jobs which usually run asynchronous in sidekiq in parallel are inlined in the current task. Sidekiq jobs like federating to other pods are meant to fail, but this shouldn’t affect migrations/imports in productions where you have a running sidekiq.

[flaburgan]

@supertux88

@flaburgan

The validation went well, but the actual migration failed, is that just because I’m running on localhost

@flaburgan

Actually if I launch it again it works, so I don’t know if the initial cert errors were only about some old pods…

That’s weird, and shouldn’t happen. A pod being down or having an invalid cert shouldn’t fail an import, this needs to be checked.

I created an issue about this a few days ago but @tclaus actually closed it as wontfix: Import shouldn't fail when a pod certificate is invalid · Issue #8300 · diaspora/diaspora · GitHub should it be reopened?

Edit: Thorsten reopened it.