Framasphere.org is going to close

It has been announced in 2019, Framasoft, the french NGO which hosts dozens of free software as a service, wants to reduce the number of them running. If you have time, read the blogpost above, it really is interesting (and as we knew how important it is, we translated it in English). One year and a half later, it is time to plan the end of framasphere, which really takes a lot of energy to maintain, mainly because of moderation (and the too simple tools provided to deal with by diaspora).

The end of Framasphere was initially planned before 2021, but I convinced the other members of Framasoft to wait for the account migration feature release. Unfortunately, it still isn’t there and they don’t want to wait anymore.

Because Framasphere is one of the biggest diaspora* pod, its end will impact the project, so I wanted to tell you the news in advance (we still have a few months), to give us time to maybe react to this.

1 Like

In a way that would wipe all that dormant accounts that populate framasphere…

Too bad the migration GUI is not done and completed.
d* is stable and solid, at the expense of a very slow progress.

How to boost up diaspora dev ?

So, the work on migration is ongoing, hopefully at least the backend part will be merged a few weeks before the 7th of October so that users can migrate to https://diaspora-fr.org

On another topic, I would like to know how I could properly close Framasphere. Is closing all accounts still on the pod at that time enough for other pods to know that they should not send anything anymore to Framasphere? Is there something else I should do?

Another question, how hard do you think it would be to redirect all users to a simple page only allowing to do the export of the data? Would that code in application.html.haml and application.mobile.haml do the trick? (I didn’t test the code, it’s just for you to have an idea)

- if user_signed_in? && request.request_uri != "/pod_closed"
  %meta{"http-equiv" => "refresh", :content => "0; url = '/pod_closed'"}/ 

Or is there something else I should be careful about? The idea is to allow both the users to still download their archive and the pod to still be able to process the Migration messages which will come through the Federation.

If you want to migrate all active users to diaspora-fr.org it would be much better to take over framasphere and keep it running. Migrating all the accounts will create a lot of load both on framasphere (creating the exports) and on your pod (importing the archives), and in the end your pod will basically be framasphere, so just take the whole database and migrate that would be a lot easier and using less resources.

If you still want to close it later, you can still do that, but then wait at least until there are more pods where active users from framasphere can migrate to (so after 0.8 is released), migrating from one huge pod to another one isn’t a good idea and doesn’t solve any problems.

I don’t know how involved you are with framasphere and if you have contacts to maybe arrange that you can take over the complete database.

You could also make use of the remove_old_users feature which closes inactive accounts, that would shrink down the database already and probably reduce the resources it needs to keep it running. For example set after_days to “30” so it will be really aggressive and only keep the most active users, and set warn_days to “10” so people have 10 days to reactivate their account, so if you activate that now, it will start removing accounts on the day it’s supposed to be shut down, so if users didn’t login by then, their account will eventually be closed. You probably also want to increase limit_removals_to_per_day a bit, since it only closes that many accounts per day, but since this also affects other pods (which receive the close messages), don’t put that too high, so maybe increase it from the default of 100 to a 1000 (it still takes a while until all accounts are closed then).

Also here, don’t close all accounts at once, since that would also affect all the other pods which then receive a ton of closing messages and need to process them. Using the remove_old_users feature is a better way of doing that.

And you don’t need to, and also can’t do anything so other pods stop sending messages. Closing accounts never reaches all pods, but if a pod goes down diaspora stops sending messages to that pod after two weeks automatically.

If you plan to keep the pod running, then do the remove_old_users thing I described above and still allow active users to continue using it would be the better idea.

But if you still want to close it down but allow people to export their data, something like you described should probably work (but I also didn’t test it). I’d probably remove everything from the routes.rb not related to the export and then describe on the 404 page that the pod was closed.

if the pod is down anyway and is meant to stay down, it doesn’t need to process migration messages. It’s only important/required to process them if it either is still up (so if active users are still allowed to use it until there are more pods where they can migrate to), or if you plan to re-open it again later (so you maybe want to continue using the database/domain in the future again). If it’s down and stays down permanently it doesn’t need to process the migrations.

But again, if keeping the server running is an option anyway, and based on your last two messages, keeping it available for active users is a better idea in my opinion, and then close it after the users were able to migrate to more different pods and not only to diaspora-fr.

It’s also a bit weird that registrations and invites are still enabled, but then the URLs just redirect to the startpage. Instead registrations and invites should just be disabled in the diaspora.yml/diaspora.toml, then diaspora also doesn’t render a “Create account” link in the header anymore which doesn’t work at the moment anyway.

Thank you for answering me. I have full control of the framasphere server so I can technically do what I want. However, Framasphere is too tied to the Framasoft association to be taken over by me. First of all, the name is linked to the association, meaning people will continue to believe that Framasoft is supporting it, putting load on the support team and creating false expectations. But even if I was able to change the domain name, people registered on the pod because they trusted Framasoft, not me as an individual. To automatically change the terms of use so that I’m the “owner” of 96k users data would send a bad signal, I really prefer to see people migrating manually, understanding that they are not hosted by Framasoft anymore.

So, for those reasons, Framasphere is going to close, and we won’t migrate data automatically. To not overflow the whole federation with thousands of messages of closing is why I was planning to “close” it for the users, but keep the pod running for some time, to allow people to continue to export their data and migrate, and then at some point to close the remaining accounts by batches (maybe 100 per day or something like that). It is still way cleaner to me than to simply disconnect the pod, other pod will be able to tell that the user accounts don’t exist anymore, when you search for it or try to mention it for example.

What I really need now is a working migration feature.

I’m not the one who did the closing of signup, I didn’t notice this. I will fix it.

We already did taking over a pod by a new owner with joindiaspora, and I think it should work equally well with framasphere if not a lot better.

First, if you already have full access to the server, you also already have access to all existing 96k users, so if you continue maintaining the pod as an individual, not much would change here. If people trusted framasoft, they trust them to not give access to the server/data to somebody framasoft doesn’t trust (if I trust a company, I trust them to only hire admins they trust, I can’t trust every admin personally, and this is a similar situation). And if you already involved into framasphere, you already have access to it anyway and people need to trust you (either direct or indirect through framasoft trusting you) anyway. Joindiaspora was taken over by a before unrelated person to that pod, if you are already involved in the pod that would be a much better situation than JD, so I don’t see a problem with that.

Second, it would probably only be temporary, until the migration/import is officially released and there are enough targets for people to migrate to. While they could also migrate again from diaspora-fr to another pod then, I think it will be a much better experience for the current still active users if they can just continue using the pod and then only need to migrate once. Migration isn’t 100% perfect (I don’t know if the problem with private-messages is fixed yet or if it can be fixed at all, but I remember jonne having some problems with it after the migration … and also otherwise migration probably isn’t perfect, there will always be pods which don’t receive the message or something, so less hops probably ends in a better result).

Then, you could also start closing accounts, and then only do the official take over after inactive accounts were closed, so it would be a lot less than 96k. But if you close 1000 a day it would still take 3 months until most of inactive accounts were closed. You could leave limit_removals_to_per_day at 100, but change the schedule for the QueueUsersForRemoval worker to once per hour instead of once per day, so it would do 100 per hour, which is probably better to spread them out anyway instead of doing only once per day. That way it would do 2400 per day, so it would only be 1.5 month to close inactive accounts.

Also, I don’t know about how much support it generates, but closing the pod would probably generate as much support as if you keep continue running it. You can add an information about the current situation (what the plan is and whom to contact if there is a problem) to the page, like there is currently the close notice … people who ignore that and still contact framasoft would also still contact framasoft if you shut down the pod with a placeholder page which also explains the situation. So I don’t think that keeping it running would end up in a lot more work for framasoft support.

I’m still pretty busy and don’t have much time in the next few weeks (but it should be better again in 3-4 weeks) … so having the migration in a state where I feel comfortable to migration 1000 accounts probably doesn’t happen before the currently planned closing date. But since there is a lot of work happening at it now and I also will have more time again in a few weeks, it’s probably doable until the end of the year (more or less) if the current speed continues.

That’s why I think, if the server continues running anyway, it’s probably a better solution for most of the still active users to just allow them to continue using it, but start closing inactive accounts. You can communicate the plan to the active users, so they know who to contact for support and also if they don’t trust you, then they can close their account manually before the official date (but if you have access to the server already anyway, they need to trust you already anyway). If support is the biggest concern, I’m sure this can be solved with clear communication, and whoever still contacts framasoft probably would also contact them if the pod is shut down.

And then a few weeks after migration/import is officially released the server can be shut down completely, because inactive users are closed by then, and active users can migrate to other pods then (and choose a new pod they trust with a much bigger selection), and the remaining can then be closed quickly since there probably aren’t many left.

While I totally agree with everything you’re saying, I honestly don’t know what to do. The closure of Framasphere has been voted by the whole association and now that all the users have been warned about the closing date, I’m not sure what I can do. To delay the closure by one week (or a bit more) telling the users that we need to polish migration first is certainly doable, but to the end of the year… They have a huge warning on every page for more than a month now saying that the pod is going to close. The consequence is that the active users already created a new account on diaspora-fr.org or elsewhere. The main advantage of the migration being to not have to re-add all your contacts, to deliver this in 3 months will be useless for them. And the inactive users well, we don’t care much about them.
So, I don’t know, I feel like now that most of the active users created new accounts, it’s too late to delay. What is the risk if the migration code is not working perfectly? If some data is missing that is acceptable I would say. If it breaks federation because something goes bad like mixing keys this is another story obviously. How do you feel about it at the moment?

You can update the warning to the current situation, telling the users that they can continue using the pod for a few more months, or switch to another pod earlier if they wish. It’s not forcing anybody to stay longer than they want, but after you wrote that you plan to keep the server running for a while anyway, why not also allow the users to still use it? It would be an offer to the users, not a force.

Keeping the pod running for a few more weeks/months wont change much about when migration will be possible, but it probably changes how well the user experience with it will be and how big the disruption for the users will be. It allows us to polish a few more things, then start testing with a few users, then probably fix a few more things, test again, and so on. Migrating all 1000 users on the day the pod shuts down is not really realistic and if something goes wrong there is no time left to fix anything, not a great experience for users. And if the migration isn’t finished enough at all then they don’t have anything?

If they already created a new account, they can either manually migrate to that, or use the migration in whatever state it is at the moment, or just keep this new account and continue using the old account until the more polished migration/import is done and then import their old account a few weeks/months later. The users are free to do whatever they want.

Same, but I thought if you plan to keep the server running, both to still allow active users to still export their data, and also to close some of the inactive accounts every day, why not allow users still to use it to also read/write posts. My initial thought were, that it’s planned to hard pull the plug and shut it down, also to save resources/cost, but if the server stays there for a few months anyway, why not keep it available to the remaining active users?

If users already created accounts, they probably already migrated some contacts manually, either that’s enough until they can import their profile, or they go back to the old account for some weeks. Or again, they are free to use the migration in whatever state the migration is. Keeping the pod available won’t change anything about the current state of the migration/import. It just gives more options to the users, and gives us the possibility to fix bugs as they are found and then import some users with an improved/fixed version of the code, instead of being forced to import 1000 users at once with the same probably not 100% stable code.

The migration was designed to never break federation for new entities, even if migration/import completely fails, so the new account will always work as a new account. But it can lead to inconsistency and broken UX, one user thinking they share with each other, but the other user/pod not knowing about that, or as said, I think there was a problem with existing private-messages with Jonne, so they were migrated on one pod, but not imported on the other (or something like that), so one user could think they can answer on that, but it never is received on the other pod, because they don’t know about that conversation (but here again, new conversations will always work, it’s only existing/migrated stuff that maybe breaks) … so stuff like that, inconsistency and bad UX for existing/migrated stuff.

Photos are also still an open topic, Thorsten added some code to import the photos in the new pod (but I didn’t have a full look at it yet), but other pods still have the old URLs, so as soon as framasphere would go down completely all posts with images will look broken on all other pods. To fix that something would need to be added to 0.7.x (so another minor release there, or we release it with 0.8.x, but then pods would need to upgrade to that before running the majority of the migrations), so they also change the URLs of the images to the new target.

So stuff like that, where a few more weeks/months could drastically improve the experience for the users. Again, I’m not forcing anybody to not migrate right now, but I feel for most users it would be better to delay the migration a few weeks/months. If we force everybody to migrate right now there will be broken stuff (hopefully nothing catastrophic, but there will be broken stuff). So that’s why I think more options for the active users would be a better solution, so users can choose when to migrate, spread over a few weeks/months with the possibility to fix/improve stuff in between.

Just to be clear, because I’m not sure I was: I never plan to write a script or anything to automatically migrate accounts from Framasphere to diaspora-fr.org. It’s actually the opposite, I want people to manually do the change so that they understand that they are switching pod and that Framasoft isn’t hosting them anymore.
This will spread the load between several days and allow me to also monitor that things are going OK and not overflow the federation with migration message. So yeah we’re talking about users manually using the UI to do their import.
Also yes, I didn’t think about it but I should actually add a message next to the import button boldly saying “this is beta software, no guarantee”.
IMO the migration code works pretty well, it’s just the photo migration part which has to be finished (and the UI but I can work on this). Maybe we can all (cc @tclaus) take a day off (on Thursday?) to really focus on it, do tests with the rake task, and then I will activate it on diaspora-fr. With people warned, they should understand and if they don’t want to take the risk now they will still be able to migrate later if they exported their account. What do you think?

But if the pod is being closed next week, we don’t have a lot of days anymore to spread it, and especially we don’t have time in between these days to fix stuff.

That doesn’t help the people if they are forced to use it (or the alternative would just be to not use it, create a new account, migrate contacts manually, close the old account and loose all old posts). If the old server is running anyway and they would still be allowed to use it a few more months, then they would actually have a choice to either volunteer as beta testers now (maybe the people who already created a new account and want to migrate now), and help finding bugs and stabilize the feature, or if they want to wait a bit longer until some bugs were fixed and then act as second round of beta-testers, or wait until maybe even more things are fixed. That way it would be spread over weeks/months, and we have actually a chance to fix stuff between imports and also users have a choice with what level of stability/beta-testing they want to use it.

For most parts of the data it works well, but there are probably still a few edge cases, and also situations where migrations happen in parallel aren’t tested a lot yet, and photos aren’t handled really at all on other pods. (see Backup and restore – account migration)

I can already tell you, I’ll not have time on that day, and as said, I’m completely booked for the next 3-4 weeks (maybe a few hours here and there, but not a lot and no guarantees).

Of course you can do it without me, but even then, there isn’t much time left and still a lot of open questions, possible bugs to check and other things to do. And I feel that it’s probably ready enough to do some testing with some accounts, but I don’t feel well importing/migrating 1000 accounts in a few days without the possibility to fix stuff in between it. I know it won’t break their new accounts irreparable, so that’s at least something, but it probably will cause inconsistencies (which are hard to find and stay for a long time or forever and that can and probably will cause bugreports in the future for weird stuff happening. Having more time to focus on a few beta-tester migrations and check if everything worked there and then fix stuff would probably help a lot with that (especially since you have access to both framasphere and diaspora-fr and can check logs/database stuff). But if 1000 users are migrated you can’t focus on anything.

That’s why I’m suggesting if the server keeps running for a while anyway, still allow the users to access the frontend instead of blocking them out. That would give the users a choice if they want to migrate now, or in 2 weeks, or in 1 or 2 months, and it will give use time to focus on single edge-cases and check what has gone wrong, and fix it and then test let more users test it. That will not end up in a better result for a lot of the still existing active users on framasphere, it will also end up in a better and more polished migration/import feature in the future. When we can focus on single beta-tester-imports, we can check what has gone wrong, fix it, and then have more users testing the fix again. If you force all users to migrate in a few days, probably a lot of the problems will be overlooked (if the import doesn’t fail, but still something went wrong), and even for the things you’ll find, you then don’t have another round of users to then test the fix again. So having time to focus will end up in a more stable import/migration feature in general.

I wouldn’t ask that if you would say you need to shut the server down at the deadline-date anyway to save resources/cost or something … but if the server literally stays running for a few more weeks/month, it just looks like a sane thing to do to still allow active users to use it while you close/cleanup inactive users in the background and while we stabilize the migration feature. It will be a better situation for us to have time to focus on fixes, it will be a better situation for the active users who maybe still want to wait for some more fixes and then migrate in a few weeks, and it will even be a better situation for future migrations unrelated to framasphere if we are able to build a more stable migration.

Sure, the final shutdown will happen eventually, but we can make everything a much smother migration if we spread if out over a few weeks/months and not only a few days. And I don’t see what speaks against it if the server is still running anyway. If support is your only concern, I’m sure we can find a solution for that (like you already have the warning box, just keep that up-to-date explaining the current information so still active users see what the plan is, whom to contact (you, or us in general in case of issues with the migration). Situations like this need good communication anyway, and I think with that we can minimize the support by framasoft to an acceptable minimum (people who don’t read the current situation in the warning box an still contact them probably would also contact them if the pod is completely down, there is probably not much we can do to guarantee zero support requests).

If you’ll close all accounts on framasphere then their old account will be closed too, so they can import it again later on their new pod, but their old posts will be gone on other pods. Also, they would then would need to migrate their contacts manually to a new pod if they don’t want to use the beta-migration yet. So again, being allowed to continue using the framasphere as long as the server is running anyway (and then migrate a few weeks later when they feel confident of using the stabilized import) would be a better option for the user.

I am still in, but time is the limiting factor.
This week I have a few days off, but I have some private tasks to do - still more time than usual.
What I see is, that photos are imported well, but someone should “tell” the other pods for the new photo location on existing posts.

How can this be done? A kind of “updated” - message should be send. the post with the photo in it, or just the photo entity.
@supertux88 do you have a coding proposal how this can be done?

For users already created a new account manually - they might import their profile using the existing UI.

The background PR has no blockers in it (I think) and I pushed a new commit fixing the review issues.
If this is can be approved (and merged) this helps making other coding tasks smaller.

OK so I discussed internally and here is the result:

  • Framasoft still wants to say that the service is closed. It means a state where the pod is still up, but users can only export their archives.
  • The pod can stay in that state for some weeks, that is fine
  • The official closure date is delayed by one week, to the 14th of October. I will probably need to take a day off to deal with this, so it will likely be pushed in production Friday the 15th of October

Then, actions on which I have full control:

  • I agree that we want to learn from migration, and also that a 0.7.16 will be needed for the photos to be migrated correctly. So I don’t think I will make the UI to migrate available to d-fr users before I actually deployed 0.7.16-pre to Framasphere. Obviously, it depends how much progress we did during the 15 days we have above us.
  • We should still be able to handle migration by batches. The first ones will be volunteers, that I will migrate with the rake task. I have roughly 10 users OK to do it, I’m sure I can find more of them if needed. This should allow us to discover most of the problems. Second batch will be making the UI to migrate available in diaspora-fr and doing another another round with volunteers (without telling everyone that the UI is available, only people who would discover it would use it, probably low usage). Third batch, officially saying that the UI is available with a message in diaspora*, active users will try the feature. At that point, with the feedback we will have from the 3 batches, the feature should be polished. So fourth batch (probably in more than one month from now), sending an e-mail to all old users of Framasphere saying that the UI is available in d-fr.

This seems like an acceptable compromise between Framasoft and diaspora* needs. I was not able to have better anyway.

A message per photo to every pod will be way too much, we would DDoS ourselves with that. This information would need to be included in the migration message, so pods can do the change in bulk. But this would need a protocol change/extension.

I’ll try to have a look at this PR again.

I still don’t have a lot of time in the next 3 weeks or so, so I also can’t do much until the 14th. I don’t know if @jhass has more time and is able to do a 0.7.16 release (if a solution for the photos is even coded, reviewed and merged until then), and then I also don’t know how many podmins can update their pods before the 14th. So I fear this will still end up in a lot of chaos and inconsistency for the still active framasphere users and be a much worse user experience for them than if they could just continue using the framasphere UI for some more time if the server is running anyway … but if they don’t want to allow that then there is not much we can do about that :man_shrugging:

If somebody has time to debug problems occurring when testing, fix them, review and merge them in between the batches.

And what should the users do between the 14th and when you push the UI to d-fr in a month? Take a social break?

The planned closing date is very impractical for me to be able to help. I would have a lot more time again in 3 to 4 weeks, but looks like that’s too late then …

Yeah I know and I’m sorry about this. I’m not the one taking the decision now. But the good thing is people don’t have to wait for the import tool, they can already create an account on diaspora-fr and do the import later. The only downside is they have to manually import some contacts. It should be acceptable.

Then you need to make sure that you don’t close any accounts of people who are currently active but don’t want to import yet, but later. Because they will be inactive if they can’t use the pod anymore, so only close accounts which are inactive for a longer time.

I don’t plan to close any accounts at the moment. This will be the last step, once everybody would have gotten the chance to migrate.