Partial pod-connection-issues (sidekiq)

Support Podmin Support
1

Hey there, that’s my first post, so i hope it’s fine the way i describe my issue.

I don’t think, the problem is new, but i wonder, if i participate to the solution or to the issue.

The affected pod is https://thepirate.party

Some lines of error’s:

  • Failed to fetch Post:5afae160c82a013606a2543d7ed6cc36 from ****@pod.dapor.net: DiasporaFederation::Federation::Fetcher::NotFetchable: Failed to fetch Post:bd2dfaa0c7fa01369d6d4061862b8e7b from renate@diasp.eu: ActsAsTaggableOn::DuplicateTagError: ‘demo’ has already been taken

  • Failed to fetch Post:14e37c60c82b0136046e0218b7f409ec from urba***n@diasp.org: DiasporaFederation::Federation::Fetcher::NotFetchable: Failed to fetch Post:bd8281f0c7f40136fb33101b0e8ace24 from ***@pod.geraspora.de: ActiveRecord::Deadlocked: Mysql2::Error: Deadlock found when trying to get lock; try restarting transaction: INSERT INTO posts (author_id, public, guid, type, text, created_at, updated_at) VALUES (8226, 1, ‘bd8281f0c7f40136fb33101b0e8ace24’, ‘StatusMessage’, ‘Freak Wave\r\n#photography #fotografie #travel #meer #sea #stahlratte #foto #photo #caribbean #sailing #segeln #boat #vacation #ferien #urlaub #sailboat #ship #sailingship #sailingtrip #sea #ocean #boattrip #sail #panama #colombia #cuba #myphoto #mywork #island’, ‘2018-11-11 15:31:10’, ‘2018-11-12 01:56:51’)

  • DiasporaFederation::Entities::Retraction::TargetNotFound

They are all from sidekiq, but the detailed exception is in the diaspora-logfile.

Also i tried around with ipv6, there’s still a provider-problem, but i don’t think this cause at least the deadlock.

Sometimes, i have even more, but around 80%+ of the connections working with success.

What can i improve on the setup? Or should i do a bug-report at github?

Regards

hersche@thepirate.party

2

Are you using mysql? Can you please check the encoding of your database?

3

Yes, i am using mysql. And i should read the existing stuff better - CSammy’s tutorial was nice!

I mean, it’s just patched successfull 3min ago, so there’s may something left, but until yet, i saw no single fail anymore.

If others have the same issues, this was my solution: Help migrating a MySQL database from utf8_general_ci to utf8mb4_bin

EDIT: do this only if your mysql-db-encoding is wrong and you have tag-issues like descripted in the post. Do also the test-part! (as addition to Dennis Schuberts warning)

Thanks for your answer, Flaburgan :slight_smile:

4

There is a thick fat warning above that post to never run it without a team member telling you explicitly to do so, and I hereby want to reaffirm that statement. This script is not a joke, and it might cause serious damage, or even worse, leave your pod in a state that is even less obviously broken than it was before.

5

@denschub That script isn’t a joke for shure. For me, it was still help by compare the errors.

So, a while after re-set the db-encoding, there are still some failing requests. I talk here about the dashboard-chart of sidekiq on one month-view.

Before, i had like 22’000 requests, 3500 failing, now on about same many requests, 300-800 are failing, and it’s never over 1000. So it’s definently better and the only left error is:

  • not found: Post:adf174d5495831a61cd89c9956195219

one thing i think of is the use of

url: 'https://relay.iliketoast.net/receive/public'

(in the diaspora.yml). as i read, iliketoast is a test-instance only? but it was the default… maybe it’s anyway wrong(?)

or maybe, 300-800 fails are simply usual (because of broken pod’s or something)?
would be interessting to know, how your pods are working there.