Checksum mismatch in camo

So I think I have the same issue as was addressed in this post;

Although the solution in that post did not resolve my issue. I am getting the same errors;

Nov  8 16:48:31 borg node[30760]: --------------------------------------------
Nov  8 16:48:31 borg node[30760]: [2019-11-08T23:48:31.622Z] checksum mismatch cbba088e5d07512cfe86fd31f1293efc803ed323:e0cc3afb0df00d6d8efce0706bc5d29a98c2dc7c: unknown
Nov  8 16:48:31 borg node[30760]: --------------------------------------------
Nov  8 16:48:31 borg node[30760]: {
Nov  8 16:48:31 borg node[30760]:   type: 'path',
Nov  8 16:48:31 borg node[30760]:   url: '/28a05d5211af218922e9f45c6a82330b9ee1aebf/68747470733a2f2f706c7573706f72612e636f6d2f75706c6f6164732f696d616765732f7363616c65645f66756c6c5f33643666613039336238306332626230663735342e504e47',
Nov  8 16:48:31 borg node[30760]:   headers: {
Nov  8 16:48:31 borg node[30760]:     host: 'borg.zbkbie.com',
Nov  8 16:48:31 borg node[30760]:     'user-agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:70.0) Gecko/20100101 Firefox/70.0',
Nov  8 16:48:31 borg node[30760]:     accept: 'image/webp,*/*',
Nov  8 16:48:31 borg node[30760]:     'accept-language': 'en-US,en;q=0.5',
Nov  8 16:48:31 borg node[30760]:     'accept-encoding': 'gzip, deflate, br',
Nov  8 16:48:31 borg node[30760]:     referer: 'https://borg.zbkbie.com/stream',
Nov  8 16:48:31 borg node[30760]:     x_forwarded_proto: 'https',
Nov  8 16:48:31 borg node[30760]:     via: '1.1 borg.zbkbie.com',
Nov  8 16:48:31 borg node[30760]:     'x-forwarded-for': '192.168.1.123',
Nov  8 16:48:31 borg node[30760]:     'x-forwarded-host': 'borg.zbkbie.com',
Nov  8 16:48:31 borg node[30760]:     'x-forwarded-server': 'borg.zbkbie.com',
Nov  8 16:48:31 borg node[30760]:     connection: 'Keep-Alive'
Nov  8 16:48:31 borg node[30760]:   },
Nov  8 16:48:31 borg node[30760]:   dest: 'https://pluspora.com/uploads/images/scaled_full_3d6fa093b80c2bb0f754.PNG',
Nov  8 16:48:31 borg node[30760]:   digest: '28a05d5211af218922e9f45c6a82330b9ee1aebf'

my server info;

Ubuntu 16
diaspora 0.7.12.0-p09a425cb
Postgresssql
apache2

my camo keys are identical;
systemd camo.service;

[Unit]
Description=Camo Proxy Server for Diaspora

[Service]
User=diaspora
WorkingDirectory=/home/diaspora/camo
ExecStartPre=/bin/bash --login -c 'env > /tmp/.magic-environment-file-camo'
#ExecStart=SERVICE_COMMAND
EnvironmentFile=-/tmp/.magic-environment-file-camo
#Environment=CAMO_KEY=XXXXXXXXXXXXXXXXXX
Environment=CAMO_KEY="camohatesmetodeath"
Environment=NODE_TLS_REJECT_UNAUTHORIZED=0
Environment=CAMO_LENGTH_LIMIT=10485760
#Environment=CAMO_HEADER_VIA=Camo asset Proxy at <your POD>
Environment=CAMO_HEADER_VIA="Camo asset Proxy at https://borg.zbkbie.com"
Environment=CAMO_LOGGING_ENABLED="debug"
ExecStart=/usr/bin/node /home/diaspora/camo/server.js
Restart=always
StandardInput=null
KillMode=control-group

[Install]
WantedBy=multi-user.target

diaspora.yml;

 > ## Root of your Camo installation
>       #root: "https://example.com/camo/"
>       root: "https://borg.zbkbie.com/camo/"
> 
>       ## Shared key of your Camo installation
>       #key: "example123example456example!"
>       key: "camohatesmetodeath"

my apache2 reverse proxy snippet;

> RewriteEngine On
> 
>  RewriteCond %{HTTP_HOST} !^borg\.zbkbie\.com [NC]
>  RewriteRule ^/(.*)$ https://borg\.zbkbie\.com/$1 [L,R,QSA]
> 
>  # For Camo support
>  RewriteRule ^/camo/(.*)$ balancer://camo/$1 [P,QSA,L]
>  
>  RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
>  RewriteRule ^/(.*)$ balancer://upstream%{REQUEST_URI} [P,QSA,L]
> 
>  <Proxy balancer://upstream>
>   # Recommended, using a unix socket (Requires Apache >= 2.4)
>   #BalancerMember unix:///tmp/diaspora.sock|http://
>   BalancerMember unix:///home/diaspora/diaspora/tmp/diaspora.sock|http://
>   # Alternatively let diaspora listen on a local port (Use this for Apache < 2.4)
>   # BalancerMember http://localhost:3000
>  </Proxy>
> 
>  # For Camo support
>  <Proxy balancer://camo>
>    #BalancerMember http://localhost:8081
>    BalancerMember http://127.0.0.1:8081
>  </Proxy>
> 
>  ProxyRequests Off
>  ProxyVia On  
>  ProxyPreserveHost On
>  RequestHeader set X_FORWARDED_PROTO https
> 
>  <Proxy *>
>   # Apache < 2.4
>   #Order allow,deny
>   #Allow from all
>   # Apache >= 2.4
>   Require all granted
>  </Proxy>

Any help is greatly appreciated. Please let me know what else would be helpful to see and thank you very much for your time.

Make sure CAMO_KEY matches the camo.key value in diaspora.yml, and restart both Camo and diaspora* after changing it, if needed. Also, I see in your example that you’re using a quote - that’s not needed, and the quoation markes will be, in fact, part of the key, if you do that, because bash is funny. So it probably should be

Environment=CAMO_KEY=camohatesmetodeath

in your case.

If that doesn’t help you, let me know and we can dig further!

from systemd camo.service file;

Environment=CAMO_KEY="camohatesmetodeath"

Then from /home/diaspora/diaspora/config/diaspora.yml;

key: "camohatesmetodeath"

So they do match. I am sort of stuck

Update.

To do a level set. I left camo running didn’t touch the apache2 reverse proxy and turned off all 3 of these;

proxy_markdown_images
proxy_opengraph_thumbnails
proxy_remote_pod_images

With them all off, everything comes back, user profile pics and photos. Then I turned one on at a time.

With proxy_remote_pod_images turned on I do get some checksum mismatch errors but user profile pics work and most of the photos are there.

Same with proxy_opengraph_thumbnails but a few more checksum mismatch errors. Still have all user profiles pics and most of the photos. Notice more embeded news articles missing photos.

with proxy_remote_pod_images turned on I loose everything and I am back to only local hosts photos and profile pics being served.

Something I thought of to ask. My server is behind a home router. I have port 80 & 443 forwarded to it. Do I need to open port 8081 for camo?

Again thanks for your time and any help you can offer.

So another try. I did a complete reinstall of camo and I am able to run with these two turned on and get most remote images and all the user profile pictures.

proxy_markdown_images
proxy_opengraph_thumbnails

If I turn on the third option; proxy_remote_pod_images. I lose everything.

Still getting mismatch errors but my keys are exact and match as noted above in earlier posts.

Nov 9 22:04:53 borg node[9799]: --------------------------------------------
Nov 9 22:04:53 borg node[9799]: { type: ‘path’,
Nov 9 22:04:53 borg node[9799]: url: ‘/0cfda7ee5ef5f796c763e0bbf1061360a36be802/68747470733a2f2f696d6167652e627573696e657373696e73696465722e636f6d2f3564633163666437336166643337376639653466376334323f77696474683d3132303026666f726d61743d6a706567’,
Nov 9 22:04:53 borg node[9799]: headers:
Nov 9 22:04:53 borg node[9799]: { host: ‘borg.zbkbie.com’,
Nov 9 22:04:53 borg node[9799]: ‘user-agent’: ‘Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:70.0) Gecko/20100101 Firefox/70.0’,
Nov 9 22:04:53 borg node[9799]: accept: ‘image/webp,/’,
Nov 9 22:04:53 borg node[9799]: ‘accept-language’: ‘en-US,en;q=0.5’,
Nov 9 22:04:53 borg node[9799]: ‘accept-encoding’: ‘gzip, deflate, br’,
Nov 9 22:04:53 borg node[9799]: referer: ‘https://borg.zbkbie.com/stream’,
Nov 9 22:04:53 borg node[9799]: x_forwarded_proto: ‘https’,
Nov 9 22:04:53 borg node[9799]: via: ‘1.1 borg.zbkbie.com’,
Nov 9 22:04:53 borg node[9799]: ‘x-forwarded-for’: ‘192.168.1.123’,
Nov 9 22:04:53 borg node[9799]: ‘x-forwarded-host’: ‘borg.zbkbie.com’,
Nov 9 22:04:53 borg node[9799]: ‘x-forwarded-server’: ‘borg.zbkbie.com’,
Nov 9 22:04:53 borg node[9799]: connection: ‘Keep-Alive’ },
Nov 9 22:04:53 borg node[9799]: dest: ‘https://image.businessinsider.com/5dc1cfd73afd377f9e4f7c42?width=1200&format=jpeg’,
Nov 9 22:04:53 borg node[9799]: digest: ‘0cfda7ee5ef5f796c763e0bbf1061360a36be802’ }
Nov 9 22:04:53 borg node[9799]: --------------------------------------------
Nov 9 22:04:53 borg node[9799]: [2019-11-10T05:04:53.787Z] checksum mismatch 418e47275023a1064987d0a57a765923c8904c09:0cfda7ee5ef5f796c763e0bbf1061360a36be802: unknown
Nov 9 22:05:01 borg CRON[10183]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)

Please let me know if there are any logs or configs you’d like posted as I am out of things to try at this point. Thanks for taking the time to read all of this.

I’ll say it again: do not use quotes in camo.service:

I pulled them right after my second update. Sorry I should have clarified.

diaspora.yml;

  ## Root of your Camo installation
  #root: "https://example.com/camo/"
  root: "https://borg.zbkbie.com/camo/"

  ## Shared key of your Camo installation
  #key: "example123example456example!"
  key: camohatesmetodeath

/etc/systemd/system/camo.service;

[Unit]
Description=Camo Proxy Server for Diaspora

[Service]
User=diaspora
WorkingDirectory=/home/diaspora/camo
ExecStartPre=/bin/bash --login -c ‘env > /tmp/.magic-environment-file-camo’
#ExecStart=SERVICE_COMMAND
EnvironmentFile=-/tmp/.magic-environment-file-camo
#Environment=CAMO_KEY=XXXXXXXXXXXXXXXXXX
Environment=CAMO_KEY=camohatesmetodeath
Environment=NODE_TLS_REJECT_UNAUTHORIZED=0
Environment=CAMO_LENGTH_LIMIT=10485760
Environment=CAMO_HEADER_VIA=“Camo asset Proxy at borg.zbkbie.com
Environment=CAMO_LOGGING_ENABLED=“debug”
ExecStart=/usr/bin/node /home/diaspora/camo/server.js
Restart=always
StandardInput=null
KillMode=control-group

[Install]
WantedBy=multi-user.target

Thank you for taking the time to read all this and offering advice.

#key: "example123example456example!"
 key: camohatesmetodeath

Now maybe here they are needed? I have the key in quotes in my diaspora.yml but no quotes in camo.service.

ok added quotes back to diaspora.yml and left them off camo.service;

  ## Shared key of your Camo installation
  #key: "example123example456example!"
  key: "camohatesmetodeath"

Still getting the checksum mismatch errors;

Nov 14 12:03:28 borg node[1071]: url: ‘/bcb2693b7fc8a285233d1e9abb2a2a1f87c4b6c0/68747470733a2f2f64696173706f72612e746f776e2f75706c6f6164732f696d616765732f7468756d625f736d616c6c5f30613435393063363232303766383166386164382e6a7067’,
Nov 14 12:03:28 borg node[1071]: headers:
Nov 14 12:03:28 borg node[1071]: { host: ‘borg.zbkbie.com’,
Nov 14 12:03:28 borg node[1071]: ‘user-agent’: ‘Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:70.0) Gecko/20100101 Firefox/70.0’,
Nov 14 12:03:28 borg node[1071]: accept: ‘image/webp,/’,
Nov 14 12:03:28 borg node[1071]: ‘accept-language’: ‘en-US,en;q=0.5’,
Nov 14 12:03:28 borg node[1071]: ‘accept-encoding’: ‘gzip, deflate, br’,
Nov 14 12:03:28 borg node[1071]: referer: ‘https://borg.zbkbie.com/stream’,
Nov 14 12:03:28 borg node[1071]: x_forwarded_proto: ‘https’,
Nov 14 12:03:28 borg node[1071]: via: ‘1.1 borg.zbkbie.com’,
Nov 14 12:03:28 borg node[1071]: ‘x-forwarded-for’: ‘192.168.1.106’,
Nov 14 12:03:28 borg node[1071]: ‘x-forwarded-host’: ‘borg.zbkbie.com’,
Nov 14 12:03:28 borg node[1071]: ‘x-forwarded-server’: ‘borg.zbkbie.com’,
Nov 14 12:03:28 borg node[1071]: connection: ‘Keep-Alive’ },
Nov 14 12:03:28 borg node[1071]: dest: ‘https://diaspora.town/uploads/images/thumb_small_0a4590c62207f81f8ad8.jpg’,
Nov 14 12:03:28 borg node[1071]: digest: ‘bcb2693b7fc8a285233d1e9abb2a2a1f87c4b6c0’ }
Nov 14 12:03:28 borg node[1071]: --------------------------------------------
Nov 14 12:03:28 borg node[1071]: [2019-11-14T19:03:28.654Z] Requesting from self: unknown

Thanks everyone for the help. Let me know if you’d like to see any other data

Is it also key mismatch error? Maybe Camo gets confused about something else. Did you remove all unnecessary quotes from camo.service, not just around the key?

Mine looks like this:

[Unit]
Description=Camo Proxy Server for Diaspora

[Service]
User=diaspora
WorkingDirectory=/home/diaspora/camo
ExecStartPre=/bin/bash --login -c 'env > /tmp/.magic-environment-file-camo'
EnvironmentFile=-/tmp/.magic-environment-file-camo
Environment=CAMO_KEY=adf8aeRohniasheecocheisahhahpoosiegimaijepaigahfohpiudeefeubcdoi
Environment=NODE_TLS_REJECT_UNAUTHORIZED=0
Environment=CAMO_LENGTH_LIMIT=20971520
Environment=CAMO_HEADER_VIA='Camo Asset Proxy at some.pod'
ExecStart=/usr/bin/node /home/diaspora/camo/server.js
Restart=always
StandardInput=null
KillMode=control-group

[Install]
WantedBy=multi-user.target

From your latest snippet I can see you still have some:

Environment=CAMO_HEADER_VIA=“Camo asset Proxy at borg.zbkbie.com http://borg.zbkbie.com
Environment=CAMO_LOGGING_ENABLED=“debug”

Also you can list the resulting environment file and see if everything populated as you expected.

cat /tmp/.magic-environment-file-camo

UPD: now I notice I also have quotes there, just not doublequotes and probably none are needed but it works for me. Anyway it is worth checking as the error you got last time triggers when Camo checks CAMO_HEADER_VIA.

I removed the quotes as you suggested and restarted camo.service and diaspora.target and got the same results with images. With all three camo options on I loose all non locally hosted images. With

proxy_remote_pod_images: true

Commented out, I get most images back.

current /etc/systemd/system/camo.service file;

[Unit]
Description=Camo Proxy Server for Diaspora

[Service]
User=diaspora
WorkingDirectory=/home/diaspora/camo
ExecStartPre=/bin/bash --login -c ‘env > /tmp/.magic-environment-file-camo’
#ExecStart=SERVICE_COMMAND
EnvironmentFile=-/tmp/.magic-environment-file-camo
#Environment=CAMO_KEY=XXXXXXXXXXXXXXXXXX
Environment=CAMO_KEY=camohatesmetodeath
Environment=CAMO_HOSTNAME=borg.zbkbie.com
Environment=NODE_TLS_REJECT_UNAUTHORIZED=0
#Environment=CAMO_LENGTH_LIMIT=10485760
Environment=CAMO_LENGTH_LIMIT=20971520
Environment=CAMO_HEADER_VIA=Camo asset Proxy at borg.zbkbie.com
Environment=CAMO_LOGGING_ENABLED=debug
ExecStart=/usr/bin/node /home/diaspora/camo/server.js
Restart=always
StandardInput=null
KillMode=control-group

[Install]
WantedBy=multi-user.target

Current /tmp/.magic-environment-file-camo file;

CAMO_LOGGING_ENABLED=debug
rvm_bin_path=/home/diaspora/.rvm/bin
GEM_HOME=/home/diaspora/.rvm/gems/ruby-2.4.6
SHELL=/bin/bash
CAMO_LENGTH_LIMIT=10485760
IRBRC=/home/diaspora/.rvm/rubies/ruby-2.4.6/.irbrc
MY_RUBY_HOME=/home/diaspora/.rvm/rubies/ruby-2.4.6
rvm_stored_umask=0022
USER=diaspora
rvm_path=/home/diaspora/.rvm
rvm_prefix=/home/diaspora
PATH=/home/diaspora/.rvm/gems/ruby-2.4.6@diaspora/bin:/home/diaspora/bin:/home/diaspora/.local/bin:/home/diaspora/.rvm/gems/ruby-2.4.6/bin:/home/diaspora/.rvm/gems/ruby-2.4.6@global/bin:/home/diaspora/.rvm/rubies/ruby-2.4.6/bin:/home/diaspora/.rvm/gems/ruby-2.4.6@diaspora/bin:/home/diaspora/bin:/home/diaspora/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin:/home/diaspora/.rvm/bin:/home/diaspora/.rvm/bin:/home/diaspora/.rvm/bin:/home/diaspora/.rvm/bin
rvm_loaded_flag=1
PWD=/home/diaspora/camo
LANG=en_US.UTF-8
NODE_TLS_REJECT_UNAUTHORIZED=0
rvm_version=1.29.9 (latest)
CAMO_HOSTNAME=borg.zbkbie.com
SHLVL=2
HOME=/home/diaspora
CAMO_HEADER_VIA=borg.zbkbie.com
LOGNAME=diaspora
XDG_DATA_DIRS=/usr/local/share:/usr/share:/var/lib/snapd/desktop
GEM_PATH=/home/diaspora/.rvm/gems/ruby-2.4.6:/home/diaspora/.rvm/gems/ruby-2.4.6@global
CAMO_KEY=camohatesmetodeath
RUBY_VERSION=ruby-2.4.6
rvm_user_install_flag=1
_=/usr/bin/env

Let me know if there is something else I can try or if you need more diagnostic info. Thanks again for your time on this. I very much appreciate it.

UPDATE: Checksum mismatch is now gone from /var/log/syslog but I am still getting the behavior detailed above with these messages in syslog now;

Nov 16 23:39:31 borg node[5667]: { type: ‘path’,
Nov 16 23:39:31 borg node[5667]: url: ‘/5968dd3343ac4f899bbe0209937e320f1172bc69/68747470733a2f2f70686f746f2e6461706f722e6f72672f75706c6f61642f323031392f30372f30392f32303139303730393036333930362d39323333303661392e6a7067’,
Nov 16 23:39:31 borg node[5667]: headers:
Nov 16 23:39:31 borg node[5667]: { host: ‘borg.zbkbie.com’,
Nov 16 23:39:31 borg node[5667]: ‘user-agent’: ‘Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:70.0) Gecko/20100101 Firefox/70.0’,
Nov 16 23:39:31 borg node[5667]: accept: ‘image/webp,/’,
Nov 16 23:39:31 borg node[5667]: ‘accept-language’: ‘en-US,en;q=0.5’,
Nov 16 23:39:31 borg node[5667]: ‘accept-encoding’: ‘gzip, deflate, br’,
Nov 16 23:39:31 borg node[5667]: referer: ‘https://borg.zbkbie.com/stream’,
Nov 16 23:39:31 borg node[5667]: x_forwarded_proto: ‘https’,
Nov 16 23:39:31 borg node[5667]: via: ‘1.1 borg.zbkbie.com’,
Nov 16 23:39:31 borg node[5667]: ‘x-forwarded-for’: ‘192.168.1.123’,
Nov 16 23:39:31 borg node[5667]: ‘x-forwarded-host’: ‘borg.zbkbie.com’,
Nov 16 23:39:31 borg node[5667]: ‘x-forwarded-server’: ‘borg.zbkbie.com’,
Nov 16 23:39:31 borg node[5667]: connection: ‘Keep-Alive’ },
Nov 16 23:39:31 borg node[5667]: dest: ‘https://photo.dapor.org/upload/2019/07/09/20190709063906-923306a9.jpg’,
Nov 16 23:39:31 borg node[5667]: digest: ‘5968dd3343ac4f899bbe0209937e320f1172bc69’ }
Nov 16 23:39:31 borg node[5667]: --------------------------------------------
Nov 16 23:39:31 borg node[5667]: [2019-11-17T06:39:31.255Z] Requesting from self: unknown

UPDATE #2 (RESOLVED)

When the checksum mismatch error ceased, I started seeing this in the apache2 error log;

[Sun Nov 17 01:22:49.417556 2019] [proxy:error] [pid 1734:tid 140590394050304] (111)Connection refused: AH02454: HTTP: attempt to connect to Unix domain socket /home/diaspora/diaspora/tmp/diaspora.sock () failed
[Sun Nov 17 01:22:49.417585 2019] [proxy:error] [pid 1734:tid 140590394050304] AH00959: ap_proxy_connect_backend disabling worker for () for 60s

Through reivew of the diaspora.conf file showed I had mis copyed a block for

<VirtualHost *:80>
ServerName borg.zbkbie.com
ServerAlias www.borg.zbkbie.com

Redirect Permanent / https://borg.zbkbie.com/
#RewriteEngine on
#RewriteCond %{SERVER_NAME} =borg.zbkbie.com [OR]
#RewriteCond %{SERVER_NAME} =www.borg.zbkbie.com
#RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]

<VirtualHost *:443>
ServerName borg.zbkbie.com
ServerAlias www.borg.zbkbie.com

DocumentRoot /home/diaspora/diaspora/public

RewriteEngine On

RewriteCond %{HTTP_HOST} !^borg.zbkbie.com [NC]
RewriteRule ^/(.*)$ https://borg.zbkbie.com/$1 [L,R,QSA]

When I pounded out the block after the redirect permanent line and switched 127.0.0.1 back to localhost and kicked the webserver it all started working. Even rebooted it and everything just runs now. Amazing… How a few simple errors can cause so much confusion. I am certain that these were the two mistakes I made

  1. the quotes on the keys in both diaspora.yml and camo.service.
  2. mis copying a block in the apache2 config.

I want to thank the folks that took the time to read this and offer advice and wisdom in troubleshooting these stupid mistakes I made. Thank the communitiy for being open and helpful. Again thank you all for helping me and I very much apprciate your time and the Diaspora* product.

1 Like

Thanks for sharing the solution, including your config. This will be interesting for someone in the future, I’m sure.