Profile pages are, as I understand it, supposed not to be indexed by search engines & crawlers. However, this is not what’s happening.
Profile pages are being indexed by some crawlers at least (Google for example) - not the content of the pages, but the existence of the profile and its URL - and clicking on the URL takes you to the profile page, even if you’re not logged in.
If I put “Diaspora HQ” into Google, I get:
A description for this result is not available because of this site’s robots.txt – learn more.
The same happens for my own profile page.
So things are a bit confused. Either we want search engines to index profile pages properly, or we don’t want them to index them at all.
The robots.txt disallow for /u/ and /people/ seems to stop the crawlers from indexing the content of the profile pages, but not their existence and URL, and it’s easy then to view the content.
to the header of each profile page prevent the crawlers from indexing profile pages completely? If so, could it easily be coded so that this line is added to profile pages whenever they’re created?
Note: This discussion was imported from Loomio. Click here to view the original discussion.