Question

I have a site where users can enter their profile and password-protect certain details. I would like search engines to crawl the ‘unprotected’ parts of the profile (which varies from user to user). Similar to how if you enter a user’s name in facebook, their Facebook profile comes up in the search results. Do I have to do anything special to ensure that the bot doesn’t crawl the password-protected sections, but still crawls the (always-public) username?
I’m not sure if this is even an issue, but I’d like to update my robots.txt to allow for this.

Also, how do I ensure that the usernames are available to the bots (in a safe manner)? Do I have to create a separate directory with a list of names, or is there a better way?

Thanks for any advice

Answer 1

The search engines will only index what an anonymous user sees. If you don’t already, I would create a listing page to browse the user profiles in which you only show the data you want to. This ensures that a link exists for every userProfile.aspx?uid=XXXXXX you have. The search engine spiders will not be able to see any data that is behind a password protected.

I would also add a site map to ensure the search engine spiders get to the listing page. Don’t assume that Google will magically find ALL of your pages though typically they do based on links to your content. Submit a site map to Google.

Edit regarding Site Maps and Search Results
In order for spiders to crawl search results, I would specify an entry in the site map that points spiders to the search results page that displays all (e.g. search.aspx?param=all).