We implemented a rating system on a site a while back that involves a link to a script. However, with the vast majority of ratings on the site at 3/5 and the ratings very even across 1-5 we’re beginning to suspect that search engine crawlers etc. are getting through. The urls used look like this:
http://www.thesite.com/path/to/the/page/rate?uid=abcdefghijk&value=3
When we started we add the following to our robots.txt:
User-agent: *
Disallow: /rate
Is this incorrect or are googlebot and others simply ignoring our robots.txt?
You should use POST for actions which change things as search engine usually do not submit forms. Additionally, this will prevent users who download your website recursively (e.g. with wget) from submitting tons of votes.
Depending on your site, handling voting though javascript might be a solution, too.
Regarding your robots.txt:
It has to be in the root path – i.e. http://www.thesite.com/robots.txt – and if your rating system is at /blah/rate you need to use
Disallow: /blah/rateinstead ofDisallow: /rate