Question

I own some webspace which is registered with a University. Google has unfortunately found my CV (resume) on the site, but has mis-indexed it as a scholarly publication, which is screwing up things like citation counts on Google Scholar. I tried to upload a robots.txt into my local subdirectory. The problem is that google ignores this file, and instead uses the rules listed for the school domain.

That is, the url looks like

http://www.someschool.edu/~myusername/mycv.pdf

I have uploaded a robots.txt, which can be found here

http://www.someschool.edu/~myusername/robots.txt

And Google is ignoring it and instead using the robots.txt for the school’s domain

http://www.someschool.edu/robots.txt

How can I make Googlebot ignore my CV?

Answer 1

Sadly, robots.txt is defined to be whatever you get when you GET /robots.txt, so you can’t use it for your subdirectory.

What you can do is use the X-Robots-Tag HTTP header, if you can use custom .htaccess files. Here’s Google’s documentation on X-Robots-Tag.