Question

I want to prevent google from indexing pdf’s on my website.

I have modified my .htaccess file to include the following lines, as suggested by google’s webmaster tools:

<Files ~ "\.pdf$">   
    Header set X-Robots-Tag "noindex, nofollow" 
</Files>

I know that apache is running properly and reading my .htaccess file, because I can block access to the file entirely, but I cannot tell whether the above command is working.

The google webmaster tools claim that the crawlers can still see the pdfs, but they seem to be intended for only use with robots.txt. Is there a 3rd party tool (for linux) that I can use to check the meta tags with?

Answer 1

You could use wget on some of the PDFs and look at the headers:

wget -S http://host/something.pdf