Question

If I want to protect a folder from being crawled by robots that respect standards I can disallow it from robots.txt.

Now, the problem I get is that by hiding a folder, I am showing its existence to others.

So, do I have to specify a folder I do not want crawled in robots.txt if there are no links to it? “Good” crawlers only follow links right, they don’t search for folders and files randomly.

Thank you.

Answer 1

Since the Disallow lines in robots.txt are prefixes, you could just mention a prefix to your “hidden” directory that it doesn’t share with any “public” directories.

So if your “hidden” directory is called /topsecrete_donotread/ then you could use Disallow: /tops to avoid it being crawled.