Home/ Questions/Q 6945113
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-27T13:23:40+00:00

I’m getting some directories queried on Google, Bing, etc. that I don’t necessarily want

  • 0

I’m getting some directories queried on Google, Bing, etc. that I don’t necessarily want the world to see. How can I prevent it from crawling these pages/directories? Also how do I remove previous entries?

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    The friendly web crawlers (Google, Bing, Yahoo, Baidu, etc.) will respect your robots.txt file. An example from the very helpful http://www.robotstxt.org/:

    User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/

    Of course, if you really want to restrict your private content, you’d be better served by using your webserver’s authentication and authorization tools or restrict access by address.

    • 0