Home/ Questions/Q 704561
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-14T03:57:20+00:00

After reading and commenting on this question PHP Library for Keeping your site index

  • 0

After reading and commenting on this question PHP Library for Keeping your site index by Google, Bing, etc, I was curious to look at StackOverFlow’s sitemap. This returned a 404 error which I am guessing is just a protected page by determining if your are a Index Bot or simply doesnt exists.

This then lead me to look at the robots.txt for StackOverFlow. I was surprised to see the comment “Yahoo bot is evil” along with a couple other Indexing bots (Spinn3r and KSCrawler) . I am unfamilular with Spinn3r and KSCrawler but my question is, why are these bots (particular Yahoo) considered as evil? Surely any and all indexing of any Search Engine is a good thing?

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    These are very aggressive bots that do not throttle requests and can bring a site down.

    See this forum thread for an example for the yahoo bot.

    • 0