I am implementing a simplistic filter on how much of my site unregistered users can acces. Naturally, I want to give SEO bots free reign/access to most of the site.
I know this is simplistic. But its not worth doing anything more complicated. I need to compile a list of names of user agents I will allow, for this, I need a list of the names of the bots, starting with Googlebot (I don’t even know if that is the official spelling of Gogles web crawling bot).
Anyone I would like a link to a definite resource that gives the names of the SEO indexing bots?. I tried http://www.user-agents.org/ but the granularity is not fine enough – it appears to list every user agent ever created!
Try this list: http://www.useragentstring.com/pages/useragentstring.php?typ=Crawler
Although the combination of Google, Yahoo, Bing, Baidu, Ask, and AOL represent virtually 100% of the search engine market. So I would recommend adding the crawler user agents for just those to your filter and you really don’t need to worry about the rest.