Home/ Questions/Q 226633
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-11T19:31:50+00:00

I found this question very interesting : Programmatic Bot Detection I have a very

  • 0

I found this question very interesting : Programmatic Bot Detection
I have a very similar question, but I’m not bothered about ‘badly behaved bots’.

I am tracking (in addition to google analytics) the following per visit :

  • Entry URL
  • Referer
  • UserAgent
  • Adwords (by means of query string)
  • Whether or not the user made a purchase
  • etc.

The problem is that to calculate any kind of conversion rate I’m ending up with lots of ‘bot’ visits that are greatly skewing my results.

I’d like to ignore as many as possible bot visits, but I want a solution that I don’t need to monitor too closely, and that won’t in itself be a performance hog and preferably still work if someone has javascript disabled.

Are there good published lists of the top 100 bots or so? I did find a list at http://www.user-agents.org/ but that appears to contain hundreds if not thousands of bots. I don’t want to check every referer against thousands of links.

Here is the current googlebot UserAgent. How often does it change?

 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    I realized that its probably actually easier to do the exact reverse of what I was attempting.

    i.e.

    select count(*) as count, useragent from sessionvisit 
where useragent not like '%firefox%' 
and useragent not like '%chrome%'
and useragent not like '%safari%'
and useragent not like '%msie%'
and useragent not like '%gecko%'
and useragent not like '%opera%'
group by useragent order by count desc

    What I’m actually trying to do is get an accurate conversion rate, and it seems to make more sense to include good browsers rather than exclude bots (good or bad).

    In addition if i ever find a ‘session’ where a ‘robot’ has made a purchase it probably means there is a new browser (think chrome). Currently none of my robots have made a purchase!

    • 0