Question

I’m storing the IP addresses with each request to see how many visitor views I received.

Dim clientIPAddress As String = Request.ServerVariables("REMOTE_ADDR")
locationsDAL.AddLocationView(locationId, "", User.Identity.Name, clientIPAddress, "website")

But I notice that this also stores the MSN/Google bots etc that crawl my site.

How can I only store the IP-addresses of non-bots, so of real visitors?

Answer 1

ok, so what I did is this:

Dim clientIPAddress As String = Request.ServerVariables("REMOTE_ADDR")
If Not CheckIfCrawler(Dns.GetHostEntry(clientIPAddress).HostName) Then
 'log view
end 

Public Shared Function CheckIfCrawler(ByVal hostname As String) As Boolean
    If hostname.Contains("googlebot") Then
        Return True
    ElseIf hostname.Contains("msnbot") Then
        Return True
    ElseIf hostname.Contains("baiduspider") Then
        Return True
    ElseIf hostname.Contains("nipple3.mail.ru") Then
        Return True
    ElseIf hostname.Contains("reverse.wowrack.com") Then
        Return True
    ElseIf hostname.Contains("crawl") Then
        Return True
    ElseIf hostname.Contains("spider") Then
        Return True
    ElseIf hostname.Contains("nipple2.mail.ru") Then
        Return True
    Else
        Return False
    End If
 End Function

This way I can exclude most bots by their hostname.
Once every 6 months I will check the DB again to see if any particular hostname has had a lot of views. I then check manually using a IP GeoDB if that hostname/IPAddress belongs to a bot and if so manually add it to the CheckIfCrawler function.