Question

I have an application that logs clicks by users. The problem is, these clicks are being pushed through Twitter, which shortens every single link with t.co. Because of this, Twitter appears to hit the link between 7-15 times from different IPs, probably to do things like logging and SPAM protection. The issue is that this logs 7-15 “clicks” on my app that didn’t come from actual users.

I am wondering if there is a way to detect if a visit is coming from an actual user or is simply being cURL’d or something of the sort from a bot or spider.

The one method that seemed it could have worked was using http://www.projecthoneypot.org/ ‘s API to see if the IPs hitting my site are coming from known bots. I found a gem to help (http://cl.ly/GlT8) but kept getting a NET DNS error while trying to use it.

I am fresh out of ideas. Would really appreciate any assistance!

Answer 1

Twitter should set its User-Agent: http header properly so you can filter those out. This can be forged of course but it’s a start.

You can obtain the header in rails with request.headers["User-Agent"].