Question

I’m developing a website and am sensitive to people screen scraping my data. I’m not worried about scraping one or two pages — I’m more concerned about someone scraping thousands of pages as the aggregate of that data is much more valuable than a small percentage would be.

I can imagine strategies to block users based on heavy traffic from a single IP address, but the Tor network sets up many circuits that essentially mean a single user’s traffic appears to come from different IP addresses over time.

I know that it is possible to detect Tor traffic as when I installed Vidalia with its Firefox extension, google.com presented me with a captcha.

So, how can I detect such requests?

(My website’s in ASP.NET MVC 2, but I think any approach used here would be language independent)

Answer 1

I’m developing a website and am
sensitive to people screen scraping my
data

Forget about it. If it’s on the web and someone wants it, it will be impossible to stop them from getting it. The more restrictions you put in place, the more you’ll risk ruining user experience for legitimate users, who will hopefully be the majority of your audience. It also makes code harder to maintain.

I’ll post countermeasures to any ideas future answers propose.