Question

I recently wrote a java crawler program that finds the video links in a web site and save in a text file. But there is a serious problem.

To prevent web page crawling, they use some method that changes the path of videos in the server. I know that they won’t dynamically change the actual path of video links. It’s too costly. However, I can come up with a guess that they encrypt the file paths with some key like session-id.

Now, I get the HTTP 410 – Gone error from the web server. Any ideas how did they prevent crawling and solutions to overcome these guys’ clever method ?

Answer 1

There’s a variety of methods that people can implement to protect their resources from theft / scraping:

• Rejection of suspicious User Agent strings
• Banning of IP addresses that have a non-typical usage pattern – including accessing too quickly, concurrently, selective downloading (if the CSS/JS are never downloaded, it’s an indication of scraping) or being too methodical in your approach (downloading each area sequentially, which a normal user wouldnt do)
• Requiring a HTTP Referrer to be set, and for that to be on a page within the site, potentially from a valid link source (e.g. if the video isn’t linked on the home page, then it may reject if the header is set as that – it may also be logged / IP blocked when it sees this)

If they have copyright claims over the information they publish (or the information isn’t otherwise in the public domain), which is implied if they are trying to prevent this sort of access; then what you are doing is likely to be illegal in most territories around the world.