Question

I have used the following python scripts to crawl google scholar from python:

import urllib

filehandle = urllib.urlopen('http://www.techyupdates.blogspot.com')

for lines in filehandle.readlines():
   print lines

filehandle.close()

but I am doing it repeatedly so I am getting blocked by the site-google scholar saying:

This page appears when Google automatically detects requests coming from your computer network which appear to be in violation of the Terms of Service. The block will expire shortly after those requests stop. In the meantime, solving ….

Is there an easy way to avoid this? Any suggestions?

Answer 1

[edit]

Put some kind of throttling into your script so you lightly load Google Scholar (wait for 60s or 600s or 6000s between queries, for example).

And I do mean lightly load Google Scholar. If caching the Google Scholar results is possible, that would also reduce the Google Scholar load.

You should also look at batch processing, so you can run your crawl overnight at a steady but slow speed.

The goal is that Google Scholar should not care about your additional queries, thereby fulfilling the spirit of the ToS if not the letter. But if you can fulfill both, that would be the Right Thing to Do.