Question

I have a program where I need to open many webpages and download information in them. The information, however, is in the middle of the page, and it takes a long time to get to it. Is there a way to have urllib only retrieve x lines? Or, if nothing else, don’t load the information afterwards?

I’m using Python 2.7.1 on Mac OS 10.8.2.

Answer 1

The returned object is a file-like object, and you can use .readline() to only read a partial response:

resp = urllib.urlopen(url)
for i in range(10):
    line = resp.readline()

would read only 10 lines, for example. Note that this won’t guarantee a faster response.