Question

I was condering when I use urllib2.urlopen() does it just to header reads or does it actually bring back the entire webpage?

IE does the HTML page actually get fetch on the urlopen call or the read() call?

handle = urllib2.urlopen(url)
html = handle.read()

The reason I ask is for this workflow…

• I have a list of urls (some of them with short url services)
• I only want to read the webpage if I haven’t seen that url before
• I need to call urlopen() and use geturl() to get the final page that link goes to (after the 302 redirects) so I know if I’ve crawled it yet or not.
• I don’t want to incur the overhead of having to grab the html if I’ve already parsed that page.

thanks!

Answer 1

I just ran a test with wireshark. When I called urllib2.urlopen( ‘url-for-a-700mbyte-file’), only the headers and a few packets of body were retrieved immediately. It wasn’t until I called read() that the majority of the body came across the network. This matches what I see by reading the source code for the httplib module.

So, to answer the original question, urlopen() does not fetch the whole body over the network. It fetches the headers and usually some of the body. The rest of the body is fetched when you call read().

The partial body fetch is to be expected, because:

1. Unless you read an http response one byte at a time, there is no way to know exactly how long the incoming headers will be and therefore no way to know how many bytes to read before the body starts.

2. An http client has no control of how many bytes a server bundles into each tcp frame of a response.

In practice, since some of the body is usually fetched along with the headers, you might find that small bodies (e.g. small html pages) are fetched entirely on the urlopen() call.