Question

I’m getting a weird error that I can’t seem to find a solution for.

This error does not occur every time I hit this segment of code, and neither does it happen for the same iteration through the loop (it happens in a loop). If I run it enough, it doesn’t seem to encounter the error and the program executes successfully. Regardless, I’d still like to figure out why this is happening.

Here is my error, versions, trace, etc: http://dpaste.com/681658/

It seems to happen with the following line in my code:

page = urllib2.urlopen(url)

Where url is…. a URL obviously.

And do have import urllib2 in my code.

Answer 1

The BadStatusLine exception is raised when you call urllib2.urlopen(url) and the remote server responds with a status code that python cannot understand.

Assuming that you don’t control url, you can’t prevent this from happening. All you can do is catch the exception, and manage it gracefully.

from httplib import BadStatusLine

try:
    page = urllib2.urlopen(url)
    # do something with page
except BadStatusLine:
    print "could not fetch %s" % url