Question

the normal behavior of urllib/urllib2 is if an error code is sent in the header of the response (i.e 404) an Exception is raised.

How do you look for specific errors i.e (40x, or 50x) based on the different errors, do different things. Also, how do you read the actual data being returned HTML/JSON etc (The data usually has error details which is different to the HTML error code)

Answer 1

urllib2 raises a HTTPError when HTTP errors happen. You can get to the response code using code on the exception object. You can get the response data using read():

>>> req = urllib2.Request('http://www.python.org/fish.html')
>>> try:
>>>     urllib2.urlopen(req)
>>> except urllib2.HTTPError, e:
>>>     print e.code
>>>     print e.read()
>>>
404
<actual data response will be here>