Question

I’m writing a web-app that uses several 3rd party web APIs, and I want to keep track of the low level request and responses for ad-hoc analysis. So I’m looking for a recipe that will get Python’s urllib2 to log all bytes transferred via HTTP. Maybe a sub-classed Handler?

Answer 1

Well, I’ve found how to setup the built-in debugging mechanism of the library:

import logging, urllib2, sys

hh = urllib2.HTTPHandler()
hsh = urllib2.HTTPSHandler()
hh.set_http_debuglevel(1)
hsh.set_http_debuglevel(1)
opener = urllib2.build_opener(hh, hsh)
logger = logging.getLogger()
logger.addHandler(logging.StreamHandler(sys.stdout))
logger.setLevel(logging.NOTSET)

But I’m still looking for a way to dump all the information transferred.