Question

How do I take advantage of HTTP 1.1’s compression when downloading web pages using Python?

I am currently using the built-in urllib module for downloading web content. Reading through the documentation I couldn’t find any information that is indeed using compression.

Is it already built-in into urllib or is there another library that I can use?

Answer 1

httplib2 supports ‘deflate’ and ‘gzip’ compression.

Example

import httplib2
h = httplib2.Http(".cache")
resp, content = h.request("http://example.org/", "GET")

The content is decompressed as necessary.