Question

I am using urllib2 to interact with a website that sends back multiple Set-Cookie headers. However the response header dictionary only contains one – seems the duplicate keys are overriding each other.

Is there a way to access duplicate headers with urllib2?

Answer 1

According to urllib2 docs, the .headers attribute of the result URL object is an httplib.HTTPMessage (which appears to be undocumented, at least in the Python docs).

However,

help(httplib.HTTPMessage)
...

If multiple header fields with the same name occur, they are combined
according to the rules in RFC 2616 sec 4.2:

Appending each subsequent field-value to the first, each separated
by a comma. The order in which header fields with the same field-name
are received is significant to the interpretation of the combined
field value.

So, if you access u.headers[‘Set-Cookie’], you should get one Set-Cookie header with the values separated by commas.

Indeed, this appears to be the case.

import httplib
from StringIO import StringIO

msg = \
"""Set-Cookie: Foo
Set-Cookie: Bar
Set-Cookie: Baz

This is the message"""

msg = StringIO(msg)

msg = httplib.HTTPMessage(msg)

assert msg['Set-Cookie'] == 'Foo, Bar, Baz'