Question

How would you go about getting a completely unparsed response to a HTTPS request?

Are there any HTTP libraries that will allow you to get at the raw response, or is constructing a request manually using sockets the only way?

I’m specifically trying to see what newline characters the server is sending.

Answer 1

Example from the docs:

import socket, ssl, pprint

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

# require a certificate from the server
ssl_sock = ssl.wrap_socket(s,
                           # http://curl.haxx.se/ca/cacert.pem
                           ca_certs="cacert.pem",
                           cert_reqs=ssl.CERT_REQUIRED)

ssl_sock.connect(('www.verisign.com', 443))

print repr(ssl_sock.getpeername())
print ssl_sock.cipher()
print pprint.pformat(ssl_sock.getpeercert())

# Set a simple HTTP request -- use httplib in actual code.
ssl_sock.write("""GET / HTTP/1.0\r
Host: www.verisign.com\r\n\r\n""")

# Read a chunk of data.  Will not necessarily
# read all the data returned by the server.
data = ssl_sock.read()
print repr(data)[:79]

# note that closing the SSLSocket will also close the underlying socket
ssl_sock.close()

Output

(ip, 443)
('DHE-RSA-AES256-SHA', 'TLSv1/SSLv3', 256)
{'notAfter': 'May 25 23:59:59 2012 GMT',
 'subject': ((('1.3.6.1.4.1.311.60.2.1.3', u'US'),),
             (('1.3.6.1.4.1.311.60.2.1.2', u'Delaware'),),
             (('businessCategory', u'V1.0, Clause 5.(b)'),),
             (('serialNumber', u'2497886'),),
             (('countryName', u'US'),),
             (('postalCode', u'94043'),),
             (('stateOrProvinceName', u'California'),),
             (('localityName', u'Mountain View'),),
             (('streetAddress', u'487 East Middlefield Road'),),
             (('organizationName', u'VeriSign, Inc.'),),
             (('organizationalUnitName', u' Production Security Services'),),
             (('commonName', u'www.verisign.com'),))}
'HTTP/1.1 200 OK\r\nDate: Thu, 19 Jan 2012 01:47:31 GMT\r\nServer: Apache\r\nSe

Example with M2Crypto.