Question

When I try to post data from http to https, urllib2 does not return desired https webpage instead website asks to enable cookies.

To get first http page:

    proxyHandler = urllib2.ProxyHandler({'http': "http://proxy:port" })
    opener = urllib2.build_opener(proxyHandler)
    opener.addheaders = [('User-agent', 'Mozilla/5.0 (Windows NT 6.1; rv:8.0) Gecko/20100101 Firefox/8.0')]
    urllib2.install_opener(opener)
    resp = urllib2.urlopen(url)
    content = resp.read()

When I extract data from above page and post data to second https page, urllib2 returns success status 200 and page asks to enable cookies.

I’ve checked the post data, its fine. I’m getting cookies from website but not sure whether they are being sent with next request or not as I read in python docs that urllib2 automatically handles cookies.

To get second https page:

    resp = urllib2.urlopen(url, data=postData)
    content = resp.read()

I also tried to set proxy handler to this as read in a reply to similar problem on stackoverflow somewhere but got same result:

    proxyHandler = urllib2.ProxyHandler({'https': "http://proxy:port" })

Answer 1

urllib2 “handles” cookies in responses but it doesn’t not automatically store them and resend them with later requests. You’ll need to use the cooklib module for that.

There are some examples in the documentation that show how it works with urllib2.