Home/ Questions/Q 3610154
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-18T21:43:00+00:00

I’m writing a script in Python 2.7 which uses a urllib2.OpenerDirector instance via urllib2.build_opener()

  • 0

I’m writing a script in Python 2.7 which uses a urllib2.OpenerDirector instance via urllib2.build_opener() to take advantage of the urllib2.HTTPCookieProcessor class, because I need to store and re-send the cookies I get:

opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookielib.CookieJar()))

However, after making several requests and moving the cookies around, eventually I need to retrieve a list of URLs. I wanted to use urllib.urlretrieve() because I read it downloads the file in chunks, but I cannot because I need to carry my cookies on the request and
urllib.urlretrieve() uses a urllib.URLOpener, which doesn’t have support for cookie handlers like OpenerDirector has.

What’s the reason of this strange way of splitting functionality, and how can I achieve my goal?

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    urlretrieve is a old interface from urllib. It was there much before urllib2 came into existence. It does not have any session handling capabilities. It just downloads the files. The updated urllib2 provides much better way with the deal with sessions, passwords, proxies extra using its Handler interfaces OpenerDirector class. In order to just download the urls as files, you may just use the urlopen call of urllib2 using the same request object that you created. This will maintain the session.

    • 0