Question

I need to use Python to download a large number of URLS, but they require a password to access them (similar to systems like cpanel, for example).

Is there a way I can do this, storing the cookie?

I’d like to use urllib2 if possible.

EDIT: To clarify, it’s my website and I have the login details.

UPDATE:
OK I’m using this:

cj          = cookielib.CookieJar()
opener      = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
login_data  =   urllib.urlencode({'login_name' : username, 'password' : password})
opener.open(loginURL, login_data)
productlist = opener.open(productURL)
print productlist.read()

But it just spits out the login page again. What isn’t working?
(Variables are there, I just didn’t show you what they are for security)

Answer 1

You have to use the urllib2.HTTPCookieProcessor, like this:

import urllib2
from cookielib import CookieJar

cookiejar = CookieJar()
opener = urllib2.build_opener()
cookieproc = urllib2.HTTPCookieProcessor(cookiejar)
opener.add_handler(cookieproc)

Then you just use opener.open() to access URLs, and cookies will automatically be saved and reused in future requests.