Question

The site that I’m trying to scrape uses js to create a cookie. What I was thinking was that I can create a cookie in python and then use that cookie to scrape the site. However, I don’t know any way of doing that. Does anybody have any ideas?

Answer 1

Please see Python httplib2 – Handling Cookies in HTTP Form Posts for an example of adding a cookie to a request.

I often need to automate tasks in web
based applications. I like to do this
at the protocol level by simulating a
real user’s interactions via HTTP.
Python comes with two built-in modules
for this: urllib (higher level Web
interface) and httplib (lower level
HTTP interface).