Question

I am writing a crawler to log all cookies being deployed by a set number of sites. I can pick up 1st party cookies being set on page visit using selenium, but a limitation in the software means that it won’t pick up 3rd party cookies. Are there any other tools which are available which can do pick all cookies?

Thanks.

Answer 1

If you are doing this as a one-time task, you can use something like the FireCookie extension to the Firefox browser, which lets you export all the cookies:
http://www.softwareishard.com/blog/firecookie/

If you want to automate this task and run it periodically, consider a solution like the following:

1. First get a list of pages that need to be crawled.
2. Then load each page consecutively into a web browser. It’s not enough to simply fetch the HTML of the page, because you need to load and process all javascript, iframes, and so forth that might set cookies. It could probably be a headless browser such as PhantomJS ( http://www.phantomjs.org/ ) or some other solution as long as it actually renders the page like a browser would do.
3. Use a web proxy such as Charles proxy ( http://www.charlesproxy.com/ ) to record all the network requests from the browser. The recorded session can be saved and processed to extract all the cookie headers. Charles proxy has an API that can be used to export the session to an XML file, so you might be able to automate this part as well.