Question

I am looking to scrape some content from some websites for research and I was hoping that using python and web scraping might speed up my process. I have used python and beautiful soup before for one small project to convert an xml from one format to another.

Answer 1

Depending on how redundant is the structure of the data you want to extract, you could use several tools.

• If you’re looking for extracting data always stored in the same DOM structure, Scrapy could do the job.
• If the data is sparse and is stored in various places, maybe BeautfulSoup4 or lxml could help you.
• If the data is generated by some JS code, have a look at Selenium

Here are a couple of resources you might find useful:

• PyCon 2012 Tutorial about web-scraping: http://pyvideo.org/video/609/web-scraping-reliably-and-efficiently-pull-data/
• http://isbullsh.it/2012/04/Web-crawling-with-scrapy/ (full disclosure, I wrote that)
• http://www.packtpub.com/article/web-scraping-with-python
• http://wwwsearch.sourceforge.net/mechanize/