Question

So far I’ve been using wget and curl to do screen scraping. Now I would like to switch to perl. What’s a good tutorial that will cover basic web programming in perl (preferably without restating the basics)? I’m talking about basic things like getting and parsing pages, submitting forms, proxies, etc.

Answer 1

Tools you will need besides Perl:

1. HTTP Live Headers (Firefox extension) or eqv. to reverse engineer Javascript requests to primitive GET / POST so you can mimic that with Mechanize or LWP, etc.
2. As already mentioned by other posters, a good headless-browser is WWW::Mechanize module.
3. I would suggest spending some time learning HTML::TreeBuilder & especially HTML::TreeBuilder::XPath and HTML::Query. the last two will become very handy when you will want to get actual data from HTML documents.
4. HTML::TableExtract is also a nice module to extract data from HTML tables when needed.

basically, using all of the above will give you the ability to crawl most sites.

Have fun crawling (-: