Question

I need to write a script that go to a web site, logs in, navigates to a page and downloads (and after that parse) the html of that page.

What I want is a standalone script, not a script that controls Firefox. I don’t need any javascript support in that just simple html navigation.

If nothing easy to do this exists.. well then something that acts though a web browser (firefox or safari, I’m on mac).

thanks

Answer 1

I’ve no knowledge of pre-built general purpose scrapers, but you may be able to find one via Google.

Writing a web scraper is definitely doable. In my very limited experience (I’ve written only a couple), I did not need to deal with login/security issues, but in Googling around I saw some examples that dealt with them – afraid I don’t remember URL’s for those pages. I did need to know some specifics about the pages I was scraping; having that made it easier to write the scraper, but, of course, the scrapers were limited to use on those pages. However, if you’re just grabbing the entire page, you may only need the URL(s) of the page(s) in question.

Without knowing what language(s) would be acceptable to you, it is difficult to help much more. FWIW, I’ve done scrapers in PHP and Python. As Ben G. said, PHP has cURL to help with this; maybe there are more, but I don’t know PHP very well. Python has several modules you might choose from, including lxml, BeautifulSoup, and HTMLParser.

Edit: If you’re on Unix/Linux (or, I presume, CygWin) You may be able to achieve what you want with wget.