I’d like to collect plane ticket prices from a certain website, for many dates and destinations. I can specify source, destination and dates on the URL, but the website fetches the data using AJAX, so the prices aren’t readily available on the page’s response. In such case I could use any programming language to get the data.
I figured this task would be better accomplished using the web browser to load each URL one after another, letting it render the page, and then I’d just look for the desired tag (using CSS selectors or JS, I guess) and save it to some file or log, and move to the next URL. Later I could review the data and find the best prices.
But I unfortunately couldn’t find any browser extension/add-on to do this task (any Linux browser is fine, Firefox and Chrome the more likely). I’m already familiar with GreaseMonkey, but it is not the kind of task he’s designed to do, but I imagine it would be a similar tool or operate in a similar manner.
Does anybody know some tool that I can use for this task? Other approaches are welcome too!
I would use cURL, check the source of pages to to see the post / get data passed to pages, and just build your own gui to display the data. You could run it off your own web server with php curl very easily and quickly.