I am trying to automate data extraction from a website and I really don’t know where to start. One of our suppliers is giving us access to some equipment logging data through a “Business Objects 11” online application. If you are not familiar with this online app, think of it as a web based report generator. The problem is that I am trying to monitor a lot of equipment and this supplier has only created a request to extract one log at a time. This request takes the equipment number, the start date and the end date… To make matters worse, we can only export to the binary Excel format since de “csv” export is broke and they refuse to fix it… hence we are limited by Excel’s 65 536 row limitation… (that amounts to 3-4 days of data recording in my case). I can’t create a new resquest as only the supplier has the necessary admin rights.
What do you think would be the most elegant way of running a lot of requests (around 800) through a web GUI ? I guess I could hardcode mouse positions, click events, and keystrokes with delays and everything… But there has to be a better way.
I read about AutoHotKey and AutoIt scripting but they seem to be limited as to what they can do on the web. Also… I am stuck with IE6… But if you know a way that involves another browser, I am still very interested in your answer.
(once I have the log files locally, extracting the data is not a problem)
There are some things you might try. If the site is a html and reports can be requested by a simple POST or GET then urlib/urlib2 and cookielib Python modules should be enough to fetch an excel document.
Then you can try this: xlrd to extract data from excel.
Also, take a look at: http://pamie.sourceforge.net/. I never tried it myself but looks promising and easy to use.