Question

Last year I made an Android application that scrapped the informations on my train company in Belgium ( application is BETrains: http://www.cyrket.com/p/android/tof.cv.mpp/)

This application was really cool and allowed users to talk with other people in the train ( a messagery server is runned by me) and the conversations wre also on Twitter: http://twitter.com/betrains

Everybody in Belgium loved it. The company tried to avoid us to use their data, make some users websites closed, but their was some lawyers that attack the company and finally we have no more problems and the websites are open: http://blog.tuinslak.org/2010/07/irail-is-back

So, legally my application is ( for now) totally correct and legal, but I get no help from the train company.

So my question is a little help to get the datas. I am now an android/java beginner and spend some weeks to try to find a solution, but maybe people like will fint it in a few minuts.

So the problem is the next one. You may have a look at the following URL, and you will find 2 cities names within URL: Mons and Tournai, and also informations on the date and time. That was the old method that worked one year:

http://hari.b-holding.be/Hafas/bin/query.exe/en?&REQ0JourneyStopsS0A=1&REQ0JourneyStopsS0G=MONS%20%5Bb%5D&REQ0JourneyStopsZ0A=1&REQ0JourneyStopsZ0G=TOURNAI%20%5Bb%5D&REQ0JourneyDate=27.010.10&REQ0JourneyTime=19:030&Timesel=depart&ViaName=&ViaMode=NEE&DateMode=ANDERS&PLANNER=TRUE&start=1&queryPageDisplayed=yes

But now, the URL bring me on a confirmation page and I have to click on the confirm button to get to the next page.

So my code won’t work anymore, I need to click on this button programmatically to arrive on the correct webpage.

Have you any idea on how to simulate a click on this button? For now my code is the classic scrapping code with the URL given a few line on the top. I assumed that the Url give me the result page. That was the case till last week.

DefaultHttpClient httpclient = new DefaultHttpClient();
        HttpGet get = new HttpGet(mon_url);
        HttpResponse response;
            try {
                  response = httpclient.execute(get);
                  HttpEntity entity = response.getEntity();        
                  BufferedReader buf = new BufferedReader(new InputStreamReader(entity.getContent()));
                 etc...

Have you any idea on how to improve the code?

As the software is free, I cannot send paypal money, but a whole country would be really thankfull to the man that might help!

Thank a lot.

Answer 1

Instead of trying to automate clicking the JavaScript button, try monitoring what request is sent and then replicate this in your app. There are various firefox extensions that will help you do this, such as TamperData, Firebug, and LiveHttp.