Question

I have the following code which uses the WWW::Mechanize and HTML::TableExtract modules. Everything works like a charm, except that I’m not capable of moving to the next pages. I’m trying to get a list of hotspots from http://www.wigle.net/gps/gps/main. The UserID is natty_a, the password is natty. Click on [searching], and then on Query.

My script should accept coordinates and data and bring that table. It does that, but only for the first page. It seems like I cannot move to the next pages, cannot click on the Next100 >> button.

I tried the following solutions, to no avail:

$mech_browser->post(
    'https://wigle.net/gps/gps/main/confirmquery/', [
        pagestart => $i,
        Query     => 'Next100 >>'
    ]
)

$mech_browser is a WWW::Mechanize instance and i is just the number of results to get. I used Live HTTP Headers to find it out.

$mech_browser->click_button(value => 'Next100 >>')

This does not work, either.

Answer 1

There is a very cool module, WWW::Mechanize::Shell, which allows one to interactively play with a page, click various buttons etc.
Once a satisfactory result is reached, one can generate a Perl script that uses WWW::Mechanize and that performs precisely the actions that one has performed while playing.

Then what is left is to edit out actions which are not needed from the generated script.

I’d suggest you use that to find your solution.