Question

I’ve been working on this for the past few hours, and after going through tons of Google results, I still can’t get it to work.

I’m writing a perl script that connects to https://web5.uottawa.ca/rezweb/search.php, selects either “Off/On-Campus Accomodation”, clicks search, and prints the results.

Here is what I have so far:

#!/user/bin/perl
use strict; use warnings;
use LWP::UserAgent;
use HTTP::Request::Common qw(POST);
use Crypt::SSLeay; #Since it's HTTPS

my $ua = LWP::UserAgent->new();
my $url = 'https://web5.uottawa.ca/rezweb/search.php';
my $formdata = ['accomodationType' => '1', 'submit' => 'Search'];

my $response = $ua->post($url, $formdata);
die "Error: ", $response->status_line . "\n"
  unless $response->content;

Unfortunately, it’s still the page source of the form–not the search results page.

Answer 1

Related extra doc: Mojo::DOM.

use strict;
use warnings;
use LWP::UserAgent;
use Mojo::DOM;

my $ua = LWP::UserAgent->new();
my $url = 'https://web5.uottawa.ca/rezweb/search.php';
my $formdata = [
                accomodationType => 1,
                maximum => 9999,
                maximumOnCampus => 9999,
                minimum => 0,
                minimumOnCampus => 0,
                "search_op[]" => 5,
                "search_op[]" => 6,
                "search_op[]" => 7,
                "search_op[]" => 8,
                "search_op[]" => 9,
                "search_op[]" => 10,
                "search_op[]" => 11,
                srch_rental_type4 => "on",
                srch_rental_type5 => "on",
                submit => "Search",
                zone => 0,
                ];

my $response = $ua->post($url, $formdata);
die "Error: ", $response->status_line, "\n"
    unless $response->content;

my $dom = Mojo::DOM->new( $response->decoded_content );
my @matches = map { $_->parent->parent }
    grep { $_->attrs("href") =~ /search\.php\?id=\d+/ }
    $dom->find("tr > td > a")->each;

for my $row ( @matches ) {
    print $row->all_text, $/;
}

__END__

157 Laurier Ave E 1 Hyman Soloway Sublet 1 bedroom 0400 01/08/2011
75 Louis Pasteur Pvt 1 Brooks Sublet 1 bedroom 0650 01/08/2011
75 Louis Pasteur Pvt 1 Brooks Sublet 1 bedroom 0750 25/07/2011

I arrived at the POST parameter list by turning off JS—their site uses it for enhancement—and inspecting the request. While this solution works, it’s fragile. Make sure that you’re not violating the site’s ToS or abusing the form by hammering it.