Question

I am using LWP to download content from web pages, and I would like to limit the amount of time it waits for a page. This is accomplished in LWP like this:

my $ua = LWP::UserAgent->new;
$ua->timeout(10);
$ua->get($url);

And this works fine, except for whenever the timeout reaches its limit, it just dies, and I can’t continue on with the script. I’d really like to handle this timeout properly so that I can record that the URL had a timeout and then move on to my next one. Does anyone know how to do this?

Answer 1

LWP::Agent‘s get() returns a HTTP::Response object that you can use for checking errors:

use LWP::Agent;
use HTTP::Status ();

my $ua = LWP::UserAgent->new;
$ua->timeout(10);
my $response = $ua->get($url);

if ($response->is_error) {
    printf "[%d] %s\n", $response->code, $response->message;

    # record the timeout
    if ($response->code == HTTP::Status::HTTP_REQUEST_TIMEOUT) {
        ...
    }
}

Btw, the better practice nowadays is to use Try::Tiny instead of eval {...}. It gives you try {...} catch {...}. and it resolves some problems with checking if $@ (see the background section in the Try::Tiny documentation).