Question

Using Perl, I’m looking for a simple way to perform a handful of HTTP requests in parallel, where I get responses back in the same order I sent them after they complete, e.g.:

my ($google, $perl) = foobar(GET => 'http://www.google.com/',
                             GET => 'http://www.perl.org/');

Is there a module I should be looking at?

I know I can do the bookkeeping by hand, but I feel spoiled after being able to do this using jQuery’s when method, and I’d love to have as simple a solution using Perl.

Thanks for your help.

Answer 1

use threads;
use LWP::UserAgent qw( );

my $ua = LWP::UserAgent->new();
my @threads;
for my $url ('http://www.google.com/', 'http://www.perl.org/') {
   push @threads, async { $ua->get($url) };
}

for my $thread (@threads) {
   my $response = $thread->join;
   ...
}

The best part is that the parent doesn’t wait for all requests to be completed. As soon as the right request is completed, the parent will unblock to process it.

---

If you used Parallel::ForkManager or something else where you can’t wait for a specific child, you can use the following code to order the results:

for my $id (0..$#urls) {
   create_task($id, $urls[$id]);
}

my %responses;
for my $id (0..$#urls) {
   if (!exists($responses{$id})) {
      my ($id, $response) = wait_for_a_child_to_complete();
      $responses{$id} = $response;
      redo;
   }

   my $response = delete($responses{$id});
   ...
}