Question

I’ve been offloaded some maintenance tasks on a couple of Perl scripts. One of the requirements is to download a couple of dozen files (HTTP) in parallel. I went looking on CPAN for the easiest solution and found this module called IO::Lambda::HTTP.

Unfortunately, I have absolutely no experience with functional programming (and beginner-level Perl experience), so while I see that all of the examples work as documented, I can’t really modify any of them to suit my needs.

For example, the sample that comes with the module:

#!/usr/bin/perl
# $Id: parallel.pl,v 1.7 2008/05/06 20:41:33 dk Exp $
# 
# This example fetches two pages in parallel, one with http/1.0 another with
# http/1.1 . The idea is to demonstrate three different ways of doing so, by
# using object API, and explicit and implicit loop unrolling
#

use lib qw(./lib);
use HTTP::Request;
use IO::Lambda qw(:lambda);
use IO::Lambda::HTTP qw(http_request);
use LWP::ConnCache;

my $a = HTTP::Request-> new(
  GET => "http://www.perl.com/",
);
$a-> protocol('HTTP/1.1');
$a-> headers-> header( Host => $a-> uri-> host);

my @chain = ( 
  $a, 
  HTTP::Request-> new(GET => "http://www.perl.com/"),
);

sub report
{
  my ( $result) = @_;
  if ( ref($result) and ref($result) eq 'HTTP::Response') {
    print "good:", length($result-> content), "\n";
  } else {
    print "bad:$result\n";
  }
#   print $result-> content;
}

my $style;
#$style = 'object';
#$style = 'explicit';
$style = 'implicit';

# $IO::Lambda::DEBUG++; # uncomment this to see that it indeed goes parallel

if ( $style eq 'object') {
  ## object API, all references and bindings are explicit
  sub handle {
    shift;
    report(@_);
  }
  my $master = IO::Lambda-> new;
  for ( @chain) {
    my $lambda = IO::Lambda::HTTP-> new( $_ );
    $master-> watch_lambda( $lambda, \&handle);
  }
  run IO::Lambda;
} elsif ( $style eq 'explicit') {
  #
  # Functional API, based on context() calls. context is
  # $obj and whatever arguments the current call needs, a RPN of sorts.
  # The context though is not stack in this analogy, because it stays
  # as is in the callback
  #
  # Explicit loop unrolling - we know that we have exactly 2 steps
  # It's not practical in this case, but it is when a (network) protocol
  # relies on precise series of reads and writes
  this lambda {
    context $chain[0];
    http_request \&report;
    context $chain[1];
    http_request \&report;
  };
  this-> wait;
} else {
  # implicit loop - we don't know how many states we need
  # 
  # also, use 'tail'
  this lambda {
    context map { IO::Lambda::HTTP-> new( $_, async_dns => 1 ) } @chain;
    tails { report $_ for @_ };
  };
  this-> wait;
}

Works as advertised, but I can’t for the life of me figure out how to modify either the ‘object’ or ‘implicit’ examples to be limited to N parallel instances like the following from IO::Lambda’s synopsis:

# http://search.cpan.org/~karasik/IO-Lambda/lib/IO/Lambda.pm
# crawl for all urls in parallel, but keep 10 parallel connections max
print par(10)-> wait(map { http($_) } @hosts);

Can someone show me an example of what the lambda code would look like given the above constraint (e.g limit to N instances)?

Also, what’s the best way to start learning functional programming? It seems totally alien to me.

Answer 1

There are good other options than IO::Lambda for this task, for example AnyEvent::HTTP. See this previous SO question.

Even though I’m familiar with functional programming, the above IO::Lambda sample code looks rather hard to understand to me.