Question

What is the easiest way (without opening a shell to curl and reading from stdin) in Perl to stream from another HTTP resource? I’m assuming here that the HTTP resource I’m reading from is a potentially infinite stream (or just really, really long)

Answer 1

HTTP::Lite‘s request method allows you to specify a callback.

The $data_callback parameter, if used, is a way to filter the data as it is received or to handle large transfers. It must be a function reference, and will be passed: a reference to the instance of the http request making the callback, a reference to the current block of data about to be added to the body, and the $cbargs parameter (which may be anything). It must return either a reference to the data to add to the body of the document, or undef.

However, looking at the source, there seems to be a bug in sub request in that it seems to ignore the passed callback. It seems safer to use set_callback:

#!/usr/bin/perl

use strict;
use warnings;

use HTTP::Lite;

my $http = HTTP::Lite->new;
$http->set_callback(\&process_http_stream);
$http->http11_mode(1);

$http->request('http://www.example.com/');

sub process_http_stream {
    my ($self, $phase, $dataref, $cbargs) = @_;
    warn $phase, "\n";
    return;
}

Output:

C:\Temp> ht
connect
content-length
done-headers
content
content-done
data
done

It looks like a callback passed to the request method is treated differently:

#!/usr/bin/perl

use strict;
use warnings;

use HTTP::Lite;

my $http = HTTP::Lite->new;
$http->http11_mode(1);

my $count = 0;
$http->request('http://www.example.com/',
    \&process_http_stream,
    \$count,
);

sub process_http_stream {
    my ($self, $data, $times) = @_;
    ++$$times;
    print "$$times====\n$$data\n===\n";
}