Question

Is there a way to grab raw, unmodified response headers from an HTTP request made with LWP? This is for a diagnostic tool that needs to identify problems with possibly malformed headers.

The closest thing I’ve found is:

use LWP::UserAgent;
my $ua = new LWP::UserAgent;
my $response = $ua->get("http://somedomain.com");
print $response->headers()->as_string();

But this actually parses the headers, and then reconstructs a canonicalized, cleaned-up version of them from the parsed data. I really need the entire header text in exactly the form in which it was returned by the server, so anything malformed or non-standard will be clearly identifiable.

If it turns out there is no way to do this with LWP, is there perhaps some other Perl module that can do this?

Answer 1

Net::HTTP provides lower level access with less processing. Since it is a subclass of IO::Socket::INET you can read directly from the object after making the request.

use Net::HTTP;

# Make the request using Net::HTTP.
my $s = Net::HTTP->new(Host => "www.perl.com") || die $@;
$s->write_request(GET => "/", 'User-Agent' => "Mozilla/5.0");

# Read the raw headers.
my @headers;
while(my $line = <$s>) {
    # Headers are done on a blank line.
    last unless $line =~ /\S/;
    push @headers, $line;
}
print @headers;