Question

Simple code for my crawler is:

      #!/usr/bin/perl -w

      use WWW::Scripter;

      $w = new WWW::Scripter('agent' => 'myAgent');
      $w->use_plugin('JavaScript');

      ### need to set a referrer header here ###

      $w->get('http://website-url');

      print $w->content, "\n";

I need to set a referrer header before get is executed. Alternatively I will need also set other headers, such as cookie, etc. I don’t see in documentation how to do it. There has to be a way, how to set headers. How?

Answer 1

WWW::Scripter is a subclass of WWW::Mechanize, so you should be able to use the methods of that class as well. This is how it should look:

use strict;   #ALWAYS do this
use warnings; #This too. Allows more control than -w
use WWW::Scripter;

#MODULE->new() is better than new Module() because of possible parsing ambiguity
my $w = WWW::Scripter->new('agent' => 'myAgent');
$w->add_header( Referer => 'http://somesite.com' );
$w->get('http://website-url');