Question

I wrote a script to check my own websites with LWP::RobotUA. I would like to avoid the frequent requests for my robots.txt.

The rules parameter for LWP::RobotUA should allow me to specify those, but I don’t qiute understand what should be passed for “allow all pages”.

my $ua = LWP::RobotUA->new(agent=>'my-robot/0.1', from=>'[e-mail]', rules=> ??? );

Answer 1

After more research, I think the intended way to supply robots rules is by subclassing WWW::RobotRules.

{
    package WWW::NoRules;
    use vars qw(@ISA);
    use WWW::RobotRules;
    @ISA = qw(WWW::RobotRules::InCore);

    sub allowed {
        return 1;
    }
}

my $ua = LWP::RobotUA->new(agent=>'my-robot/0.1', from=>'[e-mail]', rules=>WWW::NoRules->new);