Question

Can anybody tell me sample code to check if an url has been blocked by robots.txt?
We can specify full url or directory in the robots.txt.
Is there any helper function in Perl?

Answer 1

Check out WWW::RobotRules:

   The following methods are provided:

   $rules = WWW::RobotRules->new($robot_name)
  This is the constructor for WWW::RobotRules objects.  The first
  argument given to new() is the name of the robot.

   $rules->parse($robot_txt_url, $content, $fresh_until)
  The parse() method takes as arguments the URL that was used to
  retrieve the /robots.txt file, and the contents of the file.

   $rules->allowed($uri)
  Returns TRUE if this robot is allowed to retrieve this URL.