Question

Somewhat inexperienced programmer here trying to write a program to log into my courses site and download all the content (lectures homeworks etc). Obviously it is a password protected site so I have to give it that. I understand LWP::UserAgent and the likes well enough, and that I need to use credentials. What I cannot figure out is how to get to the next page. I can go to the log-in, but how does perl get the result of my log-in?

code example (i pulled out the log-info obviously):

use LWP::UserAgent;

my $ua = LWP::UserAgent->new;
my $url = 'login URL';
$ua -> credentials(
  $url,
  '',
  'user',
  'pass'
);
my $response = $ua ->get($url);
print $response->content; 

the content from response is the same content as what I would have got as if I had not passed any credentials. Obviously I’m missing something here….

Oh one other thing my own courses site does not have a unique url as far as I know.

Answer 1

You probably want to be using WWW::Mechanize, a subclass of LWP::UserAgent designed to act more like a browser, allowing you to navigate through pages of a website with cookie storage already taken care of for you.