Question

When I visit usatoday.com with IE, there’re cookie files automatically created in my Temporary Internet Files folder. But why doesn’t the following Perl script capture anything?

use WWW::Mechanize;
use strict;
use warnings;

my $browser = WWW::Mechanize->new();
my $response = $browser->get( 'http://www.usatoday.com' );
my $cookie_jar = $browser->cookie_jar(HTTP::Cookies->new()); 
$cookie_jar->extract_cookies( $response );
my $cookie_content = $cookie_jar->as_string;
print $cookie_content;

For some other sites like amazon.com, google.com and yahoo.com, the script works well, but at least it seems to me usatoday.com also sends cookie information to the browser, why am I having different results? Is there something I’m missing?

Any ideas? Thanks!

Answer 1

UsaToday uses Javascript to set the cookie. WWW::Mechanize does not parse or run Javascript.

If you need to crawl the site with a cookie, you could analyze http://i.usatoday.net/_common/_scripts/gel/lib/core/core.js and other JS files and determine how exactly the cookie is created, and create one yourself programmatically.