Question

I am using the LWP::UserAgent,
HTML::Selector::XPath and
HTML::TreeBuilder::XPath modules to get the value of the href attribute of the first YouTube video in a set of search results.

My code so far is:

use LWP::UserAgent;
use HTML::TreeBuilder::XPath;
use HTML::Selector::XPath;

my $ua = LWP::UserAgent->new;

#my $response =..
my $html = "http://www.youtube.com/results?search_query=run+flo+rida";

my $tree = HTML::TreeBuilder::XPath->new;

my $xpath = HTML::Selector::XPath::selector_to_xpath("(//*[@id = 'search-results']/li)[1]/div[2]/h3/a/@href/");
my @nodes = $tree->findnodes($xpath);
print" $nodes[0]";

I’m not sure if my printing is incorrect of if some other syntax is wrong. As of now it prints

HTML::TreeBuilder::XPath=HASH(0x1a78250)

when I am looking for it to print

/watch?v=JP68g3SYObU

Thanks for any help!

Answer 1

There are a number of problems here.

• You must always use strict and use warnings at the top of every Perl program. It will catch many errors that you would easily overlook, and is only polite when you are asking for help with your code. In this case it would have warned you that your XPath string contained array variable names @id and @href which you may not have intended to be interpolated into the string.

• You are using HTML::Selector::XPath, which translates a CSS selector to an XPath expression. But you are supplying it an XPath expression, so it will not work and the module is not needed.

• There is no need to use LWP at all, as HTML::TreeBuilder has a new_from_url constructor which will fetch the HTML page for you.

This program seems to do what you need. I have also added the URI module to derive an absolute URL from the relative one in the href attribute value.

use strict;
use warnings;

use HTML::TreeBuilder::XPath;
use URI;

my $url = "http://www.youtube.com/results?search_query=run+flo+rida";

my $tree = HTML::TreeBuilder::XPath->new_from_url($url);

my $anchor = $tree->findnodes('//ol[@id="search-results"]//h3[@class="yt-lockup2-title"]/a/@href');
my $href = URI->new_abs($anchor->[0]->getValue, $url);
print $href;

output

http://www.youtube.com/watch?v=JP68g3SYObU