Question

I have been using Mechanize::Firefox in order to do some web scraping and I am currently in the process of going fully headless. I have been using this code:

my $link = [$firefox->find_link_dom(text_regex => qr/pdf/i)]->[0]->{href} . "\n";

which provides very good results in finding a .pdf file. Is there anyway I can find link doom’s based on a text_regex using a different module that does not require the browser to be shown?

EDIT

With the Mechanize::Firefox I would get back a link out of code like:

<div id="rhc" xpathLocation="noDialog">

 <div id="download" class="rhcBox_type1">
<div class="wrap">
  <ul>
    <li class="download icon"><strong>Download:</strong>
      <a href="/article/fetchObjectAttachment.action;jsessionid=CDBD1AACEEB3CF89729EB23808FD1319?uri=info%3Adoi%2F10.1371%2Fjournal.pbio.1001193&amp;representation=PDF" title="Download article PDF">PDF</a> |
      <a href="/article/citationList.action;jsessionid=CDBD1AACEEB3CF89729EB23808FD1319?articleURI=info%3Adoi%2F10.1371%2Fjournal.pbio.1001193" title="Download citations">Citation</a> |
      <a href="/article/fetchObjectAttachment.action;jsessionid=CDBD1AACEEB3CF89729EB23808FD1319?uri=info%3Adoi%2F10.1371%2Fjournal.pbio.1001193&amp;representation=XML" title="Download Article XML">XML</a>
    </li>
    <li class="print icon"><a href="#" onclick="window.print();return false;" title="Print Article"><strong>Print article</strong></a></li>
    <li class="reprint icon"><a href="https://www.odysseypress.com/onlinehost/reprint_order.php?type=A&page=0&journal=1&doi=10.1371/journal.pbio.1001193&volume=&issue=&title=Evolutionarily Conserved Linkage between Enzyme Fold, Flexibility, and Catalysis&author_name=Arvind%20Ramanathan%2C%20Pratul%20K.%20Agarwal&start_page=1&end_page=17" title="Odyssey Press">EzReprint</a> New &amp; improved!</li>
  </ul>
</div>

to download a pdf file. Any ideas why the HTML::Treebase will not get this link??

Answer 1

use strict;
use warnings;
use HTML::TreeBuilder::XPath qw(); 

my $dom = HTML::TreeBuilder::XPath->new_from_content(q(...)); # ... your html code
my @nodes = $dom->findnodes('//a[contains(@href, "pdf")]'); 

foreach my $node (@nodes) {
  my $href = $node->findvalue('@href');
  ...
}