Home/ Questions/Q 6702043
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-26T07:00:23+00:00

These are the modules I have installed. use WWW::Mechanize; use XML::Simple; use LWP::Simple; use

  • 0

These are the modules I have installed.

use WWW::Mechanize;
use XML::Simple;
use LWP::Simple;
use Data::Dumper;
use Web::Scraper;
#use HTML::Grabber;

I am trying to get all links that end in ‘.com‘ up to an html tag: ‘<div class="nogo_class">Proceed No More</div>‘ creating an array from the results.

I have looked at various examples I found here and in documentation but, nothing that does this.
Nothing I can wrap my noob mind around anyhow.

So, using the modules I have installed, how can I get all links that end in ‘.com‘ up to that stopping point: ‘<div class="nogo_class">Proceed No More</div>‘ into an array?

So, later down I can get the links out with a loop or whatever.
eg $somearray[$counter];

I am really inexperienced and hope I asked the question properly. Verbose explanations in any examples will help me learn this.

Thanks for you help.

P.S. the ‘nogo_class’ is used multiple times in the page but, the ‘Proceed No More’ text only appears once in the page. ALSO, I am running Perl v5.8.8 and Grabber needs v5.10.0 minimum.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Use HTML::TokeParser::Simple to parse the document. Stop parsing a tags when you find the text “Proceed No More” in a div.nogo.

    #!/usr/bin/env perl

use warnings; use strict;
use HTML::TokeParser::Simple;
use URI;

my $p = HTML::TokeParser::Simple->new(handle => \*DATA);

my @interesting_links;

while (my $tag = $p->get_tag(qw'a div')) {
    if ($tag->is_start_tag('div')) {
        my $class = $tag->get_attr('class');
        if (defined($class) and $class eq 'nogo_class') {
            my $text = $p->get_text('/div');
            last if defined($text) and $text eq 'Proceed No More';
        }
    }
    elsif ($tag->is_start_tag('a')) {
        my $href = $tag->get_attr('href');
        next unless defined $href;
        my $uri = URI->new($href);
        my $host = $uri->host;
        next unless $host =~ /[.]com\z/;
        push @interesting_links, $href;
    }
}

print "$_\n" for @interesting_links;

__DATA__
<!DOCTYPE HTML>
<html>
<head>
<title>Test</title>
</head>
<body>

<p><a href="http://example.com/link1">Link 1</a>, <a
href="http://example.org/link2">Link 2</a> and <a
href="http://example.com/link3">Link 3</a></p>

<div class="nogo_class">Keep going man!</div>

<p><a href="http://example.com/link4">Link 4</a>, <a
href="http://example.org/link5">Link 5</a> and <a
href="http://example.net/link6">Link
6</a></p>

<div class="nogo_class">Keep going man!</div>

<div class="nogo_class">Proceed No More</div>

<p><a href="#">Link 7</a>, <a href="#">Link 8</a> and <a href="#">Link
9</a></p>


</body>
</html>
    • 0