Question

I am trying to parse a page like this one and I simply want to get the paragraphs after the header, the introduction I guess.

I want all the content (inclduing the paragraph tags) between <table class="infobox vcard"> and <table id="toc">. Using simple CSS selectors to get even the first paragraph:

div#bodyContent div#mw-content-text.mw-content-ltr p

does not always work because sometimes something in the infobox table has a paragraph. Also, the amount of introductory paragraphs will vary. If someone has a better approach than what I’m going for here, I will also be receptive to that.

—

Additional code requested, shortened as much as possible:

require HTTP::Request;
require LWP::UserAgent;

use LWP::Simple;
use HTML::Query 'Query';

my $pageurl = "http://en.wikipedia.org/wiki/Wayne_Rooney";
my $wikiurl = URI->new($pageurl);
my $wikirequest = HTTP::Request->new(GET => $wikiurl);
my $wikiua = LWP::UserAgent->new;
my $wikiresponse = $wikiua->request($wikirequest);
my $pagetoparse = $wikiresponse->content;

my $q2 = Query(text => $pagetoparse);
my @wikiintro = $q2->query('div#bodyContent div#mw-content-text.mw-content-ltr p')->get_elements();
my $pageintro;
if(@wikiintro) {
    if(index($wikiintro[0]->as_text(), "Appearances (Goals)") != -1){
        $pageintro = $wikiintro[1]->as_text();
    } else {
        $pageintro = $wikiintro[0]->as_text();
    }
} else {
    $pageintro = "unavailable";
}

Answer 1

One way using the non-standard module HTML::TreeBuilder.

Content of script.pl:

#!/usr/bin/env perl

use warnings;
use strict;
use HTML::TreeBuilder;

my (@p);

## Read the web page.
my $root = HTML::TreeBuilder->new_from_url( shift ) or die qq|ERROR: Malformed URL\n|;

## Get the table tag with id = 'toc'.
my $table_toc = $root->look_down(
    id => 'toc'
);

## Get inmediate previous siblings <p> tags.
for my $node ( reverse $table_toc->left ) { 
    if ( $node->tag eq 'p' ) { 
        unshift @p, $node;
    }   
    else {
        last;
    }   
}

## Print the content without the HTML tags.
for my $p ( @p ) { 
    printf qq|%s\n|, $p->as_text;
}

Run it providing the url as unique argument:

perl-5.14.2 script.pl http://en.wikipedia.org/wiki/Wayne_Rooney

With following output (I hope it to be near of what you expect):

Wayne Mark Rooney (born 24 October 1985) is an English footballer who plays as a forward for Premier League club Manchester United and the England national team.
Rooney made his senior international debut in 2003 becoming the youngest player to represent England, until he got beaten by Theo Walcott. He is England's youngest ever goalscorer.[4] He played at UEFA Euro 2004 and scored four goals, briefly becoming the competition's youngest goalscorer. Rooney featured at the 2006 and 2010 World Cups and is widely regarded as his country's best player.[5][6][7][8][9][10] He has won the England Player of the Year award twice, in 2008 and 2009. As of October 2012, he has won 78 international caps and scored 32 goals, making him England's fifth highest goalscorer in history.[11] Along with David Beckham, Rooney is the most red carded player for England, having been sent off twice.[12]
Wide character in printf at script.pl line 25.
Aged nine, Rooney joined the youth team of Everton, for whom he made his professional debut in 2002. He spent two seasons at the Merseyside club, before moving to Manchester United for £25.6 million in the 2004 summer transfer window. The same year, Rooney acquired the nickname "Wazza".[13] Since then, with Rooney in the team, United have won the Premier League four times, the 2007–08 UEFA Champions League and two League Cups. He also holds two runner-up medals from the Champions League and has twice finished second in the Premier League. In April 2012, Rooney scored his 180th goal, making him United's fourth-highest goal-scorer of all time.[14]
Wide character in printf at script.pl line 25.
In 2009–10, Rooney was awarded the PFA Players' Player of the Year and the FWA Footballer of the Year. He has won the Premier League Player of the Month award five times, a record he shares with Steven Gerrard. He came fifth in the vote for the 2011 FIFA Ballon d'Or and was named in the FIFPro World 11 for 2011. Rooney has won the 'Goal of the Season' award by the BBC's Match of the Day poll on three occasions, with his bicycle kick against rivals Manchester City winning the 'Premier League Goal of the 20 Seasons' award.[15] Rooney is the third highest-paid footballer in the world after Lionel Messi and Cristiano Ronaldo, with an annual income of €20.7m (£18m) including sponsorship deals.[16]

---

EDIT: To get as result also tags use printf qq|%s\n|, $p->as_HTML; instead of $p->as_text.