Question

I’ve been writing some code that extracts the main textual content from web pages. One strategy that’s been useful is to locate the first paragraph of content, then select all of the following sibling elements up to, but not including, the first one that isn’t a p, ul, ol, or blockquote element. In Perl, the code looks something like this:

my ($firstpara) = $document->findnodes('//p[whatever]');
my @content = ($firstpara);
for my $sibling ($firstpara->findnodes('following-sibling::*')) {
    last if $sibling->tag !~ /^(?:p|ol|ul|blockquote)\z/;
    push @content, $sibling;
}

This isn’t too bad, but it would be cool to be able to get the nodes I want using only XPath, so I could write something like this instead:

my ($firstpara) = $document->findnodes('//p[whatever]');
my @content = ($firstpara, $firstpara->findnodes('<query>'));

I’ve done a lot of experimentation, but haven’t been able to figure out how to write that last query. The closest to a valid-looking solution I’ve been able to find is something like:

$firstpara->findnodes('following-sibling::*[position() < $EXPR]');

…where $EXPR is some expression that returns the position of the next sibling whose tag is not p, ul, ol, or blockquote, but I haven’t been able to work out if such an expression is expressible in XPath.

Is there any way to do what I’ve described in XPath?

Example:

Suppose my document looks like this:

<h1>Header</h1>
<p>Paragraph 1</p>
<p id="first">Paragraph 2</p>
<p>Paragraph 3</p>
<ul><li>Item 1</li><li>Item 2</li></ul>
<p>Paragraph 4</p>
<hr>
<p>Paragraph 5</p>
<blockquote>Blockquote 1</blockquote>
...

I have a reference to the <p> element with id first. I’m after an XPath expression, using that first element as the content node, that will give me the following siblings Paragraph 3, the unordered list, and Paragraph 4. The <hr> element is not among those I want (<p>, <ul>, <ol>, and <blockquote>), so that element and all siblings after that should not be part of the returned node set.

Answer 1

As the OP explained, he wants:

all of the following sibling elements up to, but not including, the
first one that isn’t a p, ul, ol, or blockquote element

I. XPath 1.0 solution:

The nodes that are wanted are the intersection of two nodesets:

1. All elements that are following siblings of the p with id with value 'first'.

2. All elements that are preceding siblings of hr.

To find this with XPath 1.0 we use the Kayessian formula for nodeset intersection:

$ns1[count(.|$ns2) = count($ns2)]

The above XPath expression selects all nodes that belong both to the node-set $ns1 and to the node-set $ns2.

Let $vP1 is defined as /*/p[@id='first'].

Let $vFirstNotInRange is:

   $vP1/following-sibling::*
    [not(self::p or self::ul
        or self::ol or self::blockquote)
    ] [1]

This selects the first unwanted node (in this case hr), or more precisely: the first element that is a following sibling of $vP1 and that is not a p, a ul, a ol or a blockquote.

Then the two node-sets we want to intersect are all following siblings of $vP1 and all preceding siblings of $vFirstNotInRange:

Let us denote with $vFollowingP1 the first node-set — this is:

$vP1/following-sibling::*

And let us denote with $vPreceedingNotInRange the second node-set — this is:

$vFirstNotInRange/preceding-sibling::*

Finally we substitute in the Kayessina formula $ns1 with $vPreceedingNotInRange and $ns2 with $vFollowingP1. The reult of these substitutions selects exactly the wanted nodes:

$vPreceedingNotInRange
         [count(.|$vFollowingP1)
         =
          count($vFollowingP1)
         ]

If we substitute all variables until we get an expression that doesn’t contain any variables, we get:

   /*/p[@id='first']/following-sibling::*
     [not(self::p or self::ul
         or self::ol or self::blockquote
          )
     ] [1]
        /preceding-sibling::*
          [count(.| /*/p[@id='first']/following-sibling::*)
          =
           count(/*/p[@id='first']/following-sibling::*)
          ]

This expression selects exactly the wanted nodes.

Here is an XSLT – based verification:

<xsl:stylesheet version="1.0"
 xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
 <xsl:output omit-xml-declaration="yes" indent="yes"/>
 <xsl:strip-space elements="*"/>

 <xsl:variable name="vP1" select="/*/p[@id='first']"/>

 <xsl:variable name="vFirstNotInRange" select=
  "$vP1/following-sibling::*
    [not(self::p or self::ul
        or self::ol or self::blockquote)
    ] [1]"/>

 <xsl:variable name="vFollowingP1"
      select="$vP1/following-sibling::*"/>

 <xsl:variable name="vPreceedingNotInRange"
      select="$vFirstNotInRange/preceding-sibling::*"/>

 <xsl:template match="/">
  <xsl:copy-of select=
  "$vPreceedingNotInRange
   [count(.|$vFollowingP1)
   =
    count($vFollowingP1)
   ]"/>
================

  <xsl:copy-of select=
  "/*/p[@id='first']/following-sibling::*
     [not(self::p or self::ul
         or self::ol or self::blockquote
          )
     ] [1]
        /preceding-sibling::*
          [count(.| /*/p[@id='first']/following-sibling::*)
          =
           count(/*/p[@id='first']/following-sibling::*)
          ]

  "/>
 </xsl:template>
</xsl:stylesheet>

When this transformation is applied on the following XML document (the provided non-wellformed XML fragment — corrected and wrapped in order to be made wellformed):

<html>
    <h1>Header</h1>
    <p>Paragraph 1</p>
    <p id="first">Paragraph 2</p>
    <p>Paragraph 3</p>
    <ul>
        <li>Item 1</li>
        <li>Item 2</li>
    </ul>
    <p>Paragraph 4</p>
    <hr/>
    <p>Paragraph 5</p>
    <blockquote>Blockquote 1</blockquote>
</html>

the two XPath expressions (one with variables and one with all variables substituted) are evaluated and the wanted, correct selected nodes output:

<p>Paragraph 3</p>
<ul>
   <li>Item 1</li>
   <li>Item 2</li>
</ul>
<p>Paragraph 4</p>
================

  <p>Paragraph 3</p>
<ul>
   <li>Item 1</li>
   <li>Item 2</li>
</ul>
<p>Paragraph 4</p>

II. XPath 2.0 solution:

$vFirstNotInRange/preceding-sibling::*
                              [. >> $vP1]

This selects any preceding sibling of $vFirstNotInRange that is also following $vP1 and selects the same wanted nodes:

<p>Paragraph 3</p>
<ul>
   <li>Item 1</li>
   <li>Item 2</li>
</ul>
<p>Paragraph 4</p>

Explanation: Here we use the XPath 2.0 “follows” operator >>.