Question

I’m using .NET 3.5 (C#) and the HTML Agility Pack to do some web scraping. Some fields that I need to extract are structured as paragraphs within which components are separated by line-break tags. I’d like to be able to select out the individual components between the line-breaks. Each component may be formed from multiple elements (i.e., it may not be just a single string). Example:

<h3>Section title</h3>
<p>
  <b>Component A</b><br />
  Component B <i>includes</i> <strong>multiple elements</strong><br />
  Component C
</p>

I’d like to select out

<b>Component A</b>

Then:

Component B <i>includes</i> <strong>multiple elements</strong>

And then:

Component C

There may be more (<br /> separated) components, too.

I can easily get the first component with:

p/br[1]/preceding-sibling::node()

I can also easily get the last component with:

p/br[2]/following-sibling::node()

But I haven’t been able to work out how to extract the set of nodes /between/ two other tags (that is, nodes which are siblings but which precede node X and follow node Y).

The alternative is to scan through the elements manually – if that’s the easiest way to do it then that’s what I’ll do, but XPath has so-far impressed me with its terseness, so I’m hoping there’s a way of doing this, too.

Edit

Since I need to cope with the situation of having more than 3 components, it seems the answer will require multiple XPath calls at a minimum, so I shall proceed with a solution based on that (this is the answer I have ‘accepted’). AakashM’s answer has also helped me with my understanding of XPath, and so I have voted it up.

Thank you all for your help! I hope I can return the favour one day.

Edit 2

The new answer provided by Dimitre Novatchev, with some tweaks, does indeed work correctly.

Solution:

int i = 0;
do
{
    yield return para.SelectNodes(String.Format(
        "node()[not(self::br) and count(preceding-sibling::br) = {0}]", i));
    ++i;
} while (para.SelectSingleNode(String.Format("br[{0}]", i)) != null);

I should note that this loop is somewhat inefficient due to the repeated XPath queries to find out whether there are any more br tags. In my case that inefficiency is not a problem, but be aware if you want to use this answer in some other situation (then again, if you did want to do this in a performance sensitive situation you should probably be scanning through manually anyway rather than using XPath).

And full test code (a modified version of the test code helpfully included by AakashM):

using System;
using System.Collections.Generic;
using System.Xml;

namespace TestXPath
{
    class Program
    {
        static void Main(string[] args)
        {
            XmlDocument doc = new XmlDocument();
            doc.LoadXml(@"
<x>
 <h3>Section title</h3>
 <p>
  <b>Component A</b><br />
  Component B <i>includes</i> multiple <strong>elements</strong><br />
  Component C
 </p>
</x>
            ");


            foreach (var nodes in SplitOnLineBreak(doc.SelectSingleNode("x/p")))
            {
                Dump(nodes);
                Console.WriteLine();
            }

            Console.ReadLine();
        }

        private static IEnumerable<XmlNodeList> SplitOnLineBreak(XmlNode para)
        {
            int i = 0;
            do
            {
                yield return para.SelectNodes(String.Format(
                    "node()[not(self::br) and count(preceding-sibling::br) = {0}]", i));
                ++i;
            } while (para.SelectSingleNode(String.Format("br[{0}]", i)) != null);
        }

        private static void Dump(XmlNodeList nodes)
        {
            foreach (XmlNode node in nodes)
            {
                Console.WriteLine(string.Format("-->{0}<---", 
                                  node.OuterXml));                    
            }
        }
    }
}

Answer 1

This can easily be done with XPath 2.0 or with XPath 1.0 hosted by XSLT.

With XPath 1.0 hosted by .NET this can be achieved in several steps:

1. Make the appropriate “p” node the current node.

2. Find the number of all <br /> children of the current “p” node:

   count(br)

3. if N is the count, determined in step 2. for $k in 0 to N do:

   3.1 Find all nodes that are preceded by $k <br /> elements:

   node()[not(self::br) and count(preceding::br) = $k]

   3.2 For every such node found, get its string value

   3.3 Concatenate all string values obtained in step 3.2. The result of this concatenation is all the text contained in the given paragraph.

Note: In order to substitute what should stand for $k in step 3.1 it is necessary to dynamically construct this expression.