Question

I’m scraping values from HTML pages using XPath inside of a java program to get to a specific tag and occasionally using regular expressions to clean up the data I receive.

After some research, I landed on HTML Cleaner ( http://htmlcleaner.sourceforge.net/ ) as the most reliable way to parse raw HTML into a good XML format. HTML Cleaner, however, only supports XPath 1.0 and I find myself needing functions like ‘contains’. for instance, in this piece of XML:

<div>
  <td id='1234 foo 5678'>Hello</td>
</div>

I would like to be able to get the text ‘Hello’ with the following XPath:

//div/td[contains(@id, 'foo')]/text()

Is there any way to get this functionality? I have several ideas, but would prefer not to reinvent the wheel if I don’t need to:

• If there is a way to call HTML Cleaner’s evaluateXPath and return a TagNode (which I have not found), I can use an XML serializer on the returned TagNode and chain together XPaths to achieve the desired functionality.
• I could use HTML Cleaner to clean to XML, serialize it back to a string, and use that with another XPath library, but I can’t find a good java XPath evaluator that works on a string.
• Using TagNode functions like getElementsByAttValue, I could essentially recreate XPath evaluation and insert in the contains functionality using String.contains

Short question: Is there any way to use XPath contains on HTML inside an existing Java Library?

Answer 1

Regarding this:

I could use HTML Cleaner to clean to XML, serialize it back to a
string, and use that with another XPath library, but I can’t find a
good java XPath evaluator that works on a string.

This is exactly what I would do (except you don’t need to operate on a string (see below)).

A lot of HTML parsers try to do too much. HTMLCleaner, for example, does not properly/completely implement the XPath 1.0 spec (contains (for example) is an XPath 1.0 function). The good news is that you don’t need it to. All you need from HTMLCleaner is for it to parse the malformed input. Once you’ve done that, it’s better to use the standard XML interfaces to deal with the resulting (now well-formed) document.

First convert the document into a standard org.w3c.dom.Document like this:

TagNode tagNode = new HtmlCleaner().clean(
        "<div><table><td id='1234 foo 5678'>Hello</td>");
org.w3c.dom.Document doc = new DomSerializer(
        new CleanerProperties()).createDOM(tagNode);

And then use the standard JAXP interfaces to query it:

XPath xpath = XPathFactory.newInstance().newXPath();
String str = (String) xpath.evaluate("//div//td[contains(@id, 'foo')]/text()", 
                       doc, XPathConstants.STRING);
System.out.println(str);

Output:

Hello