Question

I’m new to XPath – please go easy on me.

Having trouble extracting XPath on my target pages for elements that don’t have a lot of structure.

The data set is NJ school report cards. Individual report cards look like this

I’ve figured out how to pull out tables that have a summary tag:

url <- paste("http://education.state.nj.us/rc/rc11/rcreport.php?c=",
  all_sch[i,1],";d=",all_sch[i,2],";s=",all_sch[i,3],sep = '')
doc = htmlParse(url)
admin_salaries = getNodeSet(doc, '//table[@summary="Administrative Salaries and Benefits"]')

but am having trouble where there isn’t a lot of extra identifying information to work off of.

For instance, the table that has school name and district looks like this:

        <table cellpadding="0" cellspacing="0">
          <tr>
            <td><strong>SCHOOL:</strong></td>
            <td>&nbsp;New Jersey Ave</td>
          </tr>
          <tr>
            <td><strong>COUNTY:</strong></td>
            <td>&nbsp;Atlantic</td>
          </tr>
          <tr>
            <td><strong>DISTRICT:</strong></td>
            <td>&nbsp;Atlantic City</td>
          </tr>
        </table>

My strategy here was ‘find nodes that are tables and have the text COUNTY

Reading as much as I can about XPath, I’m trying this:

names = getNodeSet(doc,'//table and //*[contains(text(),"COUNTY")]')

But instead of returning back the table node, it gives me a boolean TRUE value.

So, the question is: How can I use XPath to find tables that have the text COUNTY and SCHOOL?

I’ve tried a lot of other strategies to little avail. One approach suggested by others was simply to pull out every table data cell using something like this:

xpathApply( htmlTreeParse(url, useInt=T), "//td", function(x) xmlValue(x))

But the templates aren’t consistent for missing data – incomplete reports have pretty different structure, and elements aren’t in the same position across the 2,000+ pages.

Any help is greatly appreciated!

Answer 1

using xpath, to get all the table

xpathSApply( doc, "//table[contains(.,'SCHOOL:') 
                  and contains(.,'COUNTY') ]",xmlValue)

To get just the row

xpathSApply( doc, "//tr/td[contains(.,'SCHOOL:') 
                   and contains(.,'COUNTY') ]",xmlValue)