Question

I was attempting to parse example.com’s home page with xpath and cssselect but it seems as though either I don’t know how xpath works, or lxml’s xpath is broken, as it is missing matches.

Here’s the quick and dirty code.

from lxml.html import *
mySearchTree = parse('http://www.example.com').getroot()
for a in mySearchTree.cssselect('tr a'):
    print 'found "%s" link to href "%s"' % (a.text, a.get('href'))

print '-'*8 +'Now for Xpath' + 8*'-'
# Find all 'a' elements inside 'tr' table rows with xpath
for a in mySearchTree.xpath('.//tr/*/a'):
    print 'found "%s" link to href "%s"' % (a.text, a.get('href'))

Results:

found "About" link to href "/about/"
found "Presentations" link to href "/about/presentations/"
found "Performance" link to href "/about/performance/"
found "Reports" link to href "/reports/"
found "Domains" link to href "/domains/"
found "Root Zone" link to href "/domains/root/"
found ".INT" link to href "/domains/int/"
found ".ARPA" link to href "/domains/arpa/"
found "IDN Repository" link to href "/domains/idn-tables/"
found "Protocols" link to href "/protocols/"
found "Number Resources" link to href "/numbers/"
found "Abuse Information" link to href "/abuse/"
found "Internet Corporation for Assigned Names and Numbers" link to href "http://www.icann.org/"
--------Now for Xpath--------
found "Presentations" link to href "/about/presentations/"
found "Performance" link to href "/about/performance/"
found "Reports" link to href "/reports/"
found "Root Zone" link to href "/domains/root/"
found ".INT" link to href "/domains/int/"
found ".ARPA" link to href "/domains/arpa/"
found "IDN Repository" link to href "/domains/idn-tables/"
found "Abuse Information" link to href "/abuse/"
found "Internet Corporation for Assigned Names and Numbers" link to href "http://www.icann.org/"

Basically the xpath found every link it was supposed to, except for those that were bolded by Example.com. However, shouldn’t the asterisk wildcard have allowed for this in the xpath match ‘.//tr/*/a’?

Answer 1

Possibly something else is going on (I didn’t examine the sample document closely), but your CSS selector and XPath are not equivalent.

CSS tr a is //tr//a in XPath. .//tr/*/a means (conceptually, not precisely):

1. .: current node
2. //: all descendants of current node
3. tr: all tr elements among all descendants of current node
4. /: all children of found tr elements
5. *: any element among children of found tr elements
6. /: all children of any child element of found tr elements
7. a: all a elements which which are element children of element children of a tr element

In other words, given the following HTML:

<ul>
    <li><a href="link1"></a><li>
    <li><b><a href="link2"></a></b><li>
</ul>

//ul/*/a will only match link1.

XPath Primer

In reality, an "XPath" is a series of Location Steps separated by slashes. A Location Step consists of:

1. An axis (e.g. child::)
2. A node test (either the node’s name, or one of the special node types, e.g. node(), text())
3. Optional predicates (surrounded by []. A node is only matched if all the predicates are true.)

If we were to decompose .//tr/*/a into its Location Steps it would look like this:

1. .
2. (the "space" between the slashes in "//")
3. tr
4. *
5. a

It’s probably not evident what the heck I am talking about. This is because XPath has an abbreviated syntax. Here is the expression with abbreviations expanded (axis and node-test are separated by a ::, steps by a /):

self::node()/descendent-or-self::node()/child::tr/child::*/child::a

(Notice that self::node() is redundant.)

Conceptually what happens in a step is:

1. given a set of context nodes (default is the current node or ‘/’ for the root node)
2. For each context node, create a set of nodes which satisfy the Location Step
3. Union all per-context-node sets into one node set
4. Pass that set to the next Location Step as its given context nodes.
5. Repeat until out of steps. The set left after the final step is the set for the entire path.

Note that this is still a simplification. Read the XPath Standard for the gory details if you want them.