Question

I have a massive XML file. However, I’m only interested in a single small subtree of this massive tree. I want to parse this subtree, but I don’t want to waste time parsing the entire massive tree when I’m going to only be using a small part of it.

Ideally, I’d want to scan through the file until I find the start of this subtree, parse the subtree until I reach the end of it, and then not even bother reading the rest of the file. Maybe even describe the subtree I’m interested in with an XPath expression. Is there a way to do this using lxml? It looks like you might be able to do something like this using the iterparse() method, but based on the docs, it looks like that doesn’t produce a parsed object, which I want to use. Any suggestions?

(Using lxml is not required, but I want to use Python, and ideally I’d like it to be speedy.)

Answer 1

I get the impression that iterparse is what you want. Looking at the section “Selective tag events” at http://lxml.de/parsing.html it seems like that gives you what you desire:

context = etree.iterparse(xmlfile, tag="yourSubTree")
action, elem = context.next()
etree.iterwalk(elem, ...)...

Seems like XPath could also work but I’d guess that XPath reads in the whole tree before returning whereas I’d expect iterparse to only walk the tree until it has a match. It would be worth profiling the two approaches.