Question

I’m trying to use the pattern described in the “event-driven parsing” section of the lxml tutorial.

In my code I’m calling a function that can recursively run on elements using the iterchildren() method. I’ll just use two nested loop for illustration here.

This works as expected:

xml = StringIO("<root><a><b>data</b><c><d/></c></a><a><z/></a></root>")
for ev, elem in etree.iterparse(xml):
    if elem.tag == 'a':
        for c in elem.iterchildren():
             for gc in c.iterchildren():
                  print gc

The output is <Element d at 0x2df49b0>.

But if I add .clear() in the end:

for ev, elem in etree.iterparse(xml):
    if elem.tag == 'a':
        for c in elem.iterchildren():
             for gc in c.iterchildren():
                  print gc
    elem.clear()

— it doesn’t print anything. Why is it so and what do I do to work around this?

Notes:

• I can skip iterchildren and do for c in elem or for c in list(elem), with the same effect.
• I need to use iterative approach to keep the memory usage low.

• In the real use case, I am doing an element lookup using an attribute:

  if elem.attrib.get('id') == elem_id:
      return _get_info(elem)
  

I would like an explanation of how clear manages to erase the inner elements before they are processed, and how to keep them in memory while they’re needed for processing of the ancestors.

Answer 1

The problem is that iterparse by default yields end events. For subelements end events are generated earlier than for their ancestors:

>>> for ev, elem in etree.iterparse(xml):
       print elem

<Element b at 0x38fe320>
<Element d at 0x38fe0f0>
<Element c at 0x38fe2d0>
<Element a at 0x38fe190>
<Element z at 0x38fe230>
<Element a at 0x38fe3c0>
<Element root at 0x2df48c0>

In this simple case the problem can be solved by relying on the start events instead:

for ev, elem in etree.iterparse(xml, events=('start',)):
    ...

However, the linked docs say,

Note that the text, tail and children of an Element are not
necessarily there yet when receiving the start event. Only the end
event guarantees that the Element has been parsed completely.

That probably means that both start and end events should be processed to allow calling clear() only when it’s safe. I’m thinking of implementing some kind of a stack and pushing things there on start events, popping and clearing on end events.

I would very much welcome answers that show implementations using this or other ideas.

Note: the easiest thing to do is just indent the clear call into the if. That would allow accessing grandchildren, but all unrelated elements would remain uncleared.

---

EDIT

I am currently using a state-keeping variable in the following manner:

found = False
for event, elem in etree.iterparse(source, events=('start', 'end')):
    if event == 'start':
        if elem.attrib.get('id') == elem_id:
            found = True
    else:
        if elem.attrib.get('id') == elem_id:
            return _get_info(elem)
        if not found:
            elem.clear()