Question

I’m trying to iteratively parse a large (186MB) XML file. The file itself is just a list of complex MyNodeType nodes. E.g.,

<MyNodeTypeList>
    <MyNodeType>
        ...
    </MyNodeType>
    <MyNodeType>
        ...
    </MyNodeType>
    <MyNodeType>
        ...
    </MyNodeType>
</MyNodeTypeList>

I am trying to keep the memory usage low by iteratively parsing the file, but it seems to grind to a massive slow down around 30k records and comes to a halt at 92k. The process is also taking up around 2gb in memory, even with simple code like the one below:

import xml.etree.cElementTree as ET

def main(argv):
    it = ET.iterparse(argv[0])
    count = 0
    for (ev, el) in it:
        if (ev == "end" and el.tag == "MyNodeType"):
            count += 1
    print count

if __name__ == "__main__":
   main(sys.argv[1:])

Is there some way I can keep the processing code to a bare minimum in terms of memory usage?

Answer 1

This behaviour of iterparse is documented:

Note that iterparse still builds a tree, just like parse, but you can safely rearrange or remove parts of the tree while parsing

To avoid keeping the whole tree in memory, use the method provided in the example:

# get an iterable
context = iterparse(source, events=("start", "end"))

# turn it into an iterator
context = iter(context)

# get the root element
event, root = context.next()

for event, elem in context:
    if event == "end" and elem.tag == "record":
        ... process record elements ...
        root.clear()