Question

I’ve been trying to parse some huge XML files that LXML won’t grok, so I’m forced to parse them with xml.sax.

class SpamExtractor(sax.ContentHandler):
    def startElement(self, name, attrs):
        if name == "spam":
            print("We found a spam!")
            # now what?

The problem is that I don’t understand how to actually return, or better, yield, the things that this handler finds to the caller, without waiting for the entire file to be parsed. So far, I’ve been messing around with threading.Thread and Queue.Queue, but that leads to all kinds of issues with threads that are really distracting me from the actual problem I’m trying to solve.

I know I could run the SAX parser in a separate process, but I feel there must be a simpler way to get the data out. Is there?

Answer 1

I thought I’d give this as another answer due to it being a completely different approach.

You might want to check out xml.etree.ElementTree.iterparse as it appears to do more what you want:

Parses an XML section into an element tree incrementally, and reports what’s going on to the user. source is a filename or file object containing XML data. events is a list of events to report back. If omitted, only “end” events are reported. parser is an optional parser instance. If not given, the standard XMLParser parser is used. Returns an iterator providing (event, elem) pairs.

You could then write a generator taking that iterator, doing what you want, and yielding the values you need.

e.g:

def find_spam(xml):
    for event, element in xml.etree.ElementTree.iterparse(xml):
        if element.tag == "spam":
            print("We found a spam!")
            # Potentially do something
            yield element

The difference is largely about what you want. ElementTree’s iterator approach is more about collecting the data, while the SAX approach is more about acting upon it.