Question

I have an XML document coming in over a socket that I need to parse and react to on the fly (ie parsing a partial tree). What I’d like is a non blocking method of doing so, so that I can do other things while waiting for more data to come in (without threading).

Something like iterparse would be ideal if it finished iterating when the read buffer was empty, eg:

context = iterparse(imaginary_socket_file_wrapper)
while 1:
    for event, elem in context:
        process_elem(elem)
    # iteration of context finishes when socket has no more data
    do_other_stuff()
    time.sleep(0.1)

I guess SAX would also be an option, but iterparse just seems simpler for my needs. Any ideas?

Update:

Using threads is fine, but introduces a level of complexity that I was hoping to sidestep. I thought that non-blocking calls would be a good way to do so, but I’m finding that it increases the complexity of parsing the XML.

Answer 1

Diving into the iterparse source provided the solution for me. Here’s a simple example of building an XML tree on the fly and processing elements after their close tags:

import xml.etree.ElementTree as etree

parser = etree.XMLTreeBuilder()

def end_tag_event(tag):
    node = self.parser._end(tag)
    print node

parser._parser.EndElementHandler = end_tag_event

def data_received(data):
    parser.feed(data)

In my case I ended up feeding it data from twisted, but it should work with a non-blocking socket also.