Question

Is it possible to tell pyparsing to not store ParseResults or manually discard them?

I’m parsing a large file of items and can do all the post-processing for each item via a parse action. So as soon as an item has been parsed I don’t need it’s ParseResult any more and would like to be able to discard it as I’m hitting the memory limit of the machine I’m on.

Answer 1

Are you using parse actions to process the tokens as they are parsed? If so, you can delete the contents of the parsed tokens using del:

def parseActionThatDeletesTheParsedTokens(tokens):
    # ...
    # do something interesting with the tokens
    # ...

    # delete the contents of the parsed tokens
    del tokens[:]

Or you might want to just use scanString instead of parseString. Instead of this:

OneOrMore(blockOfText).parseString(bigHonkingString)

do:

for tokens, matchstart, matchend in blockOfText.scanString(bigHonkingString):
    # do stuff with the tokens

scanString returns a generator which yields 3-tuples containing the matched tokens, the starting, and ending location of each successive match. You can process each parsed set of tokens, then when you move on to the next set, the old set is automatically discarded. I think this may be the simplest way for you to go, with minimal changes to your program.