Question

I would like to split a string similar to

'abc "defg hijk \\"l; mn\\" opqrs"; tuv'

into

(['abc', '"defg hijk \\"l; mn\\" opqrs"'], 33)

i.e. I don’t want to break on semicolon inside (nested) quotes. What’s the easiest way, tokenize? It doesn’t hurt if it’s fast, but short is better.

Edit: I forgot one more detail that makes it even more tricky. I need the position of the semicolon that is cutting off the string, or -1 if there is none. (I’m doing changes to legacy code that used to be recursive, but stackoverflowed when the string became very long.)

Answer 1

It’s unlikely there is an easy way to solve this without a proper parser. You could probably get away with a hand built parser that doesn’t require tokenizing though.

Something like the following should be a good guide:

def parse(s):
    cur_s = []
    strings = []

    def flush_string():
        strings.push(''.join(cur_s))
        cur_s = []

    def handle_special_cases():
        # TODO: Fill this in

    for c in s:
        if c == ';':
            break
        elif c in ['\\' '"']:
            handle_special_cases()
        elif c == ' ':
            flush_string()
        else:
            cur_s.push(c)

    flush_string()
    return strings