Question

Using Python, I’m trying to parse a string like this:

"hello" "I am an example" "the man said:\"hello!\""

into these tokens:

1) hello
2) I am an example
3) the man said: "hello!"

Something like re.findall(r'"[^"]*"', str) comes close, but cannot handle the escape char (\). I’m curious what kind of pythonic ways there are to deal with the escape char without resorting to for loops or large parser packages.

Answer 1

This is a good fit for regex:

re.findall(r'"(?:\\.|[^"\\])*"', str)

Explanation:

"        # Match a "
(?:      # Match either...
 \\.     # an escaped character (\\, \" etc.)
|        # or
 [^"\\]  # any character except " or \
)*       # any number of times
"        # Match a "

This will correctly handle escaped backslashes:

>>> import re
>>> test = r'"hello" "Hello\\" "I am an example" "the man said:\"hello!\\\""'
>>> for match in re.findall(r'"(?:\\.|[^"\\])*"', test):
...     print(match)
...
"hello"
"Hello\\"
"I am an example"
"the man said:\"hello!\\\""