Question

Here’s the simplest way to explain this. Here’s what I’m using:

re.split('\W', 'foo/bar spam\neggs')
>>> ['foo', 'bar', 'spam', 'eggs']

Here’s what I want:

someMethod('\W', 'foo/bar spam\neggs')
>>> ['foo', '/', 'bar', ' ', 'spam', '\n', 'eggs']

The reason is that I want to split a string into tokens, manipulate it, then put it back together again.

Answer 1

The docs of re.split mention:

Split string by the occurrences of pattern. If capturing
parentheses are used in pattern, then the text of all groups in the
pattern are also returned as part of the resulting list.

So you just need to wrap your separator with a capturing group:

>>> re.split('(\W)', 'foo/bar spam\neggs')
['foo', '/', 'bar', ' ', 'spam', '\n', 'eggs']