Question

I have a “CSV” with which some of the data fields happen to contain the comma delimiter as in the second row of the following sample data.

"1","stuff","and","things"
"2","black,white","more","stuff"

I can’t change the source data and I don’t know how to str.split() and not split in the value “black,white”.

Ways I’ve approached my problem:

1. I looked at partition() and don’t see how that would benefit me.
2. I’m sure a regex would capture data properly but I’m not sure how to tie one into splitting.
3. Since every row in the source will always have the same number of fields I thought maybe setting maxsplit would help but talked myself out of that with the thinking that it would still split within “black,white” and I would end up loosing the last value (which would be “stuff” in this case).

Certainly this is easy to overcome so I’m looking forward to learning something new!

Your help is greatly appreciated.

Answer 1

Commas outside the strings are always followed by double-quotes. Just split on ," instead of just , (or even ",")

>>> x = '"2","black,white","more","stuff"'
>>> x
'"2","black,white","more","stuff"'
>>> x.split(',"')
['"2"', 'black,white"', 'more"', 'stuff"']
>>> [y.strip('"') for y in x.split(',"')]
['2', 'black,white', 'more', 'stuff']

Of course, edit for efficiency

YevgenYampolskiy’s suggestion of shlex is also an alternative.

>>> x = '"2","black,white","more","stuff"'
>>> x
'"2","black,white","more","stuff"'
>>> import shlex
>>> y = shlex.shlex(x)
>>> [i.strip('"') for i in y if i != ',']
['2', 'black,white', 'more', 'stuff']