Question

I have a csv file into which has crept some ^M dos line ends, and I want to get rid of them, as well as 16 spaces and 3 tabs which follow. Like, I have to merge that line with the next one down. Heres an offending record and a good one as a sample of what I mean:

"Mary had a ^M
                  little lamb", "Nursery Rhyme", 1878
"Mary, Mary quite contrary", "Nursery Rhyme", 1838

I can remove the ^M using sed as you can see, but I cannot work out how to rm the nix line end to join the lines back up.

sed -e "s/^M$             //g" rhymes.csv > rhymes.csv

UPDATE

Then I read “However, the Microsoft CSV format allows embedded newlines within a double-quoted field. If embedded newlines within fields are a possibility for your data, you should consider using something other than sed to work with the data file.” from:
http://sed.sourceforge.net/sedfaq4.html

So editing my question to ask Which tool I should be using?

Answer 1

With help from How can I replace a newline (\n) using sed?, I made this one:

sed -e ':a;N;$!ba;s/\r\n                \t\t\t/=/' -i rhymes.csv

<CR> <LF> <16 spaces> <3 tabs>

If you just want to delete the CR, you could use:

<yourfile tr -d "\r" | tee yourfile

(or if the two input and output file are different: <yourfile tr -d "\r" > output)