Question

I’ve got a “bad” tab separated file that I need to clean up. The problem lies in the fact that fields might have linebreaks. I think the easiest way to fix this is to replace the ‘wrong’ linebreaks with some sort of replacement character, say a space. now I can imagine a way to do this, if there are supposed to be n fields on a line would be (pseudocode)

var line = read n-1 fields ending in a tab, and then until the end of line
line.replace("\n", " ")
line.replace("\r", " ")
write line to output

Now these files are huge, and slurping them is not an option. Is this a reasonable approach? (I know this will trip over linebreaks in the last field, but I’m willing to live with that)
What would be a good way to read enough data? I don’t care much which language it’s in, but prefer .net, perl or python2 as I have runtimes for those available.

Answer 1

you can do this in a really quick awk script:

awk -F\t '{while(NF < (numberoffields) { line=$0; getline; $0 = line $0;} print}'