Question

We have an integration with another system that relies on passing CSV files back and forth (really old school).

The structure is generally:

ID, Name, PhoneNumber, comments, fathersname
1, tom, 555-1234, just some random text, bill
2, jill smith, 555-4234, other random text, richard

Every so often we see this:

3, jacked up, 999-1231, here  
be dragons  
amongst us, ted

The primary problem I care about is detecting that a line breaker (\n) occurs in the middle of the record when that is the record terminator.

Is there anyway I can preprocess this to reliably fix it?

Note that we have zero control over what the other system emits.

Answer 1

So you should be able to do something more or less like this:

for (int i = 0; i < lines.Count; i++)
{
    var fields = lines[i].Split(',').ToList();
    while (fields.Count < numFields)//here be dragons amonst us
    {
        i++;//include next line in this line
        //check to make sure we haven't run out of lines.

        //combine end of previous field with start of the next one, 
        //and add the line break back in.
        var innerFields = lines[i].Split(',');
        fields[fields.Count - 1] += "\n" + innerFields[0];

        fields.AddRange(innerFields.Skip(1));
    }

    //we now know we have a "real" full line
    processFields(fields);
}

(For simplicity I assumed all lines were read in at the start; I assume you could alter it to lazily fetch each line easily enough.)