Question

I have a CSV file I need to process which is a bit of a nightmare. Esentially it is the following

"Id","Name","Description"
"1","Test1","Test description text"
"2","Test2","<doc><style>body{font-family:"Calibri","sans-serif";}</style><p class="test_class"
name="test_name">Lots of word xdoc content here.</p></doc>"
"guid-xxxx-xxxx-xxxx-xxxx","Test3","Test description text 3"

I’m using the File Helpers library to process the CSV rather than reinvent the wheel. However, due to the description field containing unescaped Word xdoc xml which contains quotes it’s getting rather confused when it comes to the start and end points of each record.

The following is an example mapping class.

[DelimitedRecord(","), IgnoreFirst(1), IgnoreEmptyLines()]
public class CSVDoc
{
    #region Properties
    [FieldQuoted('"', QuoteMode.AlwaysQuoted), FieldTrim(TrimMode.Both)]
    public string Id;
    [FieldQuoted('"', QuoteMode.AlwaysQuoted), FieldTrim(TrimMode.Both)]
    public string Name;
    [FieldQuoted('"', QuoteMode.AlwaysQuoted), FieldTrim(TrimMode.Both)]
    public string Description;
    [FieldQuoted('"', QuoteMode.AlwaysQuoted), FieldTrim(TrimMode.Both)]
}

I considered (despite my hate of regex for this kind of task) replacing all " with ' and then using ((?<=(^|',))'|'(?=($|,'))) pattern to replace all ' with " at the start and end of lines and where they are formatted ','. However, the dirty file contains some lines which end with a " and some css style attributes which are formatted ","

So now I’m left scratching my head trying to figure out how to do this and how it can be automated.

Any ideas?

Answer 1

You’re going to have to re-invent the wheel, because that’s not valid CSV or indeed a reasonable file at all – it doesn’t have any sort of provably consistent escaping rules (e.g. we don’t know if the plain-text columns are escaped correctly or not).

Your best bet is to ask the person producing this to fix the bug, it should be e.g.:

"2","Test2","<doc><style>body{font-family:""Calibri"",""sans-serif"";}</style><p class=""test_class""
name=""test_name"">Lots of word xdoc content here.</p></doc>"

Which your parser should handle fine, and which should not be hard for them to produce in a simple and efficient manner.

Failing that, you’ll have to hand-code the parser to:

1. Read a line.
2. Check for unescaped ” (any "that isn’t followed by a " a , or whitespace.
3. If none found, parse as CSV.
4. If any found, parse as this horrible thing until you hit the line ending with “

It may be easier to look for < if that is consistently not used in the other lines. Or perhaps for <doc if it consistently identifies the correct rows.