Question

I have a question that should make most people go “WTF?”, but I have it nonetheless.

I’ve got a bunch of data files from a vendor. It’s in a custom flat-file format that claims to be CSV, except it’s not comma separated, and values are not quoted. So, not really CSV at all.

foo,bar,baz
alice,bob,chris

And so on, except much longer and less interesting. The problem is, some records have embedded newlines (!!!):

foo,bar
rab,baz
alice,bob,chris

That is supposed to be two records of three fields each. Normally, I would just say “No, this is stupid.”, but I inadvisedly looked closer, and discovered that it was actually a different kind of end of line than the actual line ending sequence:

foo,bar\n
rab,baz\r\n
alice,bob,chris\r\n

Note the \n on the first line. I’ve determined that this holds for all the cases I found of embedded newlines. So, I need to basically do s/\n$// (I tried this specific command, it did not do anything).

Note: I don’t actually care about the contents of the fields, so replacing a newline with nothing is fine. I just need each line in the file to have the same number of records (ideally, in the same place).

I have an existing solution in the tool I wrote to process the files:

Guid g = Guid.NewGuid();

string data = File.ReadAllText(file, Encoding.GetEncoding("Latin1"));
data = data.Replace("\r\n", g.ToString()); //just so I have a unique placeholder
data = data.Replace("\n", "");
data = data.Replace(g.ToString(), "\r\n");

However, this fails on files that are bigger than a gigabyte or so. (Also, I haven’t profiled it, but I suspect it’s dog slow as well).

The tools I have at my disposal are:

• cygwin tools (sed, grep, etc)
• .NET

What is the best way to do this?

Answer 1

Instead of reading the entire thing into memory as a big (potentially huge) string, consider a stream based approach instead.

Open the input stream and read a line at a time, making your replacements as needed. Open an output stream and write the modified line into it. Something like:

static void Main( string[] args )
{
    using( var inFs = File.OpenRead( @"C:\input.txt" ) )
    using( var reader = new StreamReader( inFs ) )
    using( var outFs = File.Create( @"C:\output.txt" ) )
    using( var writer = new StreamWriter( outFs ) )
    {
        int cur;
        char last = '0';
        while( ( cur = reader.Read() ) != -1 )
        {
            char next = (char)reader.Peek();
            char c = (char)cur;
            if( c != '\n' || last == '\r' )
                writer.Write( c );

            last = c;
        }
    }
}