Question

I’m parsing a large csv files – about 500 meg (many rows, many columns). I only need the first two columns (so up to the second comma on each line). Also, multiple threads need access to this file at the same time, so I can’t take an exclusive lock.

What’s the fastest/least memory consuming approach to this problem? What classes/methods should I be looking at? I assume that I should stay as low-level as possible – reading character by character, line by line?

Perhaps this is a way to allow simultaneous access?

using ( var filestream = new FileStream( filePath , FileMode.Open , FileAccess.Read , FileShare.Read ) )
{
     using ( var reader = new StreamReader( filestream ) )
     {
       ...
     }
}

Edit
Decided to check out http://www.codeproject.com/KB/database/CsvReader.aspx
which seems to give me the ability to read just two columns and then skip to the next line.
They also have some benchmarks showing fast performance and low memory profile.

Answer 1

If you want low memory, you’ll probably use a StreamReader and ReadLine by line.

In a similar case the other day, I was able to skip the first 20,000,000 lines in a 500 MB file and build a string (using StringBuilder) for the next 1,000,000 lines in about 7 seconds.