Question

Using C# (VS 2010 Express) I read the contents of a text file into a string. The string is rather long but reliably broken up by “\t” for tabs and “\r\n” for carriage returns/newlines.

The tabs indicate a new column of data, and new line indicates a new row of data.

I want to create an array or List of dimensions (X)(Y) such that each spot in the array can hold 1 row of data from the text file, and all of the Y columns contained in that 1 row (“\t” means a new column of data, and “\r\n” means a new row of data”).

To make things simple let’s say my text has 10 rows of data, and 2 columns. I’d like to create an array or List or whatever you think is best to store the data. How do I do this? Thanks.

This is the code that I used to read the data in the text file into a string:

// Read the file as one string.
            System.IO.StreamReader myFile = new System.IO.StreamReader("f:\\data.txt");
            string myString = myFile.ReadToEnd();

Answer 1

Just as is (you already have a string with everything):

str.Split(new string[]{"\r\n"}, StringSplitOptions.None)
  .Select(s => s.Split('\t'));

Gives you an IEnumerable<string[]> producing variantes like list of list, array of array and so on just needs the suitable ToArray() or ToList() etc.

However, if you can deal with each line one at a time, you can be better off with something that lets you do so:

public IEnumerable<string[]> ReadTSV(TextReader tr)
{
  using(tr)
    for(string line = tr.ReadLine(); line != null; line = tr.ReadLine())
      yield return line.Split('\t');
}

Then you only use as much memory as each line needs. We could go further and change the reading to emit each individual cell one at a time, but this is normally enough to read files of several hundred MB in size, with reasonable efficiency.

Edit based on comments on question:

If you really wanted to, you could get a List<string[]> from:

var myFile = new StreamReader("f:\\data.txt");
var list = ReadTSV(myFile).ToList();

Alternatively, change the line yield return line.Split('\t'); to yield return line.Split('\t'); and you get a List<List<string>>.

However, if possible then work on the results directly, rather than putting it into a list first:

var myFile = new StreamReader("f:\\data.txt");
var chunks = ReadTSV(myFile);
foreach(var chunk in chunks)
{
   DoSometingOnAChunk(chunk[0], chunk[1]);
}

It’ll use less memory, and get started faster rather than pausing to read the whole thing first. Code like this can merrily work its way through gigabytes without complaint.