Question

I need a fast and efficient method to read a space separated file with numbers into an array. The files are formatted this way:

4 6
1 2 3 4 5 6
2 5 4 3 21111 101
3 5 6234 1 2 3
4 2 33434 4 5 6

The first row is the dimension of the array [rows columns]. The lines following contain the array data.

The data may also be formatted without any newlines like this:

4 6
1 2 3 4 5 6 2 5 4 3 21111 101 3 5 6234 1 2 3 4 2 33434 4 5 6

I can read the first line and initialize an array with the row and column values. Then I need to fill the array with the data values. My first idea was to read the file line by line and use the split function. But the second format listed gives me pause, because the entire array data would be loaded into memory all at once. Some of these files are in the 100 of MBs. The second method would be to read the file in chunks and then parse them piece by piece. Maybe somebody else has a better a way of doing this?

Answer 1

How about:

    static void Main()
    {
        // sample data
        File.WriteAllText("my.data", @"4 6
1 2 3 4 5 6
2 5 4 3 21111 101
3 5 6234 1 2 3
4 2 33434 4 5 6");

        using (Stream s = new BufferedStream(File.OpenRead("my.data")))
        {
            int rows = ReadInt32(s), cols = ReadInt32(s);
            int[,] arr = new int[rows, cols];
            for(int y = 0 ; y < rows ; y++)
                for (int x = 0; x < cols; x++)
                {
                    arr[y, x] = ReadInt32(s);
                }
        }
    }

    private static int ReadInt32(Stream s)
    { // edited to improve handling of multiple spaces etc
        int b;
        // skip any preceeding
        while ((b = s.ReadByte()) >= 0 && (b < '0' || b > '9')) {  }
        if (b < 0) throw new EndOfStreamException();

        int result = b - '0';
        while ((b = s.ReadByte()) >= '0' && b <= '9')
        {
            result = result * 10 + (b - '0');
        }
        return result;
    }

Actually, this isn’t very specific about the delimiters – it’ll pretty much assume that anything that isn’t an integer is a delimiter, and it only supports ASCII (you use use a reader if you need other encodings).