Question

var fileList = Directory.GetFiles("./", "split*.dat");
int fileCount = fileList.Length;
int i = 0;
foreach (string path in fileList)
{
    string[] contents = File.ReadAllLines(path); // OutOfMemoryException
    Array.Sort(contents);
    string newpath = path.Replace("split", "sorted");
    File.WriteAllLines(newpath, contents);
    File.Delete(path);
    contents = null;
    GC.Collect();

    SortChunksProgressChanged(this, (double)i / fileCount);
    i++;
}

And for file that consists ~20-30 big lines(every line ~20mb) I have OutOfMemoryException when I perform ReadAllLines method. Why does this exception raise? And how do I fix it?
P.S. I use Mono on MacOS

Answer 1

You should always be very careful about performing operations with potentially unbounded results. In your case reading a file. As you mention, the file size and or line length is unbounded.

The answer lies in reading ‘enough’ of a line to sort then skipping characters until the next line and reading the next ‘enough’. You probably want to aim to create a line index lookup such that when you reach an ambiguous line sorting order you can go back to get more data from the line (Seek to file position). When you go back you only need to read the next sortable chunk to disambiguate the conflicting lines.

You may need to think about the file encoding, don’t go straight to bytes unless you know it is one byte per char.

The built in sort is not as fast as you’d like.

Side Note:

1. If you call GC.* you’ve probably done it wrong
2. setting contents = null does not help you
3. If you are using a foreach and maintaining the index then you may be better with a for(int i…) for readability