Question

While writing a simple library to parse a game’s data files, I noticed that reading an entire data file into memory and parsing from there was significantly faster (by up to 15x, 106s v 7s).

Parsing is usually sequential but seeks will be done every now and then to read some data stored elsewhere in a file, linked by an offset.

I realise that parsing from memory will definitely be faster, but something is wrong if the difference is so significant. I wrote some code to simulate this:

public static void Main(string[] args)
{
    Stopwatch n = new Stopwatch();

    n.Start();
    byte[] b = File.ReadAllBytes(@"D:\Path\To\Large\File");
    using (MemoryStream s = new MemoryStream(b, false))
        RandomRead(s);
    n.Stop();
    Console.WriteLine("Memory read done in {0}.", n.Elapsed);
    b = null;
    n.Reset();
    n.Start();
    using (FileStream s = File.Open(@"D:\Path\To\Large\File", FileMode.Open))
        RandomRead(s);
    n.Stop();
    Console.WriteLine("File read done in {0}.", n.Elapsed);
    Console.ReadLine();
}
private static void RandomRead(Stream s)
{
    // simulate a mostly sequential, but sometimes random, read
    using (BinaryReader br = new BinaryReader(s)) {
        long l = s.Length;
        Random r = new Random();
        int c = 0;
        while (l > 0) {
            l -= br.ReadBytes(r.Next(1, 5)).Length;
            if (c++ <= r.Next(10, 15)) continue;
            // simulate seeking
            long o = s.Position;
            s.Position = r.Next(0, (int)s.Length);
            l -= br.ReadBytes(r.Next(1, 5)).Length;
            s.Position = o;
            c = 0;
        }
    }
}

I used one of the game’s data files as input to this. That file was about 102 MB, and it produced this result (Memory read done in 00:00:03.3092618. File read done in 00:00:32.6495245.) which has memory reading about 11x faster than file.

The memory read was done before the file read to try and improve its speed via the file cache. It’s still that much slower.

I’ve tried increasing or decreasing FileStream‘s buffer size; nothing produced significantly better results, and increasing or decreasing it too much just worsened the speed.

Is there something I’m doing wrong, or is this to be expected? Is there any way to at least make the slowdown less significant?

Why is reading the entire file at once and then parsing it so much faster than reading and parsing simultaneously?

I’ve actually compared to a similar library written in C++, which uses the Windows native CreateFileMapping and MapViewOfFile to read files, and it’s very fast. Could it be the constant switching from managed to unmanaged and the involved marshaling that causes this?

I’ve also tried MemoryMappedFiles present in .NET 4; the speed gain was only about one second.

Answer 1

Is there something I’m doing wrong, or is this to be expected?

No, nothing wrong. This is entirely expected. That accessing the disk is an order of magnitude slower than accessing memory is more than reasonable.

---

Update:

That a single read of the file followed by processing is faster than processing while reading is also expected.

Doing a large IO operation and processing in memory would be faster than getting a bit from disk, processing it, calling the disk again (waiting for the IO to complete), processing that bit etc…