Question

If I need to write a large file from allocated memory to disk, what is the most efficient way to do it?

Currently I use something along the lines of:

char* data = static_cast<char*>(operator new(0xF00000000)); // 60 GB 

// Do something to fill `data` with data

std::ofstream("output.raw", std::ios::binary).
   write(data, 0xF00000000);

But I am not sure if the most straightforward way is also the most efficient, taking into account various buffering mechanisms and alike.

I am using Windows 7 64-bit and Visual Studio 2012 RC compiler with 64-bit target.

Answer 1

For Windows, you should use CreateFile API. Have a good read of that page and any links from it mentioning optimization. There are some flags you pass in to turn off buffering. I did this in the past when I was collecting video at about 800MB per second, and having to write off small parts of it as fast as possible to a RAID array.

Now, for the flags – I think it’s primarily these:

• FILE_FLAG_NO_BUFFERING
• FILE_FLAG_WRITE_THROUGH

For reading, you may want to use FILE_FLAG_SEQUENTIAL_SCAN, although I think this has no effect if buffering is turned off.

Have a look at the Caching Behaviour section

There’s a couple of things you need to do. Firstly, you should always write amounts of data that are a multiple of the sector size. This is (or at least was) 512 bytes almost universally, but you may want to consider up to 2048 in future.

Secondly, your memory has to be aligned to that sector size too. You can either use _aligned_malloc() or just allocate more buffer than you need and align manually.

There may be other memory optimization concerns, and you may want to limit individual write operations to a memory page size. I never went into that depth. I was still able to write data at speeds very close to the disk’s limit. It was significantly faster than using stdio calls.

If you need to do this in the background, you can use overlapped I/O, but to be honest I never understood it. I made a background worker thread dedicated to writing out video buffer and controlled it externally.