Question

I’v written some software that writes live video streams to disk. At a certain point (4 streams of 1024×768 at 10fps) the program dies – it’s not out of RAM and it’s not the CPU – so I’m guessing that it’s the Hard Disk that can’t write the data fast enough (there’s no error, it just stops working). Basically is there a way to find out if the hard drive is about to fall over (write buffer full)?

Answer 1

Well, it is not the disk write error. There’s a simple explanation for what you see, you are actually out of RAM.

File writes are normally very fast in Windows. A memory-to-memory copy, it writes the file data into the file system cache. Which then takes it merry time writing the data to the slow hard drive. It also optimizes the writes to disk, writing them in cylinder order so that the disk write head movement is optimized.

This works very well. Unless you relentlessly write so much file data that the cache fills up to capacity. Write speed then falls off a cliff. From about 5 gigabytes per second to 50 megabytes per second. Or less, depending on how fragmented the drive is. Head seek speed is a factor too but there isn’t much variability between drives, it is largely determined by how fast the disk platters spin. A 10,000 rpm drive performs better than a 7,200 rpm drive since it doesn’t have to wait as long for the right sector to arrive at the write head.

The size of the file system cache is determined by the amount of RAM you have in the machine. Which does need to be shared with the RAM required by other processes. Same idea as a RAM disk, but much smarter. About one gigabyte is typical on a 32-bit operating system. Getting more requires a 64-bit operating system with enough RAM installed.

But more RAM probably isn’t going to solve your problem, it will just take longer to fill up the cache. At some point, you do either need a much smarter video stream encoder so the data is compressed better. Or a faster hard drive. Ask about that at superuser.com