Question

I have some huge binary files which I’m currently reading using memory mapping on Windows. Everything works fine, but problems start when the file size is larger than memory. For instance, on a 24 GiB memory machine, I map a 64 GiB file and then I’m doing more or less streaming reads through it (not totally streaming, but ok’ish.) Nevertheless, the process winds up using all memory at some point.

I’m touching each byte in the file exactly once, so it should be enough to keep let’s say 4-5 GiB around for optimal performance. Is there a way to force Windows (7) to discard pages? FlushViewOfFile discards dirty pages only, but in my use case, I’m only reading from the file. I’ve already set the NO_BUFFER flag, but I’d like to be able to reduce the page priority somehow so they get evicted more quickly. Right now, once the memory is full, the app comes to a grinding halt as Windows pages out everything.

[Edit] It’s a 64-bit application on a 64-bit windows.

Answer 1

Don’t map it all as a single view. Call CreateFileMapping once, then call MapViewOfFile and UnmapViewOfFile as you need segments of the file. UnmapViewOfFile should cause the pages to be discarded.

You can of course have several views from the same file mapping at once, they can be overlapping or non-overlapping, and overlapping views will be coherent (even across different file mappings and different processes).