Question

How can I reserve and allocate shared memory without the backing of a file? I’m trying to reserve a large (many tens of GiBs) chunk of shared memory and use it in multiple processes as a form of IPC. However, most of this chunk won’t be touched at all (the access will be really sparse; maybe a few hundred megabytes throughout the life of the processes) and I don’t care about the data when the applications end.

So preferably, the method to do this should have the following properties:

1. Doesn’t commit the whole range. I will choose which parts to commit (actually use.) (But the pattern is quite unpredictable.)
2. Doesn’t need a memory-mapped file or anything like that. I don’t need to preserve the data.
3. Lets me access the memory area from multiple processes (I’ll handle the locking explicitly.)
4. Works in both Linux and Windows (obviously a 64-bit OS is needed.)
5. Actually uses shared memory. I need the performance.
6. (NEW) The OS or the library doesn’t try to initialize the reserved region (to zero or whatever.) This is obviously impractical and unnecessary.

I’ve been experimenting with boost::interprocess::shared_memory_object, but that causes a large file to be created on the filesystem (with the same size as my mapped memory region.) It does remove the file afterwards, but that hardly helps.

Any help/advice/pointers/reference is appreciated.

P.S. I do know how to do this on Windows using the native API. And POSIX seems to have the same functionality (only with a cleaner interface!) I’m looking for a cross-platform way here.

UPDATE: I did a bit of digging, and it turns out that the support that I thought existed in Windows was only a special case of memory-mapped files, using the system page file as a backing. (I’d never noticed it before because I had used at most a few megabytes of shared memory in the past projects.)

Also, I have a new requirement now (the number 6 above.)

Answer 1

On Windows, all memory has to be backed by the disk one way or another.

The closest I think you can accomplish on windows would be to memory map a sparse file. I think this will work in your case for two reasons:

1. On Windows, only the shared memory that you actually touch will become resident. This meets the first requirement.
2. Because the file is sparse, only the parts that have been modified will actually be stored on disk. If you looked at the properties of the file, it would say something like, “Size: 500 MB, Size on disk: 32 KB”.

I realize that this technically doesn’t meet your 2nd requirement, but, unfortunately, I don’t think that is possible on Windows. At least with this approach, only the regions you actually use will take up space. (And only when Windows decides to commit the memory to disk, which it may or may not do at its discretion.)

As for turning this into a cross-platform solution, one option would be to modify boost::interprocess so that it creates sparse files on Windows. I believe boost::interprocess already meets your requirements on Linux, where POSIX shared memory is available.