Question

My main problem is that I need to enable multiple OS processes to communicate via a large shared memory heap that is mapped to identical address ranges in all processes. (To make sure that pointer values are actually meaningful.)

Now, I run into trouble that part of the program/library is using standard malloc/free and it seems to me that the underlying implementation does not respect mappings I create with mmap.
Or, another option is that I create mappings in regions that malloc already planned to use.

Unfortunately, I am not able to guarantee 100% identical malloc/free behavior in all processes before I establish the mmap-mappings.

This leads me to give the MAP_FIXED flag to mmap. The first process is using 0x0 as base address to ensure that the mapping range is at least somehow reasonable, but that does not seem to transfer to other processes. (The binary is also linked with -Wl,-no_pie.)

I tried to figure out whether I could query the system to know which pages it plans to use for malloc by reading up on malloc_default_zone, but that API does not seem to offer what I need.

Is there any way to ensure that malloc is not using particular memory pages/address ranges?

(It needs to work on OSX. Linux tips, which guide me in the right direction are appreciate, too.)

Answer 1

I notice this in the mmap documentation:

If MAP_FIXED is specified, a successful mmap deletes any previous mapping in the allocated address range

However, malloc won’t use map fixed, so as long as you get in before malloc, you’d be okay: you could test whether a region is free by first trying to map it without MAP_FIXED, and if that succeeds at the same address (which it will do if the address is free) then you can remap with MAP_FIXED knowing that you’re not choosing a section of address space that malloc had already grabbed

The only guaranteed way to guarantee that the same block of logical memory will be available in two processes is to have one fork from the other.

However, if you’re compiling with 64-bit pointers, then you can just pick an (unusual) region of memory, and hope for the best, since the chance of collision is tiny.

See also this question about valid address spaces.