Question

I’ve seen many question regarding the malloc memory consumption, but none of them regarding what I’m dealing with.

I working on a high memory consumption process that’s also required good performance.
The code is in C.
My intent is to allow this process to use up to 70% of the machine memory as non-pageable memory (for the performance), and anything above that as pageable memory so the OS will still have some memory to “play” with.

since I need it on run time, any profiling programs (valgrind or similar) are not relevant.

the problem is I can only track the amount of memory I was requesting but not the actual size the malloc is using. so while I’m thinking I have no more than 700Mb allocated the malloc is holding ~1Gb of memory, and since I was asking for non-pageable all that memory is non-pageable and processes are starting to die on me.

If there is a way to know how much non-pageable memory malloc is holding for my process, it will be awesome. But reading the posts here I’m guessing it’s still not supported.

I’m open to any idea or advice any one can give me.

Thanks.

Answer 1

In general, if you need fine control over memory allocation, you should allocate it explicitly with mmap() and then provide your own malloc() replacement which parcels out your region. This also saves you from the problem that, if you try to mlock stuff in the general malloc() pool, you will end up locking more than you intended anyway because the locking granularity is a 4KB page and your locked allocations will be interspersed with general libc allocations.