Question

My program use a lot of memory. This is what valgrind massif tool is showing me:

--------------------------------------------------------------------------------
  n        time(i)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)
--------------------------------------------------------------------------------
 28 38,531,086,036      760,235,208      143,002,822   617,232,386            0

As you can see extra part is few times bigger than useful heap.

What should I do to reduce this extra memory?
Do less allocations?

Is this so called memory fragmentation?

OS: Linux 2.6. Program is written in C. It should work 24\7 and it manipulate a lot of data.

Answer 1

Are you allocating lots of extremely small objects — say, just a couple of bytes? There’s a certain amount of overhead associated with every allocation (because, e.g., free needs to be able to tell how big the block was).

This is what’s sometimes called “internal fragmentation”, as opposed to “external fragmentation” where there’s a certain amount of unallocated memory but you can’t use it because it’s split up into blocks that are too small for you to use. (Another reason why malloc doesn’t ever return really small blocks is because this helps reduce external fragmentation.)

If you are allocating lots of very small objects, you should consider managing them separately rather than allocating them individually on the heap. This may well be better in other ways too (e.g., improving memory locality) if you do it right.