Question

I have a C++ process running in Solaris which creates 3 threads to do some tasks.
These threads execute in loops and it runs as long as the process is running.

But, I see that the memory usage of the process grows continuously and the process core dumps once the memory usage exceeds 4GB.

Can someone give me some pointers on what could be the issue behind memory usage growth?
What can I do to prevent process from core dumping because of memory exhaustion?

Will thread restart help?

Any pointers welcome.

Answer 1

No, restarting a thread would not help.

It seems like you have a memory leak in your application.

In my experience there are two types of memory leaks:

• real memory leaks that you can see when the application exits
• ‘false’ memory leaks, like a big list that increases during the lifetime of your application but which is correctly cleaned up at the end

For the first type, there are tools which can report the memory that has not been freed by your application when it exits. I don’t know about Solaris but there are numerous tools under Windows which can do that. For Unix, I think that Valgrind does this.

For the second type, there are also tools under Windows that can take snapshots of the memory of your application. Simply take two snapshots with an interval of a few minutes or hours (depending on your application) and let them compare by the tool. There are probably simlar tools like this on Solaris.

Using these tools will probably require your application to take much more memory, since the tool needs to store the call stack of every memory allocation. Because of this it will also run much slower. However, you will only see this effect when you are actively using this tool, so there is no effect in real-life production code.

So, just look for this kind of tools under Solaris. I quickly Googled for it and found this link: http://prefetch.net/blog/index.php/2006/02/19/finding-memory-leaks-on-solaris-systems/. This could be a starting point.

EDIT: Some additional information: are you looking at the right kind of memory? Even if you only allocated 3GB in total, the total virtual address space may still reach 4GB because of memory fragmentation. Unfortunately, there is nothing you can do about this (except using another memory allocation strategy).