My application loads a data set of approx. 85bm to 100mb each time. The application’s memory limit is set to 512mb, and this is, theoretically, more than enough.
However, I found that if, in a single run of the application, I opened and closed the data set 5 times, the total memory consumption steadily increases, until I get an out-of-memory error:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6882 bguiz 20 0 679m 206m 19m S 30 13.7 0:30.22 java
6882 bguiz 20 0 679m 259m 19m S 9 17.2 0:55.53 java
6882 bguiz 20 0 679m 301m 19m S 9 20.0 1:20.04 java
6882 bguiz 20 0 679m 357m 19m S 33 23.7 1:44.74 java
6882 bguiz 20 0 679m 395m 19m S 80 26.2 2:10.31 java
Memory grew from ~14% to ~26%. It looks like a memory leak.
What’s happening is that the top level data that is being loaded is used to populate collections such as maps and lists, and then the more detailed data is used to create sub-objects of these top-level objects, and then they in turn create sub-sub-objects.
When the data set is closed, currently the application does indeed make an attempt to clear its tracks by de-populating the various collections of objects, and then explicitly calling System.gc();
Anyhow, this is the state of the application when I got to it (several years in the making before me), and I have been assigned this task.
What I need to do, is to find a way to find which sub-objects and sub-sub-objects are still referencing each other after the data set is unloaded, and rectify them.
Obviously this can be done manually, but would be very very tedious, but I felt it would be a much better option to do this by memory profiling, something which I haven’t done before.
I have read some other SO questions that asked about which memory profiling tools to use, and I have chosen to go with the one built into Netbeans IDE, since it seemed to have good reviews, and I am working in Netbeans anyway.
Has anyone undertaken a similar Java memory profiling task before, and with hindsight:
- What specific advice would you give me?
- What techniques did you find useful in tackling this problem?
- What resources did you find useful in tackling this problem?
Edit:
This application is a standard desktop application – not a web application.
Edit: Implemented solution
Basically what worked for me was to use Netbeans’ profiler in conjunction with JHAT.
I found that the Profiler built into Netbeans IDE did a really good job of creating the memory dumps at particular profiling points, and then the tool was able to filter and sort by class and drill down the references for each instance. Which was all really good.
However, it didn’t provide me with a means to compare two heap dumps. I asked a follow up question, and it looks like JHAT (comes as part of JDK) gets that job done quite well.
Thorbjørn Ravn Andersen, Dmitry and Jason Gritman: your input was really helpful, unfortunately I can only mark 1 as the correct answer, and all of you got +1 from me anyway.
I wrote up an answer to another question about the techniques to go about finding memory leaks at https://stackoverflow.com/questions/1716597/java-memory-leak-detection-tools/1717260#1717260
If you follow my advice there, a tool like JProfiler can allow you to walk the reference graph of objects and view the deep size of those objects. This can help you find whatever object or objects are still holding onto the data.
I haven’t worked with Netbeans, so I can’t tell you how it stacks up against the other profilers I’ve used. If it doesn’t look like it has that feature you can obtain a trial version of JProfiler easily, which should last you until you’ve found your leak.