Question

I have a Java web application running on Tomcat 7 that appears to have a memory leak. The average memory usage of the application increases linearly over time when under load (determined using JConsole). After the memory usage reaches the plateau, performance degrades significantly. Response times go from ~100ms to [300ms, 2500ms], so this is actually causing real problems.

JConsole memory profile of my application:

Using VisualVM, I see that at least half the memory is being used by character arrays (i.e. char[]) and that most (roughly the same number of each, 300,000 instances) of the strings are one of the following: “Allocation Failure”, “Copy”, “end of minor GC”, all of which seem to be related to garbage collection notification. As far as I know, the application doesn’t monitor the garbage collector at all. VisualVM can’t find a GC root for any of these strings, so I’m having a hard time tracking this down.

Memory Analyzer heap dump:

I can’t explain why the memory usage plateaus like that, but I have a theory as to why performance degrades once it does. If memory is fragmented, the application could take a long time to allocate a contiguous block of memory to handle new requests.

Comparing this to the built-in Tomcat server status application, the memory increases and levels off at, but doesn’t hit a high “floor” like my application. It also doesn’t have the high number of unreachable char[].

JConsole memory profile of Tomcat server status application:

Memory Analyzer heap dump of Tomcat server status applicationp:

Where could these strings be allocated and why are they not being garbage collected? Are there Tomcat or Java settings that could affect this? Are there specific packages that could be affect this?

Answer 1

I removed the following JMX configuration from tomcat\bin\setenv.bat:

set "JAVA_OPTS=%JAVA_OPTS% 
    -Dcom.sun.management.jmxremote=true 
    -Dcom.sun.management.jmxremote.port=9090
    -Dcom.sun.management.jmxremote.ssl=false
    -Dcom.sun.management.jmxremote.authenticate=false"

I can’t get detailed memory heap dumps anymore, but the memory profile looks much better:

24 hours later, the memory profile looks the same: