Question

I’ve got a Java webapp running on one tomcat instance. During peak times the webapp serves around 30 pages per second and normally around 15.

My environment is:

O/S: SUSE Linux Enterprise Server 10 (x86_64)
RAM: 16GB

server: Tomcat 6.0.20
JVM: Java HotSpot(TM) 64-Bit Server VM 1.6.0_14
JVM options:
CATALINA_OPTS="-Xms512m -Xmx1024m -XX:PermSize=128m -XX:MaxPermSize=256m
               -XX:+UseParallelGC
               -Djava.awt.headless=true
               -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps"
JAVA_OPTS="-server"

After a couple of days of uptime the Full GC starts occurring more frequently and it becomes a serious problem to the application’s availability. After a tomcat restart the problem goes away but, of course, returns after 5 to 10 or 30 days (not consistent).

The Full GC log before and after a restart is at http://pastebin.com/raw.php?i=4NtkNXmi

It shows a log before the restart at 6.6 days uptime where the app was suffering because Full GC needed 2.5 seconds and was happening every ~6 secs.

Then it shows a log just after the restart where Full GC only happened every 5-10 minutes.

I’ve got two dumps using jmap -dump:format=b,file=dump.hprof PID when the Full GCs where occurring (I’m not sure whether I got them exactly right when a Full GC was occurring or between 2 Full GCs) and opened them in http://www.eclipse.org/mat/ but didn’t get anything useful in Leak Suspects:

• 60MB: 1 instance of “org.hibernate.impl.SessionFactoryImpl” (I use hibernate with ehcache)
• 80MB: 1,024 instances of “org.apache.tomcat.util.threads.ThreadWithAttributes” (these are probably the 1024 workers of tomcat)
• 45MB: 37 instances of “net.sf.ehcache.store.compound.impl.MemoryOnlyStore” (these should be my ~37 cache regions in ehcache)

Note that I never get an OutOfMemoryError.

Any ideas on where should I look next?

Answer 1

I’ve switched from -Xmx1024m to -Xmx2048m and the problem went away. I now have 100 days of uptime.