We’re load testing a Java 1.6 application in our DEV environment. The JVM heap allocation is 2Gb, -Xms2048m -Xmx2048m. Under load testing, the app runs smooth, never uses more than 1.25Gb of heap, and garbage collection is totally normal.
In our UAT environment, we run the load test with the same parameters, the only difference is the JVM, it’s allocated 4Gb, -Xms4096m -Xmx4096m, otherwise, the hardware is exactly the same with DEV. But during load testing, the performance is horrendous, the app eats up nearly the entire heap, and garbage collection runs rampant.
We’ve run these tests over and over again, eliminated all possible symptoms that may influence performance, but the results are the same. Under what circumstances can this be the case?
The culprit could be garbage collection, normal “stop-the-world”-type collection caused us some performance problems; the server-software was running very slow, yet the load of the server was also low. Eventually we found out that there was a single “stop-the-world” -garbage collector thread holding up the entire software being run all the time under certain scenarios (operations producing loads of garbage).
Moving to concurrent garbage collection alleviated the problem with start up parameters
-XX:+UseParallelOldGC -XX:ParallelGCThreads=8. We were using “only” 2gb heaps in tests and production, but it is also worthy of noting that the amount of time the GC takes goes up with larger heap (even if your software never actually uses all of it).You might want to read more about different garbage collector -options and tuning from here: Java SE 6 HotSpot[tm] Virtual Machine Garbage Collection Tuning.
Also, answers in this question could provide some help: Java very large heap sizes.