Question

I’m learning and testing some graph libraries and am running into a weird problem(but this isn’t specific to graphs, I am pretty sure this is general Java related). I’m getting:’java.lang.OutOfMemoryError: GC overhead limit exceeded’. I understand this error means that garbage collecting spending most of the cpu time and not returning any memory but I’m not sure how to work around this.

Basically I(for learning purposes) want to see how long it takes to create a large number of graphs node in memory. My system is running cents and I have 7 gigs of ram but the program never exceeds more than 25%(I can see via ‘top’) even though I’m giving it all the memory in the system by running ‘java -jar jungtester.jar -Xmx7g -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit'(jungtester.jar is my program). It seems its not using all the available memory and dies after about 3.5 million nodes, which is weird because its just a for loops so I thought it would just be chugging along adding nodes until the memory is full.

I’m pretty new to the inner workings of the JVM so any suggestions of how to get past this would be great.

If it helps, here’s the code:

import edu.uci.ics.jung.graph.DirectedGraph;
import edu.uci.ics.jung.graph.DirectedSparseGraph;

public class main {

    /**
     * @param args
     */
    public static void main(String[] args) {
        // TODO Auto-generated method stub
        System.out.println("starting...");
        long startTime = System.currentTimeMillis();
        DirectedGraph<Integer,Integer> graph = new DirectedSparseGraph<Integer, Integer>();;
        graph.addVertex(1);
        graph.addVertex(2);
        graph.addEdge(1, 1,2);

        for (int i = 0; i < 1000000; i++) {
            graph.addVertex(i);
            //System.out.println(i + " means we are " + (float) i/1000000 + "% done.");
        }
        long endTime1 = System.currentTimeMillis();
        System.out.println("done adding 1000000 in " + (endTime1 - startTime));


        for (int i = 1000001; i < 10000000; i++) {
            graph.addVertex(i);
            graph.addEdge(i, i, i-1000000);
            System.out.println(i + " means we are " + (float) i/1000000000 + "% done.");
        }

        long endTime = System.currentTimeMillis();

        System.out.println("It took " + (endTime - startTime));
    }

}

UPDATE: I got it to work, I’m not sure why but order matters. My above command didn’t take anything after I put -jar, but when I added the -jar to the end it seems to work.

Answer 1

I think that what is happening here is an unfortunate consequence of the nature of your program and the GC settings that you have chosen

Basically, your program is creating huge numbers of nodes very quickly, and (at looks like) never releasing any of them until the very end of the run. So each time that the GC runs, it has to trace and evacuate essentially every object in the “from” space that it is working on. This work is proportional to the number of objects in the space. As your application progresses, the spaces keep getting bigger and the numbers of objects keep getting bigger, and a larger and larger percentage of your time is spent by the GC moving the objects around. This is possibly exacerbated by the fact that you are running the CMS collector which has greater overheads than the throughput collector.

This possibly explains why you only appear to be using 25% of the available memory. But it is also possible that the numbers you are getting from top are misleading. I would be more inclined to believe numbers that you get from jconsole, et al.

I’d also turn on GC logging, look to see if there is something strange going on. For instance, your application’s behaviour may be overwhelming the CMS collector, and causing the JVM to switch to the stop-the-world GC. (I vaguely recall hearing that you take a big performance hit when this happens, and that could be enough to cause the JVM to hit the GC overhead limit.)

---

So what could you do to improve the behaviour of the JVM running this application. I’d suggest the following:

• Try using the throughput collector instead of CMS.
• If you have a JVM that is capable, try using the new G1 collector.
• Set the initial heap size to 7gigabytes too: use -Xms7g.

A careful analysis of the GC logs may be able to suggest other things to try.

But probably the best thing you could do is to ditch this (I hope) unrealistic benchmark.