Question

I’d like to create a thread that keeps track of the memory usage and cpu usage.

If the application reaches a high level, I want to generate an heap dump or a thread dump.

Is there a way to generate a Thread dump runtime without restarting?

Answer 1

Here’s how we do it programmatically: http://pastebin.com/uS5jYpd4

We use the JMX ThreadMXBean and ThreadInfo classes:

ThreadMXBean mxBean = ManagementFactory.getThreadMXBean();
ThreadInfo[] threadInfos = mxBean.getThreadInfo(mxBean.getAllThreadIds(), 0);
...

You can also do a kill -QUIT pid under ~unix to dump the stacks to the standard-out. There is also jstack to dump the stack of a JVM.

We also have an automation which dumps the stack if the load average of the application is above some threshold:

private long lastCpuTimeMillis;
private long lastPollTimeMillis;

public void checkLoadAverage() {
    long now = System.currentTimeMillis();
    long currentCpuMillis = getTotalCpuTimeMillis();
    double loadAvg = calcLoadAveragePercentage(now, currentCpuMillis);
    if (loadAvg > LOAD_AVERAGE_DUMP_THRESHOLD) {
        try {
            dumpStack("Load average percentage is " + loadAvg);
        } catch (IOException e) {
            // Oh well, we tried
        }
    }
    lastCpuTimeMillis = currentCpuMillis;
    lastPollTimeMillis = now;
}

private long getTotalCpuTimeMillis() {
    long total = 0;
    for (long id : threadMxBean.getAllThreadIds()) {
        long cpuTime = threadMxBean.getThreadCpuTime(id);
        if (cpuTime > 0) {
            total += cpuTime;
        }
    }
    // since is in nano-seconds
    long currentCpuMillis = total / 1000000;
    return currentCpuMillis;
}

private double calcLoadAveragePercentage(long now, long currentCpuMillis) {
    long timeDiff = now - lastPollTimeMillis;
    if (timeDiff == 0) {
        timeDiff = 1;
    }
    long cpuDiff = currentCpuMillis - lastCpuTimeMillis;
    double loadAvg = (double) cpuDiff / (double) timeDiff;
    return loadAvg;
}