Question

I am having stability problems with OpenCMS. When i do a thread dump, many threads (400) are waiting on the synchronized (m_processingFiles) block in the following code:

public class CmsJspLoader ... {
...

private static Set m_processingFiles = Collections.synchronizedSet(new HashSet());

...
...
...
public String updateJsp(...) {
....

while (m_processingFiles.contains(jspVfsName)) {
    // wait a little bit until the first thread finishes
    try {
        synchronized (m_processingFiles) {
            m_processingFiles.wait(100);
        }
    } catch (InterruptedException e) {
        // ignore
    }
}
...
}
...
}

The code is part of OpenCMS.
There is no notify() anywhere in the code. There is no change of state or reading shared variable inside the shown sync block.
However, there are 400 threads waiting on it, that means, just to pass through this sync the last one should wait 40sec!!!

I simply do not understand the purpose of it. Is there something i am not seeing?

Answer 1

There must be some place in the code where a thread adds jspVfsName to m_processingFiles, does some more work, and then removes jspVfsName from m_processingFiles. If this wasn’t the case, then your other threads would simply wait() forever in that while loop. For some reason, the implementer didn’t want any other threads to execute updateJsp while that other processing is going on.

I recommend you examine the code to see what jspVfsName actually is, and find where in the code it could be added/removed from m_processingFiles. Perhaps then you will also understand why the author didn’t want updateJsp to run while jspVfsName is in m_processingFiles.

Once you find that, you can examine that “other” code to see if jspVfsName could ever be added to m_processingFiles and never removed. If so, that would (naturally) cause livelock, which would explain your stability problems.

Or could it be that updateJsp is being called very frequently, and the “other” code which modifies m_processingFiles is also being called very frequently, to the point that it causes a major concurrency bottleneck? Could there be something wrong with your app which is causing updateJsp to be called more often than it should (perhaps on every request, rather than every time new JSP files are placed on the server)?

If updateJsp is running very frequently, but it’s not caused by a problem with your app, you could try simply shorten the wait() period. That shouldn’t hurt anything — it will just make the waiting threads check whether jspVfsName is still in m_processingFiles more frequently. 100ms is a long, long time in CPU terms!