is the following method likely to cause a thread to hang which in turn causing other threads to hang that eventually building up a stack and crash the app server?
private static Hashtable content = new Hashtable();
private static int cleanout;
private static void cleanoutCache() {
if(cleanout > 50000) {
synchronized(PollCacheManager.class) {
Enumeration emu = content.keys();
while(emu.hasMoreElements()) {
String key = (String)emu.nextElement();
PollCacheStore bean = (PollCacheStore)content.get(key);
if((System.currentTimeMillis() - bean.getLastInitialized()) > 86400000) {
content.remove(key);
}
}
// reset cleanout
cleanout = 0;
}
}
}
Thanks for your help
I’ve pasted a snapshot of the thread dump generated when running my test case simulating the issue I mentioned earlier in my previous question.
All “Servlet.Engine.Transports” threads are those relevant to my application. It seems they all waiting on the monitor. But no further details were provided as to what resource they are waiting on. I’m a bit new in analysing these thread dumps. Doesn’t seemed like a deadlock, not a race condition and not resource contention. However, in the result of my load testing tool, process hanging was obvious. Thanks for your help
Full thread dump:
"Thread-1727" prio=5 tid=0x2aea620 nid=0x9a2 waiting on monitor [0xb6481000..0xb6481a00]
at java.lang.Thread.sleep(Native Method)
at com.ibm.websphere.personalization.util.timer.PznTimerEvents.run(PznTimerEvents.java:222)
at java.lang.Thread.run(Thread.java:479)
"Thread-1683" daemon prio=5 tid=0x1a21668 nid=0x973 runnable [0xb4581000..0xb4581a00]
at java.net.SocketInputStream.socketRead(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:85)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:181)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:220)
at java.io.BufferedInputStream.read(BufferedInputStream.java:275)
at com.sun.jndi.ldap.Connection.run(Connection.java:581)
at java.lang.Thread.run(Thread.java:479)
"Thread-1682" daemon prio=5 tid=0x4ed830 nid=0x972 runnable [0xb5301000..0xb5301a00]
at java.net.SocketInputStream.socketRead(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:85)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:181)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:220)
at java.io.BufferedInputStream.read(BufferedInputStream.java:275)
at com.sun.jndi.ldap.Connection.run(Connection.java:581)
at java.lang.Thread.run(Thread.java:479)
"Thread-1681" daemon prio=5 tid=0x1ec1a20 nid=0x971 runnable [0xb5401000..0xb5401a00]
at java.net.SocketInputStream.socketRead(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:85)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:181)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:220)
at java.io.BufferedInputStream.read(BufferedInputStream.java:275)
at com.sun.jndi.ldap.Connection.run(Connection.java:581)
at java.lang.Thread.run(Thread.java:479)
"Servlet.Engine.Transports : 387" daemon prio=5 tid=0x15386f8 nid=0x943 waiting on monitor [0xb4781000..0xb4781a00]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:415)
at com.ibm.ws.util.BoundedBuffer.take(BoundedBuffer.java:161)
at com.ibm.ws.util.ThreadPool.getTask(ThreadPool.java:422)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:669)
"Servlet.Engine.Transports : 385" daemon prio=5 tid=0x51e898 nid=0x93e waiting on monitor [0xb3281000..0xb3281a00]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:415)
at com.ibm.ws.util.BoundedBuffer.take(BoundedBuffer.java:161)
at com.ibm.ws.util.ThreadPool.getTask(ThreadPool.java:422)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:669)
"Servlet.Engine.Transports : 384" daemon prio=5 tid=0x464760 nid=0x93d waiting on monitor [0xb3381000..0xb3381a00]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:415)
at com.ibm.ws.util.BoundedBuffer.take(BoundedBuffer.java:161)
at com.ibm.ws.util.ThreadPool.getTask(ThreadPool.java:422)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:669)
"Servlet.Engine.Transports : 382" daemon prio=5 tid=0x1141de8 nid=0x8a0 waiting on monitor [0xb3581000..0xb3581a00]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:415)
at com.ibm.ws.util.BoundedBuffer.take(BoundedBuffer.java:161)
at com.ibm.ws.util.ThreadPool.getTask(ThreadPool.java:422)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:669)
"Servlet.Engine.Transports : 380" daemon prio=5 tid=0x1151ad8 nid=0x6b5 waiting on monitor [0xb3e81000..0xb3e81a00]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:415)
at com.ibm.ws.util.BoundedBuffer.take(BoundedBuffer.java:161)
at com.ibm.ws.util.ThreadPool.getTask(ThreadPool.java:422)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:669)
"Servlet.Engine.Transports : 366" daemon prio=5 tid=0x1a1d110 nid=0x3fb waiting on monitor [0xb4b81000..0xb4b81a00]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:415)
at com.ibm.ws.util.BoundedBuffer.take(BoundedBuffer.java:161)
at com.ibm.ws.util.ThreadPool.getTask(ThreadPool.java:422)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:669)
"Servlet.Engine.Transports : 365" daemon prio=5 tid=0x4e8bd8 nid=0x3fa waiting on monitor [0xb6281000..0xb6281a00]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:415)
at com.ibm.ws.util.BoundedBuffer.take(BoundedBuffer.java:161)
at com.ibm.ws.util.ThreadPool.getTask(ThreadPool.java:422)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:669)
"Servlet.Engine.Transports : 362" daemon prio=5 tid=0x17055b0 nid=0x3f7 waiting on monitor [0xb3481000..0xb3481a00]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:415)
at com.ibm.ws.util.BoundedBuffer.take(BoundedBuffer.java:161)
at com.ibm.ws.util.ThreadPool.getTask(ThreadPool.java:422)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:669)
"Servlet.Engine.Transports : 356" daemon prio=5 tid=0x1ddbae0 nid=0x3f1 waiting on monitor [0xb9c01000..0xb9c01a00]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:415)
at com.ibm.ws.util.BoundedBuffer.take(BoundedBuffer.java:161)
at com.ibm.ws.util.ThreadPool.getTask(ThreadPool.java:422)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:669)
"Servlet.Engine.Transports : 299" daemon prio=5 tid=0x2519028 nid=0x3b5 waiting on monitor [0xb6001000..0xb6001a00]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:415)
at com.ibm.ws.util.BoundedBuffer.take(BoundedBuffer.java:161)
at com.ibm.ws.util.ThreadPool.getTask(ThreadPool.java:422)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:669)
"SoapConnectorThreadPool : 3" daemon prio=5 tid=0x15d49f0 nid=0x1ae waiting on monitor [0xb2e81000..0xb2e81a00]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:415)
at com.ibm.ws.util.BoundedBuffer.take(BoundedBuffer.java:161)
at com.ibm.ws.util.ThreadPool.getTask(ThreadPool.java:422)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:669)
"Alarm : 3" daemon prio=5 tid=0x1d24c48 nid=0xa5 waiting on monitor [0xb6381000..0xb6381a00]
at java.lang.Object.wait(Native Method)
at com.ibm.ws.util.BoundedBuffer.poll(BoundedBuffer.java:192)
at com.ibm.ws.util.ThreadPool.getTask(ThreadPool.java:422)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:669)
"Keep-Alive-Timer:" daemon prio=8 tid=0x1ec9a38 nid=0x99 waiting on monitor [0xb3f81000..0xb3f81a00]
at java.lang.Thread.sleep(Native Method)
at sun.net.www.http.KeepAliveCache.run(KeepAliveCache.java:130)
at java.lang.Thread.run(Thread.java:479)
"Thread-64" daemon prio=5 tid=0x4ee700 nid=0x98 waiting on monitor [0xb4081000..0xb4081a00]
at java.lang.Thread.sleep(Native Method)
at com.ibm.ejs.j2c.poolmanager.TaskTimer.run(TaskTimer.java:119)
"RT=7:P=908030:O=0:WSTCPTransportConnection[addr=10.24.189.74,port=47148,local=9812]" daemon prio=5 tid=0x4c73f8 nid=0x97 runnable [0xb4181000..0xb4181a00]
at java.net.SocketInputStream.socketRead(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:85)
at com.ibm.rmi.iiop.Connection.readMoreData(Connection.java:909)
at com.ibm.rmi.iiop.Connection.createInputStream(Connection.java:742)
at com.ibm.rmi.iiop.Connection.doReaderWorkOnce(Connection.java:2447)
at com.ibm.rmi.transport.ReaderThread.run(ReaderPoolImpl.java:138)
"RT=6:P=908030:O=0:WSTCPTransportConnection[addr=10.24.189.74,port=9812,local=47148]" daemon prio=5 tid=0x3b7860 nid=0x96 runnable [0xb4281000..0xb4281a00]
at java.net.SocketInputStream.socketRead(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:85)
at com.ibm.rmi.iiop.Connection.readMoreData(Connection.java:909)
at com.ibm.rmi.iiop.Connection.createInputStream(Connection.java:742)
at com.ibm.rmi.iiop.Connection.doReaderWorkOnce(Connection.java:2447)
at com.ibm.rmi.transport.ReaderThread.run(ReaderPoolImpl.java:138)
"Thread-63" daemon prio=5 tid=0x1666990 nid=0x91 waiting on monitor [0xb4681000..0xb4681a00]
at java.lang.Thread.sleep(Native Method)
at com.ibm.ws.management.RoutingTable$PingThread.run(RoutingTable.java:1025)
"ProcessDiscovery : 0" daemon prio=5 tid=0x1c7b4c8 nid=0x8b waiting on monitor [0xb4881000..0xb4881a00]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:415)
at com.ibm.ws.util.BoundedBuffer.take(BoundedBuffer.java:161)
at com.ibm.ws.util.ThreadPool.getTask(ThreadPool.java:422)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:669)
"SoapConnectorThreadPool : 2" daemon prio=5 tid=0x1b82798 nid=0x8a waiting on monitor [0xb4981000..0xb4981a00]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:415)
at com.ibm.ws.util.BoundedBuffer.take(BoundedBuffer.java:161)
at com.ibm.ws.util.ThreadPool.getTask(ThreadPool.java:422)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:669)
"SoapConnectorThreadPool : 1" daemon prio=5 tid=0x1505188 nid=0x89 waiting on monitor [0xb4a81000..0xb4a81a00]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:415)
at com.ibm.ws.util.BoundedBuffer.take(BoundedBuffer.java:161)
at com.ibm.ws.util.ThreadPool.getTask(ThreadPool.java:422)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:669)
"ORB.thread.pool : 1" daemon prio=5 tid=0x24cfd20 nid=0x83 waiting on monitor [0xb4c81000..0xb4c81a00]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:415)
at com.ibm.ws.util.BoundedBuffer.take(BoundedBuffer.java:161)
at com.ibm.ws.util.ThreadPool.getTask(ThreadPool.java:422)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:669)
"ORB.thread.pool : 0" daemon prio=5 tid=0x15025c0 nid=0x82 waiting on monitor [0xb4d81000..0xb4d81a00]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:415)
at com.ibm.ws.util.BoundedBuffer.take(BoundedBuffer.java:161)
at com.ibm.ws.util.ThreadPool.getTask(ThreadPool.java:422)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:669)
"RT=5:P=908030:O=0:WSSSLTransportConnection[addr=10.25.168.38,port=33762,local=46854]" daemon prio=5 tid=0x1520f20 nid=0x81 runnable [0xb4e81000..0xb4e81a00]
at java.net.SocketInputStream.socketRead(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:85)
at com.ibm.sslite.t.a(Unknown Source)
at com.ibm.sslite.t.b(Unknown Source)
at com.ibm.sslite.t.a(Unknown Source)
at com.ibm.sslite.a.read(Unknown Source)
at com.ibm.jsse.a.read(Unknown Source)
at com.ibm.rmi.iiop.Connection.readMoreData(Connection.java:909)
at com.ibm.rmi.iiop.Connection.createInputStream(Connection.java:742)
at com.ibm.rmi.iiop.Connection.doReaderWorkOnce(Connection.java:2447)
at com.ibm.rmi.transport.ReaderThread.run(ReaderPoolImpl.java:138)
"Thread-53" prio=5 tid=0x29558 nid=0x1 waiting on monitor [0..0xffbedb10]
"java.net.MulticastSocket@4b3966" prio=5 tid=0x17dacf8 nid=0x80 runnable [0xb4f81000..0xb4f81a00]
at java.net.PlainDatagramSocketImpl.receive(Native Method)
at java.net.DatagramSocket.receive(DatagramSocket.java:387)
at com.ibm.ws.management.discovery.transport.MulticastServer.run(MulticastServer.java:196)
at java.lang.Thread.run(Thread.java:479)
"ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=9044]" prio=5 tid=0x157f330 nid=0x7f runnable [0xb5981000..0xb5981a00]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:463)
at java.net.ServerSocket.implAccept(ServerSocket.java:238)
at java.net.ServerSocket.accept(ServerSocket.java:217)
at com.ibm.jsse.bg.accept(Unknown Source)
at com.ibm.ws.http.HttpTransport.run(HttpTransport.java:235)
at java.lang.Thread.run(Thread.java:479)
"ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=9091]" prio=5 tid=0x15d10c0 nid=0x7e runnable [0xb5a81000..0xb5a81a00]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:463)
at java.net.ServerSocket.implAccept(ServerSocket.java:238)
at java.net.ServerSocket.accept(ServerSocket.java:217)
at com.ibm.ws.http.HttpTransport.run(HttpTransport.java:235)
at java.lang.Thread.run(Thread.java:479)
"ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=9444]" prio=5 tid=0x1693c28 nid=0x7d runnable [0xb5b81000..0xb5b81a00]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:463)
at java.net.ServerSocket.implAccept(ServerSocket.java:238)
at java.net.ServerSocket.accept(ServerSocket.java:217)
at com.ibm.jsse.bg.accept(Unknown Source)
at com.ibm.ws.http.HttpTransport.run(HttpTransport.java:235)
at java.lang.Thread.run(Thread.java:479)
"ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=9081]" prio=5 tid=0x24d2f78 nid=0x7c runnable [0xb5e01000..0xb5e01a00]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:463)
at java.net.ServerSocket.implAccept(ServerSocket.java:238)
at java.net.ServerSocket.accept(ServerSocket.java:217)
at com.ibm.ws.http.HttpTransport.run(HttpTransport.java:235)
at java.lang.Thread.run(Thread.java:479)
"Dispatcher-Thread-52" daemon prio=1 tid=0x509da0 nid=0x7b waiting on monitor [0xb5f01000..0xb5f01a00]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:415)
at org.apache.log4j.Dispatcher.run(AsyncAppender.java:293)
"Thread-51" prio=5 tid=0x254138 nid=0x7a waiting on monitor [0xb5c81000..0xb5c81a00]
at java.lang.Thread.sleep(Native Method)
at com.ibm.wcm.jobs.Scheduler.run(Scheduler.java:68)
at java.lang.Thread.run(Thread.java:479)
"wcp_1328597020006" prio=5 tid=0x2a5c628 nid=0x76 waiting on monitor [0xb6101000..0xb6101a00]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:415)
at com.ibm.wcp.analysis.util.MultiAccessQueue.dequeue(MultiAccessQueue.java:223)
at com.ibm.wcp.analysis.event.ListenerThread.run(ListenerThread.java:87)
…..more (due to character limit here)
You should move your “cleanout greater than 50000” test inside the synchronzed block to ensure all threads see the most up-to-date version of its value, that is modified at the end of the method.
The rest looks OK, provided all other access to the “content” and “cleanout” variables are also protected by the “PollCacheManager.class” monitor (reads included).