I have a jetty 7 based web app which is using up the CPU and has high response times. I want to know Where to target my optimization efforts. I collected an hprof report and it shows that most of the time is spent in EPollArrayWrapper.epollWait.
1 31.66% 31.66% 291140 301140 sun.nio.ch.EPollArrayWrapper.epollWait
2 16.18% 47.84% 148814 300461 java.net.PlainSocketImpl.socketAccept
3 8.11% 55.95% 74583 301321 java.net.SocketInputStream.socketRead0
4 6.82% 62.77% 62703 301521 java.net.SocketInputStream.socketRead0
5 5.27% 68.04% 48467 301492 sun.nio.ch.ServerSocketChannelImpl.accept0
6 3.53% 71.57% 32481 301561 java.net.SocketInputStream.read
7 3.38% 74.95% 31072 301695 sun.security.ssl.InputRecord.readFully
8 3.30% 78.26% 30390 301733 sun.security.ssl.InputRecord.read
The traces are shown below, I am not looking for an exact answer just pointers to what to look at by someone more familiar with Jetty internals.
TRACE 301140:
sun.nio.ch.EPollArrayWrapper.epollWait(EPollArrayWrapper.java:Unknown line)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:83)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
TRACE 300461:
java.net.PlainSocketImpl.socketAccept(PlainSocketImpl.java:Unknown line)
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:375)
java.net.ServerSocket.implAccept(ServerSocket.java:470)
java.net.ServerSocket.accept(ServerSocket.java:438)
TRACE 301321:
java.net.SocketInputStream.socketRead0(SocketInputStream.java:Unknown line)
java.net.SocketInputStream.read(SocketInputStream.java:146)
sun.security.ssl.InputRecord.readFully(InputRecord.java:312)
sun.security.ssl.InputRecord.read(InputRecord.java:350)
TRACE 301521:
java.net.SocketInputStream.socketRead0(SocketInputStream.java:Unknown line)
java.net.SocketInputStream.read(SocketInputStream.java:146)
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
java.io.BufferedInputStream.read(BufferedInputStream.java:254)
TRACE 301492:
sun.nio.ch.ServerSocketChannelImpl.accept0(ServerSocketChannelImpl.java:Unknown line)
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
org.eclipse.jetty.server.nio.SelectChannelConnector$2.run(SelectChannelConnector.java:287)
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:451)
I would ignore the IO blocking calls as these do not consume CPU. (They consume a little but far less than the timing might indicate)
This includes epollWait, socketAccept, socketRead0, accept0, read, write
This leaves calls to SSL. Given this is a CPU intensive task, its not surprising that it would be a significant CPU consumer.