I am having down time problem with my web site. I have apache 2 + mod_jk + tomcat 5.5 setup on Linux Redhat. Just recently started having my website down once a day. Only after restarting tomcat the web site gets back to normal.
Application error log on tomcat(sorry, privacy info hidden in the logs):
2010-11-14 00:40:13,726 ERROR org.apache.catalina.core.StandardWrapperValve [invoke] - Servlet.service() for servlet action threw exception
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:574)
at ******.exchange.adapter.dispatcher.impl.XMLCommandDispatcher.sendCommand(XMLCommandDispatcher.java:109)
at ******.exchange.adapter.star******.api.impl.******ApiImpl.******PricingAvailability(******ApiImpl.java:356)
at ******.exchange.adapter.star******.impl.Concrete******GatewayImpl.find******Promotion(Concrete******GatewayImpl.java:123)
at ******.exchange.service.impl.Concrete******ManagerImpl.find******Promotion(Concrete******ManagerImpl.java:199)
at sun.reflect.GeneratedMethodAccessor1142.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:282)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:155)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:122)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:56)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:144)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:174)
at $Proxy16.find******Promotion(Unknown Source)
at ******.ibe.servlet.crdms.search.******SearchPromotionServlet******.handleRequest(******SearchPromotionServlet******.java:75)
at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:44)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:595)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:543)
at org.springframework.web.servlet.FrameworkServlet.serviceWrapper(FrameworkServlet.java:390)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:341)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:283)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:773)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:703)
at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:895)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
at java.lang.Thread.run(Thread.java:595)
2010-11-14 00:40:43,762 ERROR org.apache.catalina.core.ApplicationDispatcher [invoke] - Servlet.service() for servlet jsp threw exception
2010-11-14 00:40:43,805 ERROR org.apache.catalina.core.StandardWrapperValve [invoke] - Servlet.service() for servlet jsp threw exception
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:574)
at java.util.Timer.(Timer.java:154)
at java.util.Timer.(Timer.java:122)
at com.mysql.jdbc.Connection.(Connection.java:1441)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:266)
at java.sql.DriverManager.getConnection(DriverManager.java:525)
at java.sql.DriverManager.getConnection(DriverManager.java:171)
at org.apache.jsp.frameset_005f******_005fbottom_005f1_jsp._jspService(frameset_005f******_005fbottom_005f1_jsp.java:88)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:98)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:331)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:329)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:265)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:283)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:773)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:703)
at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:895)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
at java.lang.Thread.run(Thread.java:595)
2010-11-14 01:54:25,819 ERROR org.apache.catalina.core.ApplicationDispatcher [invoke] - Servlet.service() for servlet jsp threw exception
Mod_jk log entries at time when tomcat gets down:
[Sun Nov 14 00:59:52 2010] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (127.0.0.1:8009), err=-104 [Sun Nov 14 00:59:52 2010] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet) [Sun Nov 14 01:00:01 2010] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (127.0.0.1:8009), err=-104 [Sun Nov 14 01:00:01 2010] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet) [Sun Nov 14 01:00:01 2010] [error] ajp_service::jk_ajp_common.c (1758): Error connecting to tomcat. Tomcat is probably not started or is listening on the wrong port. worker=worker1 failed [Sun Nov 14 01:00:01 2010] worker1 mydomain.com 271.622400 [Sun Nov 14 01:00:01 2010] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (127.0.0.1:8009), err=-104 [Sun Nov 14 01:00:01 2010] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet) [Sun Nov 14 01:00:06 2010] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (127.0.0.1:8009), err=-104 [Sun Nov 14 01:00:06 2010] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet) [Sun Nov 14 01:00:06 2010] [error] ajp_service::jk_ajp_common.c (1758): Error connecting to tomcat. Tomcat is probably not started or is listening on the wrong port. worker=worker1 failed [Sun Nov 14 01:00:06 2010] worker1 mydomain.com 86.999946 [Sun Nov 14 01:00:06 2010] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (127.0.0.1:8009), err=-104 [Sun Nov 14 01:00:06 2010] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet) [Sun Nov 14 01:00:10 2010] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (127.0.0.1:8009), err=-104 [Sun Nov 14 01:00:10 2010] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet) [Sun Nov 14 01:00:10 2010] [error] ajp_service::jk_ajp_common.c (1758): Error connecting to tomcat. Tomcat is probably not started or is listening on the wrong port. worker=worker1 failed [Sun Nov 14 01:00:10 2010] worker1 mydomain.com 39.000187
Meesages in /var/log/messages:
Nov 14 00:59:40 nemo kernel: possible SYN flooding on port 8009. Sending cookies. Nov 14 01:00:47 nemo kernel: possible SYN flooding on port 8009. Sending cookies. Nov 14 01:01:49 nemo kernel: possible SYN flooding on port 8009. Sending cookies. Nov 14 01:02:54 nemo kernel: possible SYN flooding on port 8009. Sending cookies. Nov 14 01:04:03 nemo kernel: possible SYN flooding on port 8009. Sending cookies. Nov 14 01:05:06 nemo kernel: possible SYN flooding on port 8009. Sending cookies. Nov 14 01:06:13 nemo kernel: possible SYN flooding on port 8009. Sending cookies. Nov 14 01:07:14 nemo kernel: possible SYN flooding on port 8009. Sending cookies. Nov 14 01:08:15 nemo kernel: possible SYN flooding on port 8009. Sending cookies. Nov 14 01:09:19 nemo kernel: possible SYN flooding on port 8009. Sending cookies. Nov 14 01:10:23 nemo kernel: possible SYN flooding on port 8009. Sending cookies. Nov 14 01:11:29 nemo kernel: possible SYN flooding on port 8009. Sending cookies. Nov 14 01:12:29 nemo kernel: possible SYN flooding on port 8009. Sending cookies. Nov 14 01:13:29 nemo kernel: possible SYN flooding on port 8009. Sending cookies. Nov 14 01:14:56 nemo kernel: possible SYN flooding on port 8009. Sending cookies. Nov 14 01:16:04 nemo kernel: possible SYN flooding on port 8009. Sending cookies. Nov 14 01:17:19 nemo kernel: possible SYN flooding on port 8009. Sending cookies. Nov 14 01:18:22 nemo kernel: possible SYN flooding on port 8009. Sending cookies. Nov 14 01:19:26 nemo kernel: possible SYN flooding on port 8009. Sending cookies. Nov 14 01:20:26 nemo kernel: possible SYN flooding on port 8009. Sending cookies. Nov 14 01:21:36 nemo kernel: possible SYN flooding on port 8009. Sending cookies. Nov 14 01:22:37 nemo kernel: possible SYN flooding on port 8009. Sending cookies. Nov 14 01:23:48 nemo kernel: possible SYN flooding on port 8009. Sending cookies. Nov 14 01:24:50 nemo kernel: possible SYN flooding on port 8009. Sending cookies. Nov 14 01:25:54 nemo kernel: possible SYN flooding on port 8009. Sending cookies. Nov 14 01:26:57 nemo kernel: possible SYN flooding on port 8009. Sending cookies. Nov 14 01:29:08 nemo kernel: possible SYN flooding on port 8009. Sending cookies. Nov 14 01:30:10 nemo kernel: possible SYN flooding on port 8009. Sending cookies. Nov 14 03:00:01 nemo setroubleshoot: SELinux is preventing mysqld (mysqld_t) "getattr" to /var/lib/mysql/mysql.sock (var_lib_t). For complete SELinux messages. run sea$ Nov 14 03:00:01 nemo setroubleshoot: SELinux is preventing mysqld (mysqld_t) "getattr" to /var/lib/mysql/nemo.me-tech.com.my.err (var_lib_t). For complete SELinux mess$ Nov 14 03:00:01 nemo setroubleshoot: SELinux is preventing mysqld (mysqld_t) "read" to ./time_zone_name.frm (var_lib_t). For complete SELinux messages. run sealert -l $
Apache conf:
# prefork MPM # StartServers: number of server processes to start # MinSpareServers: minimum number of server processes which are kept spare # MaxSpareServers: maximum number of server processes which are kept spare # MaxClients: maximum number of server processes allowed to start # MaxRequestsPerChild: maximum number of requests a server process serves ‹IfModule prefork.c› StartServers 5 MinSpareServers 5 MaxSpareServers 10 MaxClients 256 MaxRequestsPerChild 0 ‹/IfModule› # worker MPM # StartServers: initial number of server processes to start # MaxClients: maximum number of simultaneous client connections # MinSpareThreads: minimum number of worker threads which are kept spare # MaxSpareThreads: maximum number of worker threads which are kept spare # ThreadsPerChild: constant number of worker threads in each server process # MaxRequestsPerChild: maximum number of requests a server process serves ‹IfModule worker.c› StartServers 2 MaxClients 256 MinSpareThreads 25 MaxSpareThreads 75 ThreadsPerChild 25 MaxRequestsPerChild 0 ‹/IfModule›
Apache Mod_jk worker.properties:
worker.list=worker1 worker.worker1.type=ajp13 worker.worker1.host=127.0.0.1 worker.worker1.port=8009 worker.worker1.lbfactor=1 worker.worker1.socket_keepalive=1 worker.worker1.recycle_timeout=180 worker.worker1.sticky_session=False
Tomcat server.xml(AJP connector for mod_jk):
‹Connector className="org.apache.coyote.tomcat5.CoyoteConnector"
port="8009"
protocol="AJP/1.3"
enableLookups="true"
redirectPort="8443"
acceptCount="10"
debug="0"
connectionTimeout="5000"
useURIValidationHack="false"
maxPostSize="10000000"
maxThreads="256"
maxSpareThreads="10"
URIEncoding="UTF-8"
protocolHandlerClassName="org.apache.jk.server.JkCoyoteHandler"/›
Any help or advise would be highly appreciated.
Thanks a lot
According to this you should try setting your JAVA_OPTS environment variable to reduce the stack and/or heap size used by Tomcat to give the OS some breathing room.