Question

for the last four days i am facing a problem on my production server where Tomcat stops responding, and when i try to shut it down via shutdow.sh tomcat process stays alive. i will have to kill the process and start it again.
the below stack is logged directly before tomcat crashes and stops responding. i did lots of research but couldn’t solve the problem just yet.

any help is appreciated

there are two SEVERE web application errors

Jun 9, 2012 4:50:08 PM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc
SEVERE: The web application [/beta] registered the JDBC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
Jun 9, 2012 4:50:08 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/beta] appears to have started a thread named [MultiThreadedHttpConnectionManager cleanup] but has failed to stop it. This is very likely to create a memory leak.

and one MySQL error

INFO: Illegal access: this web application instance has been stopped already.  Could not load com.mysql.jdbc.SQLError.  The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
    at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1587)
    at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1546)
    at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3358)
    at com.mysql.jdbc.MysqlIO.quit(MysqlIO.java:1695)
    at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4388)
    at com.mysql.jdbc.ConnectionImpl.cleanup(ConnectionImpl.java:1368)
    at com.mysql.jdbc.ConnectionImpl.finalize(ConnectionImpl.java:2737)
    at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method)
    at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:83)
    at java.lang.ref.Finalizer.access$100(Finalizer.java:14)
    at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:160)

full stack below

INFO: Destroying Spring FrameworkServlet 'springMvcServlet'
Jun 9, 2012 4:50:08 PM org.apache.catalina.core.ApplicationContext log
INFO: Closing Spring root WebApplicationContext
Jun 9, 2012 4:50:08 PM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc
SEVERE: The web application [/beta] registered the JDBC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
Jun 9, 2012 4:50:08 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/beta] appears to have started a thread named [MultiThreadedHttpConnectionManager cleanup] but has failed to stop it. This is very likely to create a memory leak.
Jun 9, 2012 4:50:09 PM org.apache.catalina.loader.WebappClassLoader validateJarFile
INFO: validateJarFile(/home/bratecp/public_html/beta/WEB-INF/lib/ImageEditor.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class
Jun 9, 2012 4:50:09 PM org.apache.catalina.loader.WebappClassLoader validateJarFile
INFO: validateJarFile(/home/bratecp/public_html/beta/WEB-INF/lib/gwt-user.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class
Jun 9, 2012 4:50:09 PM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring root WebApplicationContext
Jun 9, 2012 4:50:11 PM org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already.  Could not load com.mysql.jdbc.SQLError.  The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
    at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1587)
    at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1546)
    at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3358)
    at com.mysql.jdbc.MysqlIO.quit(MysqlIO.java:1695)
    at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4388)
    at com.mysql.jdbc.ConnectionImpl.cleanup(ConnectionImpl.java:1368)
    at com.mysql.jdbc.ConnectionImpl.finalize(ConnectionImpl.java:2737)
    at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method)
    at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:83)
    at java.lang.ref.Finalizer.access$100(Finalizer.java:14)
    at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:160)
Jun 9, 2012 4:58:44 PM org.apache.catalina.core.AprLifecycleListener init
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/local/jdk1.6.0_29/jre/lib/amd64/server:/usr/local/jdk1.6.0_29/jre/lib/amd64:/usr/local/jdk1.6.0_29/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
Jun 9, 2012 4:58:44 PM org.apache.catalina.startup.SetAllPropertiesRule begin
WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'compression' to 'on' did not find a matching property.
Jun 9, 2012 4:58:44 PM org.apache.catalina.startup.SetAllPropertiesRule begin
WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'compressionMinSize' to '2048' did not find a matching property.
Jun 9, 2012 4:58:44 PM org.apache.catalina.startup.SetAllPropertiesRule begin
WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'noCompressionUserAgents' to 'gozilla, traviata' did not find a matching property.
Jun 9, 2012 4:58:44 PM org.apache.catalina.startup.SetAllPropertiesRule begin
WARNING: [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'compressableMimeType' to 'text/xml,text/plain,application/json,application/javascript,text/css' did not find a matching property.
Jun 9, 2012 4:58:44 PM org.apache.tomcat.util.digester.SetPropertiesRule begin
WARNING: [SetPropertiesRule]{Server/Service/Engine/Host/Context} Setting property 'debug' to '1' did not find a matching property.
Jun 9, 2012 4:58:44 PM org.apache.tomcat.util.digester.SetPropertiesRule begin
WARNING: [SetPropertiesRule]{Server/Service/Engine/Host/Context} Setting property 'debug' to '0' did not find a matching property.
Jun 9, 2012 4:58:44 PM org.apache.tomcat.util.digester.SetPropertiesRule begin
WARNING: [SetPropertiesRule]{Server/Service/Engine/Host/Context} Setting property 'debug' to '1' did not find a matching property.
Jun 9, 2012 4:58:45 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-8080"]
Jun 9, 2012 4:58:45 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["ajp-bio-8009"]
Jun 9, 2012 4:58:45 PM org.apache.catalina.startup.Catalina load

Answer 1

There are at least two problems occurring simultaneously:

1. You are registering a JDBC driver that is being loaded from within your webapp’s WEB-INF/lib directory and failing to deregister the driver when your webapp shuts down. You can deregister a JDBC driver easily using java.sql.DriverManager.deregisterDriver() from a ServletContextListener.

2. You (likely) have a non-daemon thread outliving your webapp. The only way to determine what is going on here is to take a thread dump to find out which thread is still alive (after you run shutdown.sh and wait maybe 5 seconds for everything to settle down) and where it may have come from. Basically, anywhere you start a thread in your webapp needs to have a symmetrical stop-thread when your webapp shuts down. Remember that certain operations start threads without you realizing it (for instance, creating a TimerTask, performing certain AWT-related operations, etc.).

You probably want to fix both of these problems to improve the stability of your application server.

In your stack trace above, it’s a Connection finalizer that is trying to “really” close the connection to the database. I wonder if you aren’t properly shutting-down your Connection objects (or a connection pool) before your webapp shuts down, and so the Connection finalizers are running after they can actually accomplish their goals (after the ClassLoader is dead).