I have two virtual machines with seemingly identical configuration (created from the same VM template, and diff’ing the Tomcat folder yielded no difference apart from host name configurations). On both machines, the same web applications are deployed.
One machine works fine. On the other machine, I get errors because classes are not found. I restarted tomcat 3 times, and always get a different error, also in different applications. One such error is this:
java.lang.NoClassDefFoundError: org/springframework/web/context/request/ServletRequestAttributes
org.springframework.web.context.request.RequestContextListener.requestInitialized(RequestContextListener.java:64)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:776)
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:705)
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:898)
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
java.lang.Thread.run(Thread.java:662)
In every occurence of this error so far, the missing class (e.g. ServletRequestAttributes) resides in the same jar as the class that needs it (e.g. RequestContextListener).
I am using CentOS, Apache 2.2.3, Tomcat 6.0.30 and Oracle JDK 1.6.0_24 on both machines. Tomcat is configured with plenty heap and perm gen space (also identical on both machines).
It seems that Tomcat 6 "losing" classes in production is a similar problem, but without solution so far.
One possible problem named there is “you may have reached the max number of open file handles in the OS”. With rudimentary Linux knowledge, I managed to execute some commands found on the internet, which tell me that
- max file handles is 6815744 on both machines (
cat /proc/sys/fs/file-max), - current usage is 10710 on the failing and 10200 on the correct machine (
cat /proc/sys/fs/file-nr), - max file handles per process is 1024 (
ulimit -n), - current usage by Tomcat is 641 on the failing and 686 on the correct machine (
lsof -p <TOMCAT-PID> | wc -l).
UPDATE: I restarted the failing Tomcat, and opened all applications at once in a browser. The file usage of Tomcat went up to 1926. Now I see a NoClassDefFoundError in almost every application, and I also see this message in one log file:
2011-12-12 18:38:36,225 ERROR [TP-Processor3] [rplansecurity] org.jasig.cas.client.validation.Cas20ServiceTicketValidator java.net.SocketException: Too many open files
java.net.SocketException: Too many open files
at java.net.Socket.createImpl(Socket.java:397) ~[na:1.6.0_24]
I did the same on the correct Tomcat, and it failed, too. So it seems there is no difference in the two machines here: Both seem to open too many files.
Max number of open files exceeded would be my first thought. I also think that ‘ulimit -n’ shows max file handles per user and not per process, but I might be wrong. Keep in mind that Tomcat might run under different user as well.
I would try to start tomcat with ‘-verbose:class’ to see which classes and from where are being loaded to try to see if there are any sort of pattern there.