So, I am running a web server using embedded jetty and the maven app-assembler plugin. When I invoke the app-assembler app directly (running as root), the server starts up fine. Even with running under nohup, the web server starts up completely fine.
When I register the app-assembler script as an init.d script (Runlevels 235 / Start pri 98 / Stop pri 20 ), and run the init.d script using the service command, the web server hangs during startup.
The place where the application hangs is during classpath scanning in the Struts 2 convention plugin.
If I kill the web server using ‘kill -QUIT’, the code looks like it is stuck in Unix filesystem:
"main" prio=10 tid=0x09add800 nid=0xb33 runnable [0xb7361000]
java.lang.Thread.State: RUNNABLE
at java.io.UnixFileSystem.getBooleanAttributes0(Native Method)
at java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:228)
at java.io.File.isDirectory(File.java:754)
I have done some research, and it seems that there might be performance issues with NFS, java, and filesystem access.
The problem I am having is that we aren’t running NFS on our server, and that as far as I know we are running the app in the same way, by the same user.
Even when mimmicking the way service command works (running ‘env -i’ before the script starts), running the script directly doesn’t hang.
Looking for any advice or pointers to resolving this issue, or any way to work around it.
Update (4/8/12)
I am using Sun(Oracle) JDK 1.6.0_31
/usr/java/latest/bin/java -version
java version "1.6.0_31"
Java(TM) SE Runtime Environment (build 1.6.0_31-b04)
Java HotSpot(TM) Server VM (build 20.6-b01, mixed mode)
So, I figured out the issue. It looks like the issues I was seeing with the threads being stuck in filesystem operations, is due to recursive symlinks, that Java will follow indefinitely. Any scan from the root of a Unix system, following symlinks, should result in the same problem (100% CPU usage, and the file scanning hanging).
The issue ultimately was that the service script ran from root, where calling the script directly started from my current working directory. I modified my startup script to move the root of the app assembled script before running, and all is well.