I am running a batch engine (BE) as well as Jboss instance on an EC2 with 16GB memory. Both are managed by WrapperSimpleApp.
BE is constantly processing large volumes of the information. Just to have an idea, database grows by about 10 to 15 GB every day. From the logs the BE goes down somewhere between 1 to 7 times a day. I reduced Maximum Java Heap Size from 8GB to 4GB. It had no effect. As a last resort I bounced the EC2 server and the errors are gone away. I would like to know if there is any way to find out why JVM was not responsive. The BE is doing the same processes with the same volume of work. Is it a known issue with the EC2 server? I don’t have any evidence of BE being at fault.
Here are some of the wrapper settings:
# Initial Java Heap Size (in MB)
# wrapper.java.initmemory=256
# Maximum Java Heap Size (in MB)
wrapper.java.maxmemory=4096
wrapper.ping.timeout=600
error in log file:
INFO | jvm 6 | 2012/07/03 05:46:12 | BE is doing some stuff here.
ERROR | wrapper | 2012/07/03 05:57:14 | JVM appears hung: Timed out waiting for signal from JVM.
ERROR | wrapper | 2012/07/03 05:57:14 | JVM did not exit on request, terminated
INFO | wrapper | 2012/07/03 05:57:14 | JVM exited on its own while waiting to kill the application.
STATUS | wrapper | 2012/07/03 05:57:14 | JVM exited in response to signal SIGKILL (9).
STATUS | wrapper | 2012/07/03 05:57:19 | Launching a JVM…
INFO | jvm 7 | 2012/07/03 05:57:19 | Wrapper (Version 3.2.3) http://wrapper.tanukisoftware.org
INFO | jvm 7 | 2012/07/03 05:57:19 | Copyright 1999-2006 Tanuki Software, Inc. All Rights Reserved.
INFO | jvm 7 | 2012/07/03 05:57:19 |
INFO | jvm 7 | 2012/07/03 05:57:19 | BE continues to do stuff
Thanks everyone in advance.
Capturing a thread dump using jstack or kill -3 would help find the problem.
If you are launching a batch file instead of java.exe this might help explain the issue.
http://wrapper.tanukisoftware.com/doc/english/troubleshooting.html#10