So I was monitoring a process using VisualVM and I clicked on “Perform GC”, the process then died with error:
#
# A fatal error has been detected by the Java Runtime Environment:
#
# java.lang.OutOfMemoryError: requested 4088 bytes for char in /BUILD_AREA/jdk6_24/hotspot/src/share/vm/utilities/stack.inline.hpp. Out of swap space?
#
# Internal Error (allocation.inline.hpp:39), pid=1148, tid=1082808672
# Error: char in /BUILD_AREA/jdk6_24/hotspot/src/share/vm/utilities/stack.inline.hpp
#
# JRE version: 6.0_24-b07
# Java VM: Java HotSpot(TM) 64-Bit Server VM (19.1-b02 mixed mode linux-amd64 )
# An error report file with more information is saved as:
# /net/bsrs78.pit.twosigma.com/d-1/local/eaMemory2-yao-bsrs78.pit.twosigma.com-db8rsnyc9-head-20110622-130721-31729-0/hs_err_pid1148.log
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#
But not long before I did that, I checked the machine and it seems like there’s plenty of memory left (numbers shown in MB):
free -m
total used free shared buffers cached
Mem: 96869 60999 35869 0 363 4590
-/+ buffers/cache: 56045 40823
Swap: 10243 40 10203
It doesn’t seem likely to me that GC will take up all the memory.
This is a problem because if I just let the program continue to run, at some point it will max out its allocated heap and then try to do a major GC and die the same way. Minor GC seems fine though.
So what is going on here? Why does it die with the swap space message? Please let me know if there’s anything I can do to provide more information or diagnose this problem further. I will greatly appreciate all help!
Thanks!
Edit1: I’ve uploaded the error log at http://www.mediafire.com/?kw0meup9mw5de9c, and for what it’s worth, it’s using PS Scavenge for minor GC and PS MarkSweep for major GC.
Edit2: Also, this problem has happened multiple times, consistently dying in the middle of a major garbage collection — whether induced by me via visualVM or running out of heap and being triggered by jvm itself.
Thanks for everyone’s input. It turns out that our company’s system has put a limit (via ulimit) on the amount of virtual memory of the process, which is just ~1G above the amount of heap requested. So the process was denied more memory when it maxed out this limit (apparently major GC will trigger this), even when there was still more virtual memory on the machine (the “out of swap” message is misleading!). Upping the virtual memory limit solved the problem.