I am doing web crawling on a Sun server with 32 virtual processors and 32GB memory.
I opened 1460 threads to do the job for me. The runtime parameters I set were -Xms2048 and -Xmx2048. I have run the code twice, but it crashed at different points.
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=0xff390f38, pid=3053, tid=7640
> #
> # JRE version: 6.0_15-b03
> # Java VM: Java HotSpot(TM) Server VM (14.1-b02 mixed mode solaris-sparc )
> # Problematic frame:
> # C [libc_psr.so.1+0xf38] memset+0x78
> #
> # If you would like to submit a bug report, please visit:
> # http://java.sun.com/webapps/bugreport/crash.jsp
> # The crash happened outside the Java Virtual Machine in native code.
> # See problematic frame for where to report the bug.
> #
>
EDIT: I reduced the number of threads to 40 and ran it on the same server. It crashed again at the point the value of RSS exceeded the value of Swap(both of which were around 2150M). In other words, it crashed when the memory went beyond the limit. Then I ran it on my own PC with 4G RAM and dual core processor. To my surprise, it has been doing well so far. The memory usage on the PC is kept around 1.5G and a little far from the limit. It has been so steadily running that it seems there is a mechanism on the PC that prevents the memory from reaching its limit. In contrast, it seemed to go out of control on the Sun server.
EDIT: It hasn’t crashed so far since I upgraded to the latest 64-bit Java.
Have you tried appending an ‘M’ to memory parameters? (i.e. -Xms2048M) Also, I would try setting ms to a smaller value (i.e. 1024M) in case the VM can’t reserve enough space for the heap.