Question

I have this java simulator that will need to handle a huge amount of data. It works fine, but one I get up to an array of int[100000][100][2] along with other big arrays. the program says that its out of memory. (Java.lang.outOfMemoryError)

All fine and good, I just give it more memory, but It seems to always run out around ~300M even though I allow it 2GB. This is all from watching Task manager.

Is there something wrong with my system, or is this just a java thing that I need to deal with?

@DanielPryden

OS: Win 7 32Bit 4GB of ram on board

JVM Command: java -Xmx2048M -Xms2048M Simulator

Error Data: Had to get from an IDE (using IntelliJ). I dont know how to do it from cmd. I assume this is what you are looking for.

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at Simulator.main(Simulator.java:63)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)

Answer 1

If you’re running on a 32-bit Windows OS, it’s going to be impossible to allocate a full 2GB. Even if you muck about with Windows internals, the largest usable address space you’ll ever get is 3GB, and even then it won’t all be contiguous (and the JVM requires contiguous space to build the Java heap). In practice, with the Sun/Oracle JVM, I’ve never been able to successfully allocate a heap any bigger than about 1.5GB — and if you’re using JNI at all, the maximum possible heap is reduced by any DLLs you link in.

If you really need a big heap, I would first recommend you move to a 64-bit OS if at all possible. Secondly, as other answers here point out, if you can allocate non-contiguous memory, that will be more likely to succeed. Use a LinkedList or other structure that allocates data in separate chunks. There is a bit of a trade-off here; you probably want each chunk to contain an array that’s at least 64Kb.

Finally, you might get better results if you can find a way to split up your processing into separate processes — that is, run multiple Java instances, each with its own set of data to operate on, and use sockets or files to communicate between them. That said, with a 32-bit Windows OS, you still probably won’t be able to make use of your 4GB of RAM, let alone any larger amount.