Question

I’m dealing with huge arrays in my application and need to resize them.

Let’s say you have an array of 2Gb and you want to resize it to 3Gb. Is there a way to resize it without needing temporarily 5Gb?

For instance, given a 1Gb heap using the -Xmx1G flag:

public class Biggy {
    public static void main(String[] args) {
        int[] array;

        array = new int[100 * 1000 * 1000]; // needs 400Mb, works
        array = null; // needed for GC
        array = new int[150 * 1000 * 1000]; // needs 600Mb, works
        array = null; // needed for GC
        array = new int[100 * 1000 * 1000]; // needs 400Mb, works
        array = Arrays.copyOf(array, 150 * 1000 * 1000); // needs 1000Mb, throws out of memory
    }
}

So, is there a way to resize arrays without requiring that extra temporary memory?

Answer 1

I would use a List<int[]> where each int[] is a fixed size. e.g. 128 million. To grow the whole “collection” only involves added another array. I use IntBuffer in direct memory which avoids the need to use heap. (or use memory mapped files which means it doesn’t use heap or direct memory 😉 This is ugly, and I use a wrapper class to hide the ugliness. It does perform pretty nicely. With memory mapped files I can use an “array” which is larger than the physical memory.

private final List<IntBuffer> array = new ArrayList<IntBuffer>();

public int get(long n) {
    return array.get((int)(n >> 27)).get(n & ((1 << 27) -1));
}

public void put(long n, int v) {
    return array.get((int)(n >> 27)).put(n & ((1 << 27) -1), v);
}