Question

I am implementing a program that has about 2,000,000 (2 million) arrays each of size 16,512 (128 x 129) of integers. I only need to call 200 arrays at a time (that is 3.3 MB), but I wonder if I can expand the program to have more than 2 million (say 200 million) but I still only need to call 200 arrays at a time. So what is the limit of making more and more arrays while I don’t use more than 200 arrays at a time?

Answer 1

I highly doubt that, unless you’re running on a 64 bit machine with a lot of RAM and a very generous heap.

Let’s calculate the memory you’ll need for your data:

2,000,000*128*129*8/1024/1024/1024 = 30.8GB.

You’ll need additional RAM for the JVM, the rest of your program, and the operating system.

Sounds like a poorly conceived solution to me.

If you mean “I only have 200 arrays in memory at a time” you can certainly do that, but you’ll have to move the rest out to secondary storage or a relational database. Query for them, use them, GC them. It might not be the best solution, but it’s hard to tell based on the little you’ve posted.

Update:

Does “trigger” mean “database trigger”?

Yes, you can store them on the disk. I can’t guarantee that it’ll perform. Your hard drive can certainly handle 30GB of data; it’s feasible that it’ll accomodate 300GB if it’s large enough.

Just remember that you have to think about how you’ll manage RAM. GC thrashing might be a problem. A good caching solution might be your friend here. Don’t write one yourself.

What happens if that hard drive fails and you lose all that data? Do you back it up? Can your app afford to be down if the disk fails? Think about those scenarios, too. Good luck.