Question

I am sorting a number of integers from a file, which will probably be too large to fit into memory in one go, my current idea is to get sort chucks with quicksort, then mergesort them together. I would like to make the chunks as big as possible, so I’d like to know how much I can read in in one go.

I know about Runtime.FreeMemory, but how should I go about using it. Should I carefully work out what other variables I use in the program then create an array of size (freeMemory – variablesSizes), or is that too likely to go wrong?

Thanks!

Answer 1

Experiment until you find a size that works well. The largest array you can allocate on the heap isn’t necessarily the fastest way to do it. In many circumstances, the entire heap does not fit in the computers RAM, and might be swapped out in parts. Just because you can allocate a huge array, does not mean it will be the best size for optimizing speed.

Some adaptive approach would probably be best (testing number of items sorted/second depending on array size) and adjusting for what you can fit without getting an OutOfMemoryError.

Simpler: stick with some large value that works well, but isn’t necessarily the largest you can use.

Or: use an external library/database to do what you want – working with huge amounts of data is tricky to get right in general, and you will probably get better performance and shorter development time if you don’t reinvent the wheel.