Question

This is my code:

int size = 100000000;
double sizeInMegabytes = (size * 8.0) / 1024.0 / 1024.0; //762 mb
double[] randomNumbers = new double[size];

Exception:
Exception of type ‘System.OutOfMemoryException’ was thrown.

I have 4GB memory on this machine 2.5GB is free when I start this running, there is clearly enough space on the PC to handle the 762mb of 100000000 random numbers. I need to store as many random numbers as possible given available memory. When I go to production there will be 12GB on the box and I want to make use of it.

Does the CLR constrain me to a default max memory to start with? and how do I request more?

Update

I thought breaking this into smaller chunks and incrementally adding to my memory requirements would help if the issue is due to memory fragmentation, but it doesn’t I can’t get past a total ArrayList size of 256mb regardless of what I do tweaking blockSize.

private static IRandomGenerator rnd = new MersenneTwister();
private static IDistribution dist = new DiscreteNormalDistribution(1048576);
private static List<double> ndRandomNumbers = new List<double>();

private static void AddNDRandomNumbers(int numberOfRandomNumbers) {
    for (int i = 0; i < numberOfRandomNumbers; i++) {
      ndRandomNumbers.Add(dist.ICDF(rnd.nextUniform()));                
  }
}

From my main method:

int blockSize = 1000000;

while (true) {
  try
  {
    AddNDRandomNumbers(blockSize);                    
  }
  catch (System.OutOfMemoryException ex)
  {
    break;
  }
}            
double arrayTotalSizeInMegabytes = (ndRandomNumbers.Count * 8.0) / 1024.0 / 1024.0;

Answer 1

You may want to read this: ““Out Of Memory” Does Not Refer to Physical Memory” by Eric Lippert.

In short, and very simplified, “Out of memory” does not really mean that the amount of available memory is too small. The most common reason is that within the current address space, there is no contiguous portion of memory that is large enough to serve the wanted allocation. If you have 100 blocks, each 4 MB large, that is not going to help you when you need one 5 MB block.

Key Points:

• the data storage that we call “process memory” is in my opinion best visualized as a massive file on disk.
• RAM can be seen as merely a performance optimization
• Total amount of virtual memory your program consumes is really not hugely relevant to its performance
• “running out of RAM” seldom results in an “out of memory” error. Instead of an error, it results in bad performance because the full cost of the fact that storage is actually on disk suddenly becomes relevant.