Question

I was writing a code which requires a large ‘int’ array to be allocated (size of 10^9).
While doing so i faced several issues and after reading stuff on Google i came to following conclusions of my own. Can someone see this and point out if i am missing some thing and also suggest a better way to do this.

(Machine config: VM machine Ubuntu 10.4,gcc 4.4.3 , 32bit, 2GB ram(though my host machine as 6gigs)

1.I declared the array as ‘unsigned long int’ with size 1*10^9. It didn’t worked as on compiling the code i got the error ‘array size too long’.
So i searched for this and finally realized that i cant allocate that much memory on stack as my physical memory was 2 GB.( i had already tried allocating the array as global variable which would allocate them in global area instead of stack but the same error)

1. So i tried allocating the same amount of memory using ‘malloc’ but again got the error with ‘malloc’ this time ‘Cannot alllocate memory’.

So after doing all this my understanding/problems are as follows:

3- I can’t allocate that much memory be it stack or heap as my physical mem is only 2Gb ( so this is the actual problem or some other factors also govern this mem allocation ??)

4- Is there any possible workaround where i can allocate a memory of size 10^9 on a 2gig machine( I know allocating a array or mem area this much big is neither good algo design nor efficient but i just want know the limits.)

5- any better solution for allocating this much memory ( i mean should i use 2 small arrays/heap mem instead of one big chunk)
(NOTE:Point 4 and 5 are two different approaches i would appreciate suggestion for both the approaches)

Many thanks
P.S forgive me if i am being novice ..

Answer 1

You are compiling a 32 bit process and there is simply not enough physical address space for your huge data block. A 32 bit pointer can hold 2^32 distinct values, i.e. 4GB. You can’t allocate more than that because you would have no way to refer to the memory. Each byte of memory that is mapped into your process must have a unique address.

So, nothing is going to fit your data into a 4GB address space. Even if your array was less than 4GB you may have problems allocating a single contiguous block of memory.

You could use a 64 bit process but you’d need to make sure you had enough physical memory to avoid disk thrashing when your array was swapped. Or you could find a different algorithm that did not require such a huge block of memory.