Question

I’m using numpy to create a cube array with sides of length 100, thus containing 1 million entries total. For each of the million entries, I am inserting a 100×100 matrix whose entries are comprised of randomly generated numbers. I am using the following code to do so:

import random
from numpy import *

cube = arange(1000000).reshape(100,100,100)

for element in cube.flat:
    matrix = arange(10000).reshape(100,100)
    for entry in matrix.flat:
        entry = random.random()*100
    element = matrix

I was expecting this to take a while, but with 10 billion random numbers being generated, I’m not sure my computer can even handle it. How much memory would such an array take up? Would RAM be a limiting factor, i.e. if my computer doesn’t have enough RAM, could it fail to actually generate the array?

Also, if there is a more efficient to implement this code, I would appreciate tips 🙂

Answer 1

A couple points:

• The size in memory of numpy arrays is easy to calculate. It’s simply the number of elements times the data size, plus a small constant overhead. For example, if your cube.dtype is int64, and it has 1,000,000 elements, it will require 1000000 * 64 / 8 = 8,000,000 bytes (8Mb).
• However, as @Gabe notes, 100 * 100 * 1,000,000 doubles will require about 80 Gb.
• This will not cause anything to “break”, per-se, but operations will be ridiculously slow because of all the swapping your computer will need to do.
• Your loops will not do what you expect. Instead of replacing the element in cube, element = matrix will simply overwrite the element variable, leaving the cube unchanged. The same goes for the entry = random.rand() * 100.
• Instead, see: http://docs.scipy.org/doc/numpy/reference/arrays.nditer.html#modifying-array-values