Question

How can I create a huge numpy array using pytables. I tried this but gives me the “ValueError: array is too big.” error:

import numpy as np
import tables as tb
ndim = 60000
h5file = tb.openFile('test.h5', mode='w', title="Test Array")
root = h5file.root
h5file.createArray(root, "test", np.zeros((ndim,ndim), dtype=float))
h5file.close()

Answer 1

You could try to use tables.CArray class as it supports compression but…

I think questions is more about numpy than pytables because you are creating array using numpy before storing it with pytables.

In that way you need a lot of ram to execute np.zeros((ndim,ndim) – and this is probably the place where exception: “ValueError: array is too big.” is raised.

If matrix/array is not dense then you could use sparse matrix representation available in scipy: http://docs.scipy.org/doc/scipy/reference/sparse.html

Another solution is to try to access your array via chunks if it you don’t need whole array at once – check out this thread: Very large matrices using Python and NumPy