Question

Is it possible to create an .npy file without allocating the corresponding array in memory first?

I need to create and work with a large numpy array, too big to create in memory. Numpy supports memory mapping, but as far as I can see my options are either:

1. Create a memmapped file using numpy.memmap. This creates the file directly on disk without allocating memory, but doesn’t store the metadata, so when I re-map the file later I need to know its dtype, shape, etc. In the following, notice that not specifying the shape results in the memmap being interpreted as flat array:

   In [77]: x=memmap('/tmp/x', int, 'w+', shape=(3,3))
   
   
   In [78]: x
   Out[78]: 
   memmap([[0, 0, 0],
          [0, 0, 0],
          [0, 0, 0]])
   
   
   In [79]: y=memmap('/tmp/x', int, 'r')
   
   
   In [80]: y
   Out[80]: memmap([0, 0, 0, 0, 0, 0, 0, 0, 0])
   

2. Create an array in memory, save it using numpy.save, after which it can be loaded in memmapped mode. This records metadata with the array data on disk, but requires that memory be allocated for the entire array at least once.

Answer 1

I had the same question and was disappointed when I read Sven’s reply. Seems as though numpy would be missing out on some key functionality if you couldn’t have a huge array on file and work on little pieces of it at a time. Your case seems to be close to one of the use cases in the origional rational for making the .npy format (see: http://svn.scipy.org/svn/numpy/trunk/doc/neps/npy-format.txt).

I then ran into numpy.lib.format, which seems to be full useful goodies. I have no idea why this functionality is not available from the numpy root package. The key advantage over HDF5 is that this ships with numpy.

>>> print numpy.lib.format.open_memmap.__doc__

"""
Open a .npy file as a memory-mapped array.

This may be used to read an existing file or create a new one.

Parameters
----------
filename : str
    The name of the file on disk. This may not be a filelike object.
mode : str, optional
    The mode to open the file with. In addition to the standard file modes,
    'c' is also accepted to mean "copy on write". See `numpy.memmap` for
    the available mode strings.
dtype : dtype, optional
    The data type of the array if we are creating a new file in "write"
    mode.
shape : tuple of int, optional
    The shape of the array if we are creating a new file in "write"
    mode.
fortran_order : bool, optional
    Whether the array should be Fortran-contiguous (True) or
    C-contiguous (False) if we are creating a new file in "write" mode.
version : tuple of int (major, minor)
    If the mode is a "write" mode, then this is the version of the file
    format used to create the file.

Returns
-------
marray : numpy.memmap
    The memory-mapped array.

Raises
------
ValueError
    If the data or the mode is invalid.
IOError
    If the file is not found or cannot be opened correctly.

See Also
--------
numpy.memmap
"""