Question

I am using version 1.5.1 of numpy and Python 2.6.6.

I am reading a binary file into a numpy array:

>>> dt = np.dtype('<u4,<i2,<i2,<i2,<i2,<i2,<i2,<i2,<i2,u1,u1,u1,u1')
>>> file_data = np.fromfile(os.path.join(folder,f), dtype=dt)

This works just fine. Examining the result:

>>> type(file_data)
<type 'numpy.ndarray'>

>>> file_data
array([(3571121L, -54, 103, 1, 50, 48, 469, 588, -10, 0, 102, 0, 0),
   (3571122L, -78, 20, 25, 45, 44, 495, 397, -211, 0, 102, 0, 0),
   (3571123L, -69, -48, 23, 60, 19, 317, -26, -151, 0, 102, 0, 0), ...,
   (3691138L, -53, 52, -2, -11, 76, 988, 288, -101, 1, 102, 0, 0),
   (3691139L, -11, 21, -27, 25, 47, 986, 253, 176, 1, 102, 0, 0),
   (3691140L, -30, -19, -63, 59, 12, 729, 23, 302, 1, 102, 0, 0)],
  dtype=[('f0', '<u4'), ('f1', '<i2'), ('f2', '<i2'), ... , ('f12', '|u1')])

>>> file_data[0]
(3571121L, -54, 103, 1, 50, 48, 469, 588, -10, 0, 102, 0, 0)

>>> file_data[0][0]
3571121    

>>> len(file_data)
120020

When I try to slice the first column:

>>> file_data[:,0]

I get:

IndexError: invalid index.

I have looked at simple examples and was able to do the slicing:

>>> a = np.array([(1,2,3),(4,5,6)])
>>> a[:,0]
array([1, 4])

The only difference I can see between my case and the simple example is that I am using the dtype. What I am doing wrong?

Answer 1

When you set the dtype like that, you are creating a Record Array. Numpy treats that like a 1D array of elements of your dtype. There’s a fundamental difference between

file_data[0][0]

and

file_data[0,0] 

In the first, you are asking for the first element of a 1D array and then retrieving the first element of that returned element. In the second, you are asking for the element in the first row of the first column of a 2D array. That’s why you are getting the IndexError.

If you want to access an individual element using 2D notation, you can create a view and work with that. Unfortunately, AFAIK if you want to treat your object like a 2D array, all elements have to have the same dtype.