Question

relatively new to python so excuse me if this has an obvious answer that I havent found.

I am reading some temporaly contiguous binary files into numpy record arrays with the end goal of storing them in a pytable. The problem I anticipate is that the files may not all have the same fields, or the same field order. I have been looking for a numpy function that will sort the columns (NOT the rows) of a recarray using either the field labels or an index. Even better would be a function that does this for you – and accounts for missing columns – when you append a recarray to another. Below is a sample of what I had in mind:

#-------script------------
Myarray1 = np.array([(1,2,3),(1,2,3),(1,2,3)], {'names': ('a','b','c'), 'formats': ('f4', 'f4', 'f4')})
Myarray2 = np.array([(2,1,4,3),(2,1,4,3),(2,1,4,3)], {'names': ('b','a','d','c'), 'formats': ('f4', 'f4', 'f4', 'f4')})
Myarray3 = SomeColumnSortFunction(Myarray2, sortorder=[2,1,4,3])
Myarray4 = SomeBetterVerticalStackFunction(Myarray1,Myarray2)
#
print(Myarray1)
print()
print(Myarray2)
print()
print(Myarray3)
print()
print(Myarray4)

#---------- Wished for Output -------------
[(1.0, 2.0, 3.0) (1.0, 2.0, 3.0) (1.0, 2.0, 3.0)],
 dtype=[('a', 'i4'), ('b', 'i4'), ('c', 'i4')]

[(2.0, 1.0, 4.0, 3.0) (2.0, 1.0, 4.0, 3.0) (2.0, 1.0, 4.0, 3.0)],
dtype=[('b', 'i4'), ('a', 'i4'), ('d', 'i4'), ('c', 'i4')]


[(1.0, 2.0, 3.0, 4.0) (1.0, 2.0, 3.0, 4.0) (1.0, 2.0, 3.0, 4.0)] 
dtype=[('a', 'i4'), ('b', 'i4'), ('c', 'i4'), ('d', 'i4')]

[(1.0, 2.0, 3.0, NaN) (1.0, 2.0, 3.0, NaN) (1.0, 2.0, 3.0, NaN),
 (1.0, 2.0, 3.0, 4.0) (1.0, 2.0, 3.0, 4.0) (1.0, 2.0, 3.0, 4.0)] 
dtype=[('a', 'i4'), ('b', 'i4'), ('c', 'i4'), ('d', 'i4')]

Answer 1

• If you want to reorder the fields of your structured array, just use fancy indexing:

  MyArray3 = MyArray2[['a','b','c','d']]
  

  If you want to use integers to sort your fields, you can use something like:

  order = [1,0,3,2]
  names = MyArray2.dtype.names
  MyArray3 = MyArray2[[names[i] for i in order]]
  

  (in your sortorder=[2,1,4,3], you probably forgot that the first index of an iterable is 0…)

• For stacking structured arrays, have a look to the numpy.lib.recfunctions submodule, the stack_arrays function in particular. Note that you have to use import numpy.lib.recfunctions explicitly

Here’s the docstring

stack_arrays(arrays, defaults=None, usemask=True, asrecarray=False, autoconvert=False)

Superposes arrays fields by fields

Parameters
----------
seqarrays : array or sequence
    Sequence of input arrays.
defaults : dictionary, optional
    Dictionary mapping field names to the corresponding default values.
usemask : {True, False}, optional
    Whether to return a MaskedArray (or MaskedRecords is `asrecarray==True`)
    or a ndarray.
asrecarray : {False, True}, optional
    Whether to return a recarray (or MaskedRecords if `usemask==True`) or
    just a flexible-type ndarray.
autoconvert : {False, True}, optional
    Whether automatically cast the type of the field to the maximum.

Examples
--------
>>> from numpy.lib import recfunctions as rfn
>>> x = np.array([1, 2,])
>>> rfn.stack_arrays(x) is x
True
>>> z = np.array([('A', 1), ('B', 2)], dtype=[('A', '|S3'), ('B', float)])
>>> zz = np.array([('a', 10., 100.), ('b', 20., 200.), ('c', 30., 300.)],
...   dtype=[('A', '|S3'), ('B', float), ('C', float)])
>>> test = rfn.stack_arrays((z,zz))
>>> test
masked_array(data = [('A', 1.0, --) ('B', 2.0, --) ('a', 10.0, 100.0) ('b', 20.0, 200.0)
 ('c', 30.0, 300.0)],
             mask = [(False, False, True) (False, False, True) (False, False, False)
 (False, False, False) (False, False, False)],
       fill_value = ('N/A', 1e+20, 1e+20),
            dtype = [('A', '|S3'), ('B', '<f8'), ('C', '<f8')])