Question

I’m about to try out Pytables for the first time and I need to write my data to the hdf file per time step. I’ll have over 100,000 time steps. When I’m done, I would like to sort my 100,000+ x 6 array by column 2, i.e., I currently have everything sorted by time but now I need to sort the array by order of decreasing rain rates (col 2). I’m unsure how to even begin here. I know that having the entire array in memory is unwise. Any ideas how to doe this fast and efficiently?

Appreciate any advice.

Answer 1

I know that having the entire array in memory is unwise.

You might be overthinking it. A 100K x 6 array of float64 takes just ~5MB of RAM. On my computer, sorting such an array takes about 27ms:

In [37]: a = np.random.rand(100000, 6)

In [38]: %timeit a[a[:,1].argsort()]
10 loops, best of 3: 27.2 ms per loop