Question

There is this How do you split a list into evenly sized chunks?
for splitting an array into chunks. Is there anyway to do this more efficiently for giant arrays using Numpy?

Answer 1

Try numpy.array_split.

From the documentation:

>>> x = np.arange(8.0)
>>> np.array_split(x, 3)
    [array([ 0.,  1.,  2.]), array([ 3.,  4.,  5.]), array([ 6.,  7.])]

Identical to numpy.split, but won’t raise an exception if the groups aren’t equal length.

If number of chunks > len(array) you get blank arrays nested inside, to address that – if your split array is saved in a, then you can remove empty arrays by:

[x for x in a if x.size > 0]

Just save that back in a if you wish.