Question

I have a look-up table (LUT) that stores 65536 uint8 values:

lut = np.random.randint(256, size=(65536,)).astype('uint8')

I want to use this LUT to convert the values in an array of uint16s:

arr = np.random.randint(65536, size=(1000, 1000)).astype('uint16')

and I want to do the conversion in place, because this last array can get pretty big. When I try it, the following happens:

>>> np.take(lut, arr, out=arr)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python27\lib\site-packages\numpy\core\fromnumeric.py", line 103, in take
    return take(indices, axis, out, mode)
TypeError: array cannot be safely cast to required type

And I don’t understand what is going on. I know that, without an out argument, the return is of the same dtype as lut, so uint8. But why can’t a uint8 be cast to a uint16? If you ask numpy:

>>> np.can_cast('uint8', 'uint16')
True

Obviously the following works:

>>> lut = lut.astype('uint16')
>>> np.take(lut, arr, out=arr)
array([[173, 251, 218, ..., 110,  98, 235],
       [200, 231,  91, ..., 158, 100,  88],
       [ 13, 227, 223, ...,  94,  56,  36],
       ..., 
       [ 28, 198,  80, ...,  60,  87, 118],
       [156,  46, 118, ..., 212, 198, 218],
       [203,  97, 245, ...,   3, 191, 173]], dtype=uint16)

But this also works:

>>> lut = lut.astype('int32')
>>> np.take(lut, arr, out=arr)
array([[ 78, 249, 148, ...,  77,  12, 167],
       [138,   5, 206, ...,  31,  43, 244],
       [ 29, 134, 131, ..., 100, 107,   1],
       ..., 
       [109, 166,  14, ...,  64,  95, 102],
       [152, 169, 102, ..., 240, 166, 148],
       [ 47,  14, 129, ..., 237,  11,  78]], dtype=uint16)

This really makes no sense, since now int32s are being cast to uint16s, which is definitely not a safe thing to do:

>>> np.can_cast('int32', 'uint16')
False

My code works if I set the lut‘s dtype to anything in uint16, uint32, uint64, int32 or int64, but fails for uint8, int8 and int16.

Am I missing something, or is this simply broken in numpy?

Workarounds are also welcome… Since the LUT is not that big, I guess it is not that bad to have its type match the array’s, even if that takes twice the space, but it just doesn’t feel right to do that…

Is there a way to tell numpy to not worry about casting safety?

Answer 1

Interesting problem. numpy.take(lut, ...) gets transformed into lut.take(...) whose source can be looked at here:

https://github.com/numpy/numpy/blob/master/numpy/core/src/multiarray/item_selection.c#L28

I believe the exception is thrown at line 105:

obj = (PyArrayObject *)PyArray_FromArray(out, dtype, flags);
if (obj == NULL) {
    goto fail;
}

where in your case out is arr but dtype is the one of lut, i.e. uint8. So it tries to cast arr to uint8, which fails. I have to say that I’m not sure why it needs to do that, just pointing out it does… For some reason take seems to assume you want as the output array to have the same dtype as lut.

By the way, in many cases the call to PyArray_FromArray will actually create a new array and the replacement will not be in place. This is the case for example if you call take with mode='raise' (the default, and what happens in your examples), or whenever lut.dtype != arr.dtype. Well, at least it should, and I can’t explain why, when you cast lut to int32 the output array remains uint16! This is a mystery to me – maybe it has something to do with the NPY_ARRAY_UPDATEIFCOPY flag (see also here).

Bottom line:

1. the behavior of numpy is indeed difficult to understand… Maybe someone else will provide some insight into why it does what it does
2. I would not try to process arr in place – it seems that a new array is created under the hood in most cases anyway. I’d simply go with arr = lut.take(arr) – which by the way will eventually free half of the memory previously used by arr.