Question

I’m trying to interop with C in Python, and I have the following function prototype:

int testout(unsigned char *test, int *size)

The size of the unsigned char is determined by the size parameter. So I need a way to resize the buffer in Python dynamically to adjust to the size returned from the function call. This has been very difficult for me to figure out.

From the Python ctypes manual:

“This is nice and fine, but how would one access the additional elements contained in this array? Since the type still only knows about 4 elements, we get errors accessing other elements:

>>> short_array[:]
[0, 0, 0, 0]
>>> short_array[7]
Traceback (most recent call last):
...
IndexError: invalid index
>>>

Another way to use variable-sized data types with ctypes is to use the dynamic nature of Python, and (re-)define the data type after the required size is already known, on a case by case basis.”

Great! I’m getting IndexError:invalid index, just as the example shows. And they decided not to show how to do this properly! 🙂

Does anyone know how to resize a ctype properly?

Here is my example code that works, except for the resizing:

from ctypes import *

lib = "test.so"
dll = cdll.LoadLibrary(lib)
print "Testing pointer output"
dll.testout.argtypes = [POINTER(c_ubyte), POINTER(c_int)]
sizeout = c_int(0)
mem = (c_ubyte * 20)() 
dll.testout(mem, byref(sizeout))
print "Sizeout = " + str(sizeout.value)
for i in range(0,sizeout.value):
    print "Item " + str(i) + " = " + str(mem[i])

Answer 1

Here’s an example of what we were discussing. This code expects the caller to allocate the buffer.

Test code (Windows)

// IN: test=pre-allocated buffer.
// IN: size=size of pre-allocated buffer.
// RETURNS: 0 and size set to required size.
//          1 and size set to size used.
__declspec(dllexport) int testout(void *pdata, int *psize)
{
    unsigned char * p = (unsigned char *)pdata;

    if (*psize < 5)
    {
        *psize = 5; // set size required
        return 0;  // fail
    }

    p[0] = 123;
    p[1] = 255;
    p[2] = 237;
    p[3] = 12;
    p[4] = 222;
    *psize = 5; // indicate size used

    return 1;  // pass
}

Python

from ctypes import *

test = CDLL('test')
size = c_int(0)
test.testout.argtypes=[c_void_p,POINTER(c_int)]
print "Result of NULL pointer and zero size:",test.testout(None,byref(size))
print "Returned size:",size.value
mem = (c_ubyte * 2)()
size = c_int(sizeof(mem))
print "sizeof(mem):",size.value
print "Result of small buffer:",test.testout(byref(mem),byref(size))
print "Returned size:",size.value
mem = (c_ubyte * size.value)()
print "Result of exact size buffer:",test.testout(byref(mem),byref(size))
print "Returned size:",size.value
for i in range(size.value):
    print mem[i]
mem = (c_ubyte * 20)()
size = c_int(20)
print "Result of bigger buffer:",test.testout(byref(mem),byref(size))
print "Returned size:",size.value
for i in range(size.value):
    print mem[i]

Output

Result of NULL pointer and zero size: 0
Returned size: 5
sizeof(mem): 2
Result of small buffer: 0
Returned size: 5
Result of exact size buffer: 1
Returned size: 5
123
255
237
12
222
Result of bigger buffer: 1
Returned size: 5
123
255
237
12
222