Question

I have a C++ callback function that calls into Python using ctypes. This function’s parameters are a pointer to an array of double and the number of elements.

There are a lot of elements, approximately 2,000,000. I need to send this into scipy functions.

The C++ prototype is :

bool (*ptsetDataSource)(double*, long long);

which is the following python code:

CPF_setDataSource = CFUNCTYPE(c_bool, POINTER(c_double),c_longlong)
CPF_setSelection= CFUNCTYPE(c_bool,c_char_p, c_longlong,c_longlong)
CPF_ResetSequence = CFUNCTYPE(c_bool)

def setDataSource(Data, DataLength):
    Datalist=[0.0]*100
    for i in range(0,100):
        Datalist[i]=Data[i]

    print Datalist
    return True

The problem is that print datalist returns:

[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]

which is not correct(data is filled with a lot of other numbers when checked on the c++ side.

Also, if I use this code to convert the data to a python list, it locks up the computer at the allocate step.

Is there anyway to load the data from the C++ array and then convert it to an array fit for scipy?

Answer 1

If Data were (c_double*DataLength.value) array then you could:

a = np.frombuffer(Data) # no copy. Changes in `a` are reflected in `Data`

If Data is a POINTER(c_double) you could get numpy array using numpy.fromiter(). It is the same loop as in your question but faster:

a = np.fromiter(Data, dtype=np.float, count=DataLength.value) # copy

To create a numpy array from POINTER(c_double) instance without copying you could use .from_address() method:

ArrayType = ctypes.c_double*DataLength.value
addr = ctypes.addressof(Data.contents)
a = np.frombuffer(ArrayType.from_address(addr))

Or

array_pointer = ctypes.cast(Data, ctypes.POINTER(ArrayType))
a = np.frombuffer(array_pointer.contents)

Both methods convert POINTER(c_double) instance to (c_double*DataLength) before passing it to numpy.frombuffer().

Cython-based solution

Is there anyway to load the data from the C++ array and then convert it to an array fit for scipy?

Here’s C extension module for Python (written in Cython) that provide as C API the conversion function:

cimport numpy as np
np.import_array() # initialize C API to call PyArray_SimpleNewFromData

cdef public api tonumpyarray(double* data, long long size) with gil:
    if not (data and size >= 0): raise ValueError
    cdef np.npy_intp dims = size
    #NOTE: it doesn't take ownership of `data`. You must free `data` yourself
    return np.PyArray_SimpleNewFromData(1, &dims, np.NPY_DOUBLE, <void*>data)

It could be used with ctypes as follows:

from ctypes import (PYFUNCTYPE, py_object, POINTER, c_double, c_longlong,
                    pydll, CFUNCTYPE, c_bool, cdll)

import pointer2ndarray
tonumpyarray = PYFUNCTYPE(py_object, POINTER(c_double), c_longlong)(
    ("tonumpyarray", pydll.LoadLibrary(pointer2ndarray.__file__)))

@CFUNCTYPE(c_bool, POINTER(c_double), c_longlong)
def callback(data, size):
    a = tonumpyarray(data, size)
    # call scipy functions on the `a` array here
    return True

cpplib = cdll.LoadLibrary("call_callback.so") # your C++ lib goes here
cpplib.call_callback(callback)

Where call_callback is: void call_callback(bool (*)(double *, long long)).