Question

Scipy has many functions that accept a python callable to perform some operation. In particular, I’m working with a mathematical optimization function scipy.optimize.leastsq that accepts a Python callable as objective function argument. This objective function can be called by leastsq lots of times during the minimization process.

My profiling shows that a lot of time is spent on that objective function. I have sped up some parts of the function using Cython. However, the function itself is still a Python function and calling it repeatedly (as leastsq does) has some overhead.

I think I could get a further increase in speed if the function was a Cython function (using cdef instead of def). So I put my call to leastsq inside the Cython extension and pass a Cython objective function to it. But when I do this I get a compile error at the leastsq call:

Cannot convert 'object (object, object, object)' to Python object

Is there any way to pass a Cython function as an argument to these Scipy functions that require python callables?

Or, in my case, is there any way to access the underlying leastsq implementation and pass the Cython objective function to it?

Answer 1

Passing cdef functions in is not possible currently. Passing callback functions to the underlying Fortran code is also not possible, as it’s wrapped with f2py which doesn’t know about Cython.

What you can do is:

• Write your function as Cython’s def function. These can be passed in to all Scipy’s routines. This does not remove the additional function call overhead which comes from using Python’s callback mechanism (which might not be significant!), but you can speed up the implementation of your function, and this might well be enough. Just remember to cdef the variables appearing in, as usual when writing Cython code.

• Copy the MINPACK source codes from Scipy or netlib.org, and use them directly yourself. This gets rid of the remaining function call overhead by replacing Python function callback mechanism by the low-level one.

(There have been discussions on adding a protocol for passing low-level function pointers around exactly for this purpose, which could be adopted by any Python-based system having a need for it, but AFAIK the design is not completed, and not implemented in Cython & Scipy.)