Question

I have an embedded Python program which runs in a thread in C.

When the Python interpreter switches thread context (yielding control to another thread), I’d like to be notified so I can perform certain necessary operations.

It seems that Py_AddPendingCall is exactly what I’m looking for. However, the API docs are pretty brief on this function, and I’m confused as to how Py_AddPendingCall is supposed to be used. From reading the docs, my understanding is that:

• A worker thread calls Py_AddPendingCall and assigns a handler
  function.
• When the Python interpreter runs, it calls the handler function
  whenever it yields control to another thread
• The handler itself is executed in the main interpreter thread, with
  the GIL acquired

I’ve googled around for example code showing how to use Py_AddPendingCall, but I can’t find anything. My own attempt to use it simply doesn’t work. The handler is just never called.

My worker thread code:

#include <pthread.h>
#include <stdlib.h>
#include <stdio.h>

const char* PYTHON_CODE = 
"while True:\n"
"   for i in range(0,10): print(i)\n"
"\n";

int handler(void* arg)
{
    printf("Pending Call invoked!\n");
    abort();
}

void* worker_thread(void*)
{
    PyGILState_STATE state = PyGILState_Ensure();
    int res = Py_AddPendingCall(&func, nullptr);
    cout << "Result: " << res << endl;
    PyRun_SimpleString(CODE);
    PyGILState_Release(state);

    return 0;
}

int main()
{
    Py_Initialize();
    PyEval_InitThreads();
    PyEval_ReleaseLock();

    pthread_t threads[4];
    for (int i = 0; i < 4; ++i)
        pthread_create(&threads[i], 0, worker_thread, 0);

    for (int i = 0; i < 4; ++i) pthread_join(threads[i], 0);

    Py_Finalize();
}

In this example, worker_thread is invoked in C as a pthread worker thread. In this test I run 4 worker threads, so some context switching should happen. This loops infinitely, but the pending call handler is never invoked.

So, can someone provide a minimal working example that shows how Py_AddPendingCall is supposed to be used?

Answer 1

Pending calls are only executed on the main thread, and the main thread can only service pending calls when it executes Python code. So, the main thread must be running an interpreter loop in order to service any pending calls. From Python/ceval.c:

Py_MakePendingCalls(void)
{
    ...

    /* only service pending calls on main thread */
    if (main_thread && PyThread_get_thread_ident() != main_thread)
        return 0;

    ...
}

In an embedding scenario, the “main thread” is whichever thread calls PyEval_InitThreads (note that it is also automatically called the first time you start a thread from Python code). Since your main thread sits in pthread_join, it is not executing Python code and therefore not dispatching pending calls.

Note too that the handler only executes when control is handed to the main thread — the handler doesn’t run if control goes from one worker to another. So, Py_MakePendingCalls is probably not an appropriate interface to use for this task.