Question

1) I would like to use the profiling functions in the Python C API to catch the python interpreter when it returns from specific functions.

2) I would like to pause the python interpreter, send execution back to the function that called the interpreter in my C++ program, and finally return execution to the python interpreter, starting it on the line of code after where it stopped. I would like to maintain both globals and locals between the times where execution belongs to python.

Part 1 I’ve finished. Part 2 is my question. I don’t know what to save so I can return to execution, or how to return to execution given that saved data.

From what I could get off the python API docs, I will have to save some part of the executing frame, but I haven’t found anything. Some additional questions…
What, exactly does a PyFrameObject contain? The python API docs, surprisingly, never explain that.

Answer 1

If I understand your problem, you have a C++ program that calls into python. When python finishes executing a function, you want to pause the interpreter and pick up where the C++ code left off. Some time later your C++ program needs to cal back into python, and have the python interpreter pick up where it left off.

I don’t think you can do this very easily with one thread. Before you pause the interpreter the stack looks like this:

[ top of stack ]
[ some interpreter frames ]
[ some c++ frames ] 

To pause the interpreter, you need to save off the interpreter frames, and jump back to the top-most C++ frame. Then to unpause, you need to restore the interpreter frames, and jump up the stack to where you left off. Jumping is doable (see http://en.wikipedia.org/wiki/Setjmp.h), but saving and restoring the stack is harder. I don’t know of an API to do this.

However you could do this with two threads. The thread created at the start of your c++ program (call it thread 1) runs the c++ code, and it creates thread 2 to run the python interpreter.

Initially (when were running c++ code), thread 1 is executing and thread 2 is blocked (say on a condition variable, see https://computing.llnl.gov/tutorials/pthreads/). When you run or unpause the interpreter thread 1 signals the condition variable, and waits on it. This wakes up thread 2 (which runs the interpreter) and causes thread 1 to block. When the interpreter needs to pause, thread 2 signals the condition variable and waits on it (so thread 2 blocks, thread 1 wakes up). You can bounce back and forth between the threads to your heart’s content. Hope this helps.