Question

My multi-threaded app segfaults on a call to PyImport_ImportModule("my_module").

The BT will be posted at the bottom.

Some background:

1. My app creates multiple instances of many derived C++ classes and runs the base class’s Run() function which uses a virtual method to determine what to do.
2. One derived class uses a Python Class, Grasp_Behavior(class) in grasp_behavior(module)
3. After extensive reading I have used the Python API to achieve (2) (exerpts below)
4. I generate 2 instances of said class, and run them in “parallel” (python interpr doesn’t really run parallel)
5. I attempt to generate another instance of said class, segfault at PyImport_ImportModule

My thoughts are that perhaps I cannot import a module twice in the same interpreter. But I can’t figure out how to check it. I assume I need to see if grasp_behavior is in a dictionary but I don’t know which one, perhaps I get __main__ module’s dictionary?

But I might be wrong, any advice would be incredibly helpful!

In the constructor:

//Check if Python is Initialized, otherwise initialize it
if(!Py_IsInitialized())
{
    std::cout << "[GraspBehavior] InitPython: Initializing the Python Interpreter" << std::endl;
    Py_Initialize();
    PyEval_InitThreads(); //Initialize Python thread ability
    PyEval_ReleaseLock(); //Release the implicit lock on the Python GIL
}

// --- Handle Imports ----

PyObject * pModule = PyImport_ImportModule("grasp_behavior");
if(pModule == NULL)
{
    std::cout << "[GraspBehavior] InitPython: Unable to import grasp_behavior module: ";
    PyErr_Print();
}
 // --- Get our Class Pointer From the Module ...
PyObject * pClass = PyObject_GetAttrString(pModule, "Grasp_Behavior");
if(pClass == NULL)
{
    std::cout << "[GraspBehavior] InitPython: Unable to get Class from Module: ";
    PyErr_Print();
}
Py_DECREF(pModule); //clean up, this is a new reference

behavior_instance_ = PyObject_Call(pClass, pArguments_Tuple, pArguments_Dict);
if(behavior_instance_ == NULL)
{
    std::cout << "[GraspBehavior] InitPython: Couldn't generate instance: ";
    PyErr_Print();
}
Py_DECREF(pArguments_Tuple);
Py_DECREF(pArguments_Dict);
Py_DECREF(pClass);

Here, note that I only initialize the Python interpreter if it has not been initialized. I assume it gets initialized for the entire process.

In the Run() method (ran from a boost thread):

std::cout << "[GraspBehavior] PerformBehavior: Acquiring Python GIL Lock ..." << std::endl;
PyGILState_STATE py_gilstate;
py_gilstate = PyGILState_Ensure();

/* ---- Perform Behavior Below ----- */

std::vector<std::pair<double, double> > desired_body_offsets;
//desired_body_offsets.push_back( std::pair<double, double>(0.6, 0));
PyObject * base_positions = GetTrialBasePositions(my_env_, desired_body_offsets);

PyObject * grasps = EvaluateBasePositions(my_env_, base_positions);

//Did we get any grasps? What do we do with them? [TODO]
if(grasps != NULL)
{
    std::cout << grasps->ob_type->tp_name << std::endl;
    std::cout << "Number of grasps: " << PyList_Size(grasps) << std::endl;
    successful_ = true;
}

/* --------------------------------- */

std::cout << "[GraspBehavior] PerformBehavior: Releasing Python GIL Lock ..." << std::endl;
PyGILState_Release(py_gilstate);

Here, I have gone with PyGILState lock. I read for a while and it seemed the some of the articles that many people are link use an older style of locking in Python … perhaps I may have to switch this.

---

Backtrace:

Program received signal SIGSEGV, Segmentation fault.
0x00007fffee9c4330 in ?? () from /usr/lib/libpython2.6.so.1.0
(gdb) bt
#0  0x00007fffee9c4330 in ?? () from /usr/lib/libpython2.6.so.1.0
#1  0x00007fffee99ff09 in PyEval_GetGlobals ()
   from /usr/lib/libpython2.6.so.1.0
#2  0x00007fffee9bd993 in PyImport_Import () from /usr/lib/libpython2.6.so.1.0
#3  0x00007fffee9bdbec in PyImport_ImportModule ()
   from /usr/lib/libpython2.6.so.1.0
#4  0x000000000042d6f0 in GraspBehavior::InitPython (this=0x7948690)
    at grasp_behavior.cpp:241

Answer 1

First of all, you must not call any Python API functions when the GIL is released (except the GIL acquiring calls).

This code will crash:

PyEval_ReleaseLock();

PyObject * pModule = PyImport_ImportModule("grasp_behavior");

Release the GIL after you’re done setting up and then re-acquire as needed (in your Run()).

Also, PyEval_ReleaseLock is deprecated, you should use PyEval_SaveThread in this case instead.

PyThreadState* tstate = PyEval_SaveThread();

This will save the thread state and release the GIL.

Then, right before you start finalizing the interpreter, do this:

PyEval_RestoreThread(tstate);

passing the return value of the PyEval_SaveThread call.

In Run(), you should use PyGILState_Ensure and PyGILState_Release, as you do now, but you should think about C++ exceptions. Right now PyGILState_Release will not be called if Run() throws.

One nice property of the PyGILState calls is that you can use them no matter if the GIL is acquired or not and they will do the right thing, unlike older APIs.

Also, you should initialize the interpreter once at startup in the main thread (before other threads are started) and finalize after shutting down all threads but the main one.