Question

i have a multithreaded library in which i call sleep(3) in different threads.
i have written python bindings for it using boost python.
now it looks like boost python is messing up with the sleep(3) function as it pauses the whole python program to wait.

please consider i have this boostmod.cpp file

#include <boost/python.hpp>
using namespace boost::python;
BOOST_PYTHON_MODULE( boostmod ) {
    def("waiter",&::sleep);
}

( you can compile it using: )

$ g++ -fPIC -shared boostmod.cpp `python-config --cflags --libs` -lboost_python -o boostmod.so

this is a python test file threadtest.py :

import time,sys,threading,boostmod
from ctypes import *
if __name__ == '__main__':
    libc = CDLL("libc.so.6") # this only works in linux
    for n in range(5):
         if sys.argv[1] == "boost":
            # this is slow
            threading.Thread(target=boostmod.waiter,args=(3,)).start()
         elif sys.argv[1] == "native":
            # this is fast
            threading.Thread(target=time.sleep,args=(3,)).start()
         elif sys.argv[1] == "ctypes":
            # this is fast
            threading.Thread(target=libc.sleep,args=(3,)).start()

the results are as follows:

$ time python threadtest.py boost
  real    0m15.030s
  user    0m0.024s
  sys     0m0.005s
$ time python threadtest.py native
  real    0m3.032s
  user    0m0.027s
  sys     0m0.003s
$ time python threadtest.py ctypes
  real    0m3.030s
  user    0m0.022s
  sys     0m0.008s

if you observe the situation with:

$ watch -n1 ps -C python -L -o pid,tid,pcpu,state

you can see that “native” and “ctypes” are really building up 5 threads plus the main thread while the “boost” case is only having one thread. Actually in the “boost” case the “.start()” is blocking inside the “sleep()” function.

Answer 1

first of all, i have seen that the time.sleep function in python is not using the sleep(3) systemcall. it uses a timing out call of select(2) on stdin ( in function “floatsleep” ) so that it can be interrupted.

However i have also found out that if you write a wrapper with Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS around the boost module function the phenomenon seems to dissolve.

So Here is the new boost module code that allows multithreading with sleep(3) calls:

#include <boost/python.hpp>
using namespace boost::python;
void waiter(int seconds) {
    Py_BEGIN_ALLOW_THREADS
    ::sleep(seconds);
    Py_END_ALLOW_THREADS
}
BOOST_PYTHON_MODULE( boostmod ) {
    def("waiter",&::waiter);
}