Question

I have a python app that uses multiple threads and I am curious about the best way to wait for something in python without burning cpu or locking the GIL.

my app uses twisted and I spawn a thread to run a long operation so I do not stomp on the reactor thread. This long operation also spawns some threads using twisted’s deferToThread to do something else, and the original thread wants to wait for the results from the defereds.

What I have been doing is this

while self._waiting:
    time.sleep( 0.01 )

which seemed to disrupt twisted PB’s objects from receiving messages so I thought sleep was locking the GIL. Further investigation by the posters below revealed however that it does not.

There are better ways to wait on threads without blocking the reactor thread or python posted below.

Answer 1

If you’re already using Twisted, you should never need to “wait” like this.

As you’ve described it:

I spawn a thread to run a long operation … This long operation also spawns some threads using twisted’s deferToThread …

That implies that you’re calling deferToThread from your “long operation” thread, not from your main thread (the one where reactor.run() is running). As Jean-Paul Calderone already noted in a comment, you can only call Twisted APIs (such as deferToThread) from the main reactor thread.

The lock-up that you’re seeing is a common symptom of not following this rule. It has nothing to do with the GIL, and everything to do with the fact that you have put Twisted’s reactor into a broken state.

Based on your loose description of your program, I’ve tried to write a sample program that does what you’re talking about based entirely on Twisted APIs, spawning all threads via Twisted and controlling them all from the main reactor thread.

import time

from twisted.internet import reactor
from twisted.internet.defer import gatherResults
from twisted.internet.threads import deferToThread, blockingCallFromThread

def workReallyHard():
    "'Work' function, invoked in a thread."
    time.sleep(0.2)

def longOperation():
    for x in range(10):
        workReallyHard()
        blockingCallFromThread(reactor, startShortOperation, x)
    result = blockingCallFromThread(reactor, gatherResults, shortOperations)
    return 'hooray', result

def shortOperation(value):
    workReallyHard()
    return value * 100

shortOperations = []

def startShortOperation(value):
    def done(result):
        print 'Short operation complete!', result
        return result
    shortOperations.append(
        deferToThread(shortOperation, value).addCallback(done))

d = deferToThread(longOperation)
def allDone(result):
    print 'Long operation complete!', result
    reactor.stop()
d.addCallback(allDone)

reactor.run()

Note that at the point in allDone where the reactor is stopped, you could fire off another “long operation” and have it start the process all over again.