Question

I’m writing a wsgi application that needs to use Twisted PerspectiveBroker to call some remote methods. The problem is that the wsgi needs to return a rendered webpage, but the calls to the Twisted service are asynchronous. So basically my web app needs to call the remote methods, then do some other stuff, then it has to wait for the remote calls to finish, then render the page and return it to the client.

What is the best way to do this?

I am currently planning on using Flask to write the app.

Answer 1

A WSGI application runs in its own thread (or process). When running in Twisted’s WSGI container, that is a different thread than the reactor is running in. Most of Twisted’s APIs are not thread-safe: they may only be called in the reactor thread.

So, the basic way to call a Twisted API from a WSGI application is using reactor.callFromThread, which is thread-safe and which causes a function to be called in the reactor thread:

...
reactor.callFromThread(pbRemote.callRemote, "someMethod", some, args)

However, this discards the result, which you you probably want. It’s simple to build an API on top of reactor.callFromThread that preserves the result, though, and there’s an implementation of that in Twisted, too:

from twisted.internet.threads import blockingCallFromThread

...
result = blockingCallFromThread(reactor, pbRemote.callRemote, "someMethod", some, args)

This call will block until the Deferred returned by callRemote fires and then it will return the result of that Deferred.

If you want to make the call, do some other work, and then wait for the call to finish, you have to get a little bit creative. You need to make the call and get the actual Deferred it returns, but not block on it:

resultHolder = blockingCallFromThread(
    reactor, lambda: [pbRemote.callRemote("someMethod", some, args)])

And then you can do what other work you need to do. And when you’re ready to wait for the result of the PB call:

result = blockingCallFromThread(reactor, lambda: resultHolder[0])

This is all rather more awkward than using Twisted in a single-threaded scenario, so it may indeed be easier to use Twisted Web’s native APIs rather than build a WSGI application. Remember that one of the primary goals of WSGI is to allow applications to be developed that are portable across different servers – Twisted, Apache, etc. If you’re actually using Twisted APIs in your WSGI application, then it’s not portable at all.