Question

I’m porting one of my projects from C# and am having trouble solving a multithreading issue in Python. The problem relates to a long-lived HTTP request, which is expected (the request will respond when a certain event occurs on the server). Here’s the summary:

I send the request using urllib2 on a separate thread. When the request returns or times out, the main thread is notified. This works fine. However, there are cases where I need to abort this outstanding request and switch to a different URL. There are four solutions that I can consider:

1. Abort the outstanding request. C# has WebRequest.Abort(), which I can call cross-thread to abort the request. Python urllib2.Request appears to be a pure data class, in that instances only store request information; responses are not connected to Request objects. So I can’t do this.
2. Interrupt the thread. C# has Thread.Interrupt(), which will raise a ThreadInterruptedException in the thread if it is in a wait state, or the next time it enters such a state. (Waiting on a monitor and file/socket I/O are both waiting states.) Python doesn’t seem to have anything comparable; there does not appear to be a way to wake up a thread that is blocked on I/O.
3. Set a low timeout on the request. On a timeout, check an “aborted” flag. If it’s false, restart the request.
4. Similar to option 3, add an “aborted” flag to the state object so that when the request does finally end in one way or another, the thread knows that the response is no longer needed and just shuts itself down.

Options 3 and 4 seem to be the only ones supported by Python, but option 3 is a horrible solution and 4 will keep open a connection I don’t need. I am hoping to be a good netizen and close this connection when I no longer need it. Is there any way to actually abort the outstanding request, one way or another?

Answer 1

Consider using gevent. Gevent uses non-thread cooperating units of execution called greenlets. Greenlets can “block” on IO, which really means “go to sleep until the IO is ready”. You could have a requester greenlet that owns the socket and a main greenlet that decides when to abort. When you want to abort and switch URLs the main greenlet kills the requester greenlet. The requester catches the resulting exception, closes its socket/urllib2 request, and starts over.

Edited to add: Gevent is not compatible with threads, so be careful with that. You’ll have to either use gevent all the way or threads all the way. Threads in python are kinda lame anyway because of the GIL.