If I use socket.makefile and then close the file object as well as the underlying socket, then subsequent calls to read will throw an exception, just as I’d want it to. For example, the following code works as I’d expect:
import socket
from time import sleep
from threading import Thread
ADDR = ("localhost", 4321)
def listener(sock):
client,addr = sock.accept()
sleep(1)
client.close()
sock.close()
server_sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
server_sock.bind(ADDR)
server_sock.listen(1)
Thread(target=listener, args=[server_sock]).start()
sock = socket.create_connection(ADDR)
f = sock.makefile("r+b", bufsize=0)
f.close()
sock.close()
f.read(8) # throws an exception, as I'd expect
However, if I make a call to read while the file/socket are still open then that call will block, and then if I close the socket, the read method still doesn’t return. In fact, it hangs indefinitely until the socket is closed on the other end. The following code demonstrates this distressing behavior:
import socket
from time import sleep
from threading import Thread
ADDR = ("localhost", 4321)
def reader(f):
print("about to read")
print("we read %r" % f.read(8))
print("finished reading")
def listener(sock):
client, addr = sock.accept()
sleep(3)
client.close()
sock.close()
server_sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
server_sock.bind(ADDR)
server_sock.listen(1)
Thread(target=listener, args=[server_sock]).start()
sock = socket.create_connection(ADDR)
f = sock.makefile("r+b", bufsize=0)
Thread(target=reader, args=[f]).start()
sleep(1)
print("closing pseudo-file and socket")
f.close()
sock.close()
sleep(1)
print("we still haven't finished reading!")
This is a serious problem for me, since I’d like to make a blocking call to f.read in a thread, but then still be able to close the socket and have the thread return from that call (possibly by throwing an exception) and exit. However, all that happens is that the call blocks forever so long as the other side never closes the socket.
So is there any way for Thread1 to call read on the file-like object created by socket.makefile and then have Thread2 shut down the socket in a way that causes Thread1 to stop blocking on its read call?
EDIT: I tried re-writing my program to entirely use gevent and its socket and Greenlet approach to multithreading, but my program still does exactly the same thing:
from gevent import sleep, socket, spawn
ADDR = ("localhost", 4321)
def reader(f):
print("about to read")
print("we read %r" % f.read(8))
print("finished reading")
def listener(sock):
client, addr = sock.accept()
sleep(3)
client.close()
sock.close()
server_sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
server_sock.bind(ADDR)
server_sock.listen(1)
spawn(listener, server_sock)
sock = socket.create_connection(ADDR)
f = sock.makefile("r+b", bufsize=0)
spawn(reader, f)
sleep(1)
print("closing pseudo-file and socket")
f.close()
sock.close()
sleep(1)
print("we still haven't finished reading!")
sleep(2)
I was surprised to learn that even using gevent sockets, calls to read will block even after the underlying socket is closed. If there’s no way to prevent this, I’ll probably just have to accept Thomas’ depressing “this is not possible” answer 🙁
Yes, threads have distressing behaviour, especially when touching things like sockets from multiple threads at the same time. This is not something you can make work right. There is no way to force a thread that’s blocked in a read to break out of the read or terminate — closing the socket from under it is more likely to cause your entire program to segfault (or worse) than it is to do what you want.
The right way to deal with situations like this is to use non-blocking reads, instead. A decent event framework, like Twisted, can help you with this.