Question

I’m trying to implement a FUSE-driven filesystem in Python, which serves data from both local and remote sources. The filesystem is handled by the main FUSE thread: filesystem requests are dealt with straight as they’re requested.

class MyFilesystem(Fuse):
    def read(self, path, size, offset):
        if self._isLocalFile(path):
            return self._localRead(path, size, offset)
        elif self._isRemoteFile(path):
            # get file from server
    # ...

I had thought to create a second thread on initialization which keeps the communication open between client and server. Commands flow both ways, so the client currently uses a select() call to wait for any incoming commands.

class CommsClient(threading.Thread):
    def run(self):
        conn = self._connect()
        while True:
            r, w, e = select.select([conn], [], [], 1.0)
            if conn in r:
                self._handleData(conn)
    # ...

The problem I have now is connecting the two threads. When the filesystem thread deals with a request, it might have to block until the comms thread returns a reply from the server. I think one way of accomplishing this is plugging a request stream/socket from the filesystem thread into the select() call, but I’m not sure whether sockets are best used for inter-thread communication. Shortening the select() timeout and checking for an Event or inter-thread variable would also work, I guess, but I’d like the mechanism to be as fast as possible.

Does anyone know the best way to handle this situation?

Answer 1

Using sockets for inter-thread communication is perfectly acceptable, but it will be slower than implementing it with threads and locks using in-memory data structures.
Notice that ‘slower’ is relative: Hard disk operations might still be slower by a tenfold.

Although it’s not a direct answer to the question, may I recommend you look at
ØMQ? It’s a very fast, It gives you ‘sockets’ that carry whole messages across various transports like in-process, inter-process, TCP, and multicast and it has asynchronous I/O.