Question

So I’m working on a Python IRC framework, and I’m using Python’s socket module. Do I feel like using Twisted? No, not really.

Anyway, I have an infinite loop reading and processing data from socket.recv(xxxx), where xxxx is really irrelevant in this situation. I split the received data into messages using str.split("\r\n") and process them one by one.

My problem is that I have to set a specific ‘read size’ in socket.recv() to define how much data to read from the socket. When I receive a burst of data (for example, when I connect to the IRC server and receive the MOTD.etc), there’s always a message that spans two ‘reads’ of the socket (i.e. part of the line is read in one socket.recv() and the rest is read in the next iteration of the infinite loop).

I can’t process half-received messages, and I’m not sure if there’s even a way of detecting them. In an ideal situation I’d receive everything that’s in the buffer, but it doesn’t look like socket provides a method for doing that.

Any help?

Answer 1

You should really be using select or poll, e.g. via asyncore or select, or twisted (which you prefer not to).

Reading from a socket you never know how much you’ll receive in each read. You could receive several messages in one go, or have one message split into many reads. You should always collect the data in a buffer until you can make use of it, then remove the data you’ve used from the buffer (but leave data you haven’t used yet).

Since you know your input makes sense line by line, then your receive loop can look something like:

• while true:
  • Append new data to buffer
  • Look for EOLs, process and remove all complete lines