Question

I’m trying to write an application which uses Google’s protocol buffers to deserialize data (sent from another application using protocol buffers) over a TCP connection. The problem is that it looks as if protocol buffers in Python can only deserialize data from a string. Since TCP doesn’t have well-defined message boundaries and one of the messages I’m trying to receive has a repeated field, I won’t know how much data to try and receive before finally passing the string to be deserialized.

Are there any good practices for doing this in Python?

Answer 1

Don’t just write the serialized data to the socket. First send a fixed-size field containing the length of the serialized object.

The sending side is roughly:

socket.write(struct.pack("H", len(data))    #send a two-byte size field
socket.write(data)

And the recv’ing side becomes something like:

dataToRead = struct.unpack("H", socket.read(2))[0]    
data = socket.read(dataToRead)

This is a common design pattern for socket programming. Most designs extend the over-the-wire structure to include a type field as well, so your receiving side becomes something like:

type = socket.read(1)                                 # get the type of msg
dataToRead = struct.unpack("H", socket.read(2))[0]    # get the len of the msg
data = socket.read(dataToRead)                        # read the msg

if TYPE_FOO == type:
    handleFoo(data)

elif TYPE_BAR == type:
    handleBar(data)

else:
    raise UnknownTypeException(type)

You end up with an over-the-wire message format that looks like:

struct {
     unsigned char type;
     unsigned short length;
     void *data;
}

This does a reasonable job of future-proofing the wire protocol against unforeseen requirements. It’s a Type-Length-Value protocol, which you’ll find again and again and again in network protocols.