Question

I am trying to create a p2p applications on Linux, which I want to run as efficiently as possible.

The issue I have is with managing packets. As we know, there may be more than one packet in the recv() buffer at any time, so there is a need to have some kind of message framing system to make sure that multiple packets are not treated as one big packet.

So at the moment my packet structure is:

(u16int Packet Length):(Packet Data)

Which requires two calls to recv(); one to get the packet size, and one to get the packet.

There are two main problems with this:

1. A malicious peer could send a packet with a size header of 
  something large, but not send any more data. The application will 
  hang on the second recv(), waiting for data that will never come.
2. Assuming that calling Recv() has a noticeable performance penalty
  (I actually have no idea, correct me if I am wrong) calling Recv() twice 
  will slow the program down.

What is the best way to structure packets/Recieving system for both the best efficiency and stability? How do other applications do it? What do you recommend?

Thankyou in advance.

Answer 1

I think your “framing” of messages within a TCP stream is right on.

You could consider putting a “magic cookie” in front of each frame (e.g. write the 32-bit int “0xdeadbeef” at the top of each frame header in addition to the packet length) such that it becomes obvious that your are reading a frame header on the first of each recv() pairs. It the magic integer isn’t present at the start of the message, you have gotten out of sync and need to tear the connection down.

Multiple recv() calls will not likely be a performance hit. As a matter of fact, because TCP messages can get segmented, coalesced, and stalled in unpredictable ways, you’ll likely need to call recv() in a loop until you get all the data you expected. This includes your two byte header as well as for the larger read of the payload bytes. It’s entirely possible you call “recv” with a 2 byte buffer to read the “size” of the message, but only get 1 byte back. (Call recv again, and you’ll get the subsequent bytes). What I tell the developers on my team – code your network parsers as if it was possible that recv only delivered 1 byte at a time.

You can use non-blocking sockets and the “select” call to avoid hanging. If the data doesn’t arrive within a reasonable amount of time (or more data arrives than expected – such that syncing on the next message becomes impossible), you just tear the connection down.

I’m working on a P2P project of my own. Would love to trade notes. Follow up with me offline if you like.