Question

I think I’m in a problem. I have two TCP apps connected to each other which use winsock I/O completion ports to send/receive data (non-blocking sockets).

Everything works just fine until there’s a data transfer burst. The sender starts sending incorrect/malformed data.

I allocate the buffers I’m sending on the stack, and if I understand correctly, that’s a wrong to do, because these buffers should remain as I sent them until I get the “write complete” notification from IOCP.

Take this for example:

void some_function()
{
    char cBuff[1024];

    // filling cBuff with some data

    WSASend(...); // sending cBuff, non-blocking mode

    // filling cBuff with other data

    WSASend(...); // again, sending cBuff

    // ..... and so forth!
}

If I understand correctly, each of these WSASend() calls should have its own unique buffer, and that buffer can be reused only when the send completes.
Correct?

Now, what strategies can I implement in order to maintain a big sack of such buffers, how should I handle them, how can I avoid performance penalty, etc’?
And, if I am to use buffers that means I should copy the data to be sent from the source buffer to the temporary one, thus, I’d set SO_SNDBUF on each socket to zero, so the system will not re-copy what I already copied. Are you with me? Please let me know if I wasn’t clear.

Answer 1

Take a serious look at boost::asio. Asynchronous IO is its specialty (just as the name suggests.) It’s pretty mature library by now being in Boost since 1.35. Many people use it in production for very intensive networking. There’s a wealth of examples in the documentation.

One thing for sure – it take working with buffers very seriously.

Edit:

Basic idea to handling bursts of input is queuing.

• Create, say, three linked lists of pre-allocated buffers – one is for free buffers, one for to-be-processed (received) data, one for to-be-sent data.
• Every time you need to send something – take a buffer off the free list (allocate a new one if free list is empty), fill with data, put it onto to-be-sent list.
• Every time you need to receive something – take a buffer off the free list as above, give it to IO receive routine.
• Periodically take buffers off to-be-sent queue, hand them off to send routine.
• On send completion (inline or asynchronous) – put them back onto free list.
• On receive completion – put buffer onto to-be-processed list.
• Have your “business” routine take buffers off to-be-processed list.

The bursts will then fill that input queue until you are able to process them. You might want to limit the queue size to avoid blowing through all the memory.