Question

When I call BeginSend on a socket, I pass a delegate which will be called (by a different thread) when the data has been sent.

What would happen if I call BeginSend another time while the first has not yet been ‘callbacked’?

What is the correct behavior to send data? do BeginSend, and on the callback do EndSend and start another send? Or is it actually wise to have multiple BeginSends working at the same time?

this is the BeginSend page on MSDN which doesn’t give an answer to this question: BeginSend msdn

Answer 1

As O.K.W. says, multiple pending BeginSend calls will work fine. You probably DO need to bear in mind some things though.

Firstly if this is a TCP socket then this is still a single stream of data from peer to peer.

Secondly if all your BeginSend calls occur on the same thread then the result will be that the peer receives the data in the order of the calls. If your BeginSend calls occur from different threads then the data could arrive in any order as there will likely be a race condition between each of the sends. This may or may not matter to you (depends if you’re sending discrete, complete messages with each send or not).

Thirdly if you are using TCP and sending faster than the code at the other end of the socket can receive then you may fill the TCP Window and the TCP stacks will start to perform flow control on your data stream. If you continue to issue BeginSend calls then you MAY end up in a situation where your callbacks take longer and longer to be called as the TCP stack on your server queues data to send (you only get the callback once the data has been sent and the TCP Window based flow control will be preventing new data being sent until the TCP Window is no longer ‘full’; i.e. the peer has sent ACKs for some of the data that is in flight).

You can then get into a situation whereby you are using up resources on the sending machine in an uncontrollable manner (you issue a BeginSend and have no idea when it will complete and each send uses memory for the buffer being sent and potentially non-paged pool down in the Winsock code… Non-paged pool is a system wide resource and is quite a scarce on pre Vista OSs and some badly behaved drivers can blue screen the box if non-paged pool is low or exhausted. What’s more you may also be locking pages of memory into memory and there’s another system wide limit on the number of locked memory pages.

Because of these issues it’s usually best to implement your own protocol level flow control which limits the number of BeginSend calls that can be pending at any one time (using a protocol level ACK, perhaps) or to work with the TCP Window flow control and use the completion of a pending send to issue a new send and you can queue data to send in your own memory and have complete control over the resources used and what you do if you queue “too much” data. See my blog post here for more detail on this: http://www.serverframework.com/asynchronousevents/2011/06/tcp-flow-control-and-asynchronous-writes.html

See this reply: what happens when tcp/udp server is publishing faster than client is consuming? for more information on TCP Window flow control and what happens with overlapped I/O (in C++ land) when you ignore it and issue too many overlapped sends…

In summary, posting multiple concurrent BeginSend calls is the way to optimum TCP data flow but you need to make sure you don’t send “too fast” as once you do you are consuming resources in a manner which you can’t control and which is potentially fatal for the machine on which your code is running. So don’t allow an unbounded number of BeginSend calls to be outstanding and, ideally, profile the box to ensure that you are not exhausting system wide resources.