I’m implementing a client/server pair, commnicating through TCP/IP sockets, using overlapped IO with completion routines in Windows. The debugging is being done with two VirtualBox virtual machines (client in one, server in another). The CPU is a Quad-Core.
The sequence of operations on the client is basically:
- Issues a WSARecv call (will not complete as of yet).
- Issues a WSASend call with a packet. The server will send a reply through the same socket.
- Once the completion routine for the WSASend call is executed the packet is placed in a waiting-for-reply queue.
- Once the completion routine for the WSARecv call executes (reply from server received), the code tries to locate the packet related to this reply in the queue.
- Issues a WSARecv call (will not complete as of yet).
- Repeat from the second step.
The sequence of operations on the server is basically:
- Issues a WSARecv call (will not complete as of yet).
- Once the completion routine for the WSARecv call executes (packet from the client received), the packet in processed in a secondary thread.
- Issues a WSARecv call (will not complete as of yet).
- Once the secondary thread completes a reply is sent to the client, issuing a WSASend call on the primary thread.
- Repeat from the second step.
The problem I’m having is that sometimes a reply is received on the client without having a corresponding packet in the waiting-for-reply queue. As part of the debugging effort I have two questions:
1) Is it conceivable that, having a pending WSARecv call, the completion routine of the WSARecv is scheduled for execution ahead of the completion routine for the corresponding WSASend (the one that sent the packet to which the server replied) ?
2) If a WSASend call completes immediately is the completion routine still scheduled for execution ?
I’m using a WSAWaitForMultipleEvents call as the alertable wait function.
Your code is broken. Even if the operations were guaranteed to complete in a particular order, that would not ensure that the completion indications were received in any particular order or that the code that handles those completions will run in any particular order.
The answer to your second question is yes. Generally, you can treat immediate completion the same as pending, since the completion routine will run in either case.
The ideal solution to your problem is to consider the packet to be waiting for reply as soon as you decide to send it.
If for some reason that requires too many code changes, you have another choice. If you receive a WSARecv completion and the WSASend completion has not occurred yet, simply defer processing the WSARecv completion. You can do this two ways:
Spawn a thread to wait a fraction of a second and then re-post the completion notification (by calling
PostQueuedCompletionStatus) from that thread. Presumably, the WSASend will have completed by then.Mark a flag that the WSARecv has completed and save the completion information. When the WSASend completes, notice that flag and process the WSARecv then. (Possibly by calling the completion handler directly.)