A Mac OSX network client I’m working on sometimes having trouble connecting to an HTTPS port. Looking at the network trace, we’re seeing this:
T0.0 client:port -> server:443 SYN
T0.1 server:443 -> client:port SYN, ACK
T3.1 server:443 -> client:port SYN, ACK
T6.1 server:443 -> client:port RST
The 3 second delay matches the TCP timeout for retrying on a failed SYN/ACK, so that’s expected. But what is incredible here is that the client never responds with ACK or RST. When the client tries to login a second time, it is successful. This problem reproduces on many first attempts to connect. There are other HTTPS connections from this program going on at the same time, and they appear to be fine in the network traces.
My suspicion is that there is a race condition causing the socket to sometimes be incorrectly managed. But so far I’ve been unable to recreate this in any code but the impacted client (which is a very large and complicated piece of software). Even using nmap and hping3 to hand-craft the packets, the client always sends at least a RST.
Is there any way to configure (intentionally or by mistake) a socket like this in userland code so it doesn’t respond to SYN/ACK?
No, there is no way to configure a TCP socket in user land to ignore a SYN/ACK except to close it, which would provoke an outgoing RST in response to the incoming SYN/ACK. It is sounding like a kernel bug to me. I’m wondering about all these connections though. You should be making maximum use of HTTP keep alive to minimize the number of new connections per unit time.