Question

The server (192.168.1.5:3001), is running Linux 3.2, and is designed to only accept one connection at a time.
The client (192.168.1.18), is running Windows 7. The connection is a wireless connection. Both programs are written in C++.

It works great 9 in 10 connect/disconnect cycles. The tenth-ish (randomly happens) connection has the server accept the connection, then when it later actually writes to it (typically 30+s later), according to Wireshark (see screenshot) it looks like it’s writing to an old stale connection, with a port number that the client has FINed (a while ago), but the server hasn’t yet FINed. So the client and server connections seems to get out of sync – the client makes new connections, and the server tries writing to the previous one. Every subsequent connection attempt fails once it gets in this broken state. The broken state can be initiated by going beyond the maximum wireless range for a half a minute (as before 9 in 10 cases this works, but it sometimes causes the broken state).

Wireshark screenshot behind link

The red arrows in the screenshot indicate when the server started sending data (Len != 0), which is the point when the client rejects it and sends a RST to the server. The coloured dots down the right edge indicate a single colour for each of the client port numbers used. Note how one or two dots appear well after the rest of the dots of that colour were (and note the time column).

The problem looks like it’s on the server’s end, since if you kill the server process and restart, it resolves itself (until next time it occurs).

The code is hopefully not too out-of-the-ordinary. I set the queue size parameter in listen() to 0, which I think means it only allows one current connection and no pending connections (I tried 1 instead, but the problem was still there). None of the errors appear as trace prints where “// error” is shown in the code.

// Server code

mySocket = ::socket(AF_INET, SOCK_STREAM, 0);
if (mySocket == -1)
{
  // error
}

// Set non-blocking
const int saveFlags = ::fcntl(mySocket, F_GETFL, 0);
::fcntl(mySocket, F_SETFL, saveFlags | O_NONBLOCK);

// Bind to port

// Union to work around pointer aliasing issues.
union SocketAddress
{
  sockaddr myBase;
  sockaddr_in myIn4;
};

SocketAddress address;
::memset(reinterpret_cast<Tbyte*>(&address), 0, sizeof(address));
address.myIn4.sin_family = AF_INET;
address.myIn4.sin_port = htons(Port);
address.myIn4.sin_addr.s_addr = INADDR_ANY;
if (::bind(mySocket, &address.myBase, sizeof(address)) != 0)
{
  // error
}
if (::listen(mySocket, 0) != 0)
{
  // error
}


// main loop
{
  ...
  // Wait for a connection.
  fd_set readSet;
  FD_ZERO(&readSet);
  FD_SET(mySocket, &readSet);
  const int aResult = ::select(getdtablesize(), &readSet, NULL, NULL, NULL);
  if (aResult != 1)
  {
    continue;
  }
  // A connection is definitely waiting.
  const int fileDescriptor = ::accept(mySocket, NULL, NULL);
  if (fileDescriptor == -1)
  {
    // error
  }

  // Set non-blocking
  const int saveFlags = ::fcntl(fileDescriptor, F_GETFL, 0);
  ::fcntl(fileDescriptor, F_SETFL, saveFlags | O_NONBLOCK);

  ...
  // Do other things for 30+ seconds.
  ...
  const int bytesWritten = ::write(fileDescriptor, buffer, bufferSize);
  if (bytesWritten < 0)
  {
    // THIS FAILS!! (but succeeds the first ~9 times)
  }

  // Finished with the connection.
  ::shutdown(fileDescriptor, SHUT_RDWR);
  while (::close(fileDescriptor) == -1)
  {
    switch(errno)
    {
    case EINTR:
      // Break from the switch statement. Continue in the loop.
      break;
    case EIO:
    case EBADF:
    default:
      // error
      return;
    }
  }
}

So somewhere between the accept() call (assuming that is exactly the point when the SYN packet is sent), and the write() call, the client’s port gets changed to the previously-used client port.

So the question is: how can it be that the server accepts a connection (and thus opens a file descriptor), and then sends data through a previous (now stale and dead) connection/file descriptor? Does it need some sort of option in a system call that’s missing?

Answer 1

Aha, success! It turns out the server was receiving the client’s SYN, and the server’s kernel was automatically completing the connection with another SYN, before the accept() had been called. So there definitely a listening queue, and having two connections waiting on the queue was half of the cause.

The other half of the cause was to do with information which was omitted from the question (I thought it was irrelevant because of the false assumption above). There was a primary connection port (call it A), and the secondary, troublesome connection port which this question is all about (call it B). The proper connection order is A establishes a connection (A1), then B attempts to establish a connection (which would become B1)… within a time frame of 200ms (I already doubled the timeout from 100ms which was written ages ago, so I thought I was being generous!). If it doesn’t get a B connection within 200ms, then it drops A1. So then B1 establishes a connection with the server’s kernel, waiting to be accepted. It only gets accepted on the next connection cycle when A2 establishes a connection, and the client also sends a B2 connection. The server accepts the A2 connection, then gets the first connection on the B queue, which is B1 (hasn’t been accepted yet – the queue looked like B1, B2). That is why the server didn’t send a FIN for B1 when the client had disconnected B1. So the two connections the server has are A2 and B1, which are obviously out of sync. It tries writing to B1, which is a dead connection, so it drops A2 and B1. Then the next pair are A3 and B2, which are also invalid pairs. They never recover from being out of sync until the server process is killed and the TCP connections are all reset.

So the solution was to just change a timeout for waiting on the B socket from 200ms to 5s. Such a simple fix that had me scratching my head for days (and fixed it within 24 hours of putting it on stackoverflow)! I also made it recover from stray B connections by adding socket B to the main select() call, and then accept()ing it and close()ing it immediately (which would only happen if the B connection took longer than 5s to establish). Thanks @AlanCurry for the suggestion of adding it to the select() and adding the puzzle piece about the listen() backlog parameter being a hint.