Question

So basically I’m making an MMO server in C++ that runs on linux. It works fine at first, but after maybe 40 seconds with 50 clients it will completely pause. When I debug it I find that basically the last frame its on before it stops responding is syscall() at which point it disappears into the kernel. Once it disappears into the kernel it never even returns a value… it’s completely baffling.

The 50 clients are each sending 23 bytes every 250 milliseconds. These 23 bytes are then broadcasted to all the other 49 clients. This process begins to slow down and then eventually comes to a complete halt where the kernel never returns from a syscall for the send() command. What are some possible reasons here? This is truly driving me nuts!

One option I found is Nagles algorithm which forces delays. I’ve tried toggling it but it still happens however.

Edit: The program is stuck here. Specifically, in the send, which in turn calls syscall()

bool EpollManager::s_send(int curFD, unsigned char buf[], int bufLen, int flag) 
//     Meant to counteract partial sends
{
    int sendRetVal = 0;
    int bytesSent = 0;
    while(bytesSent != bufLen)
    {
 print_buffer(buf, bufLen);
        sendRetVal = send(curFD, buf + bytesSent, bufLen - bytesSent, flag); 

        cout << sendRetVal << " ";
        if(sendRetVal == -1)
        {
            perror("Sending failed");
            return false;
        }
        else
            bytesSent += sendRetVal;
    }
    return true;
}

Also this is the method which calls the s_send.

    void EpollManager::broadcast(unsigned char msg[], int bytesRead, int sender)
    {
 for(iMap = connections.begin(); iMap != connections.end(); iMap++)
 {
  if(sender != iMap->first)
  {
   if(s_send(iMap->first, msg, bytesRead, 0)) // MSG_NOSIGNAL
   {
       if(debug)
       {
                    print_buffer(msg, bytesRead);
                    cout << "sent on file descriptor " << iMap->first << '\n';
       }
   }
  }
 }
 if(connections.find(sender) != connections.end())
        connections[sender]->reset_batch();
    }

And to clarify connections is an instance of boost’s unordered_map. The data that the program chokes on is not unique in any way either. It has been broadcast successfully to other file descriptors, but chokes on a, at least seemingly, random one.

Answer 1

The kernel keeps a finite buffer for sending data. If the receiver isn’t receiving, that buffer will fill up and the sender will block. Could that be the problem?