Question

I am implementing a Windows-based web server handling multiple specific HTTP requests from clients using WinSock2. I have a class to start and stop my server. It looks something like this:

class CMyServer
{
  // Not related to this question methods and variables here
  // ...

public:

  SOCKET m_serverSocket;

  TLM_ERROR Start();
  TLM_ERROR Stop();
  static DWORD WINAPI ProcessRequest(LPVOID pInstance);
  static DWORD WINAPI Run(LPVOID pInstance);
}

where TLM_ERROR is a type definition for my server’s errors enumeration.

bool CMyServer::Start() method starts the server creating a socket listening on configured port and creating a separate thread DWORD CMyServer::Run(LPVOID) to accept incoming connections like described here:

  // Creating a socket
  m_serverSocket = ::socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
  if (m_serverSocket == INVALID_SOCKET)
    return TLM_ERROR_CANNOT_CREATE_SOCKET;

  // Socket address
  sockaddr_in serverSocketAddr;
  serverSocketAddr.sin_family = AF_INET;                                  // address format is host and port number  
  serverSocketAddr.sin_addr.S_un.S_addr = inet_addr(m_strHost.c_str());   // specifying host
  serverSocketAddr.sin_port = htons(m_nPort);                             // specifying port number

  // Binding the socket
  if (::bind(m_serverSocket, (SOCKADDR*)&serverSocketAddr, sizeof(serverSocketAddr)) == SOCKET_ERROR)
  {
    // Error during binding the socket
    ::closesocket(m_serverSocket);
    m_serverSocket = NULL;
    return TLM_ERROR_CANNOT_BIND_SOCKET;
  }

  // Starting to listen to requests
  int nBacklog = 20;
  if (::listen(m_serverSocket, nBacklog) == SOCKET_ERROR)
  {
    // Error listening on socket
    ::closesocket(m_serverSocket);
    m_serverSocket = NULL;
    return TLM_ERROR_CANNOT_LISTEN;
  }

  // Further initialization here...
  // ...

  // Creating server's main thread
  m_hManagerThread = ::CreateThread(NULL, 0, CTiledLayersManager::Run, (LPVOID)this, NULL, NULL);

I use ::accept(...) to wait for incoming client connections in CMyServer::Run(LPVOID), and after new connection has been accepted I create a separate thread CMyServer::ProcessRequest(LPVOID) to receive a data from a client and send a response passing the socket returned by ::accept(...) as part of thread function’s argument:

DWORD CMyServer::Run(LPVOID pInstance)
{
  CMyServer* pTLM = (CMyServer*)pInstance;

  // Initialization here...
  // ...

  bool bContinueRun = true;
  while (bContinueRun)
  {
    // Waiting for a client to connect
    SOCKADDR clientSocketAddr;                            // structure to store socket's address
    int nClientSocketSize = sizeof(clientSocketAddr);     // defining structure's length
    ZeroMemory(&clientSocketAddr, nClientSocketSize);     // cleaning the structure
    SOCKET connectionSocket = ::accept(pTLM->m_serverSocket, &clientSocketAddr, &nClientSocketSize);      // waiting for client's request
    if (connectionSocket != INVALID_SOCKET)
    {
      if (bContinueRun)
      {
        // Running a separate thread to handle this request
        REQUEST_CONTEXT rc;
        rc.pTLM = pTLM;
        rc.connectionSocket = connectionSocket;
        HANDLE hRequestThread = ::CreateThread(NULL, 0, CTiledLayersManager::ProcessRequest, (LPVOID)&rc, CREATE_SUSPENDED, NULL);

        // Storing created thread's handle to be able to close it later
        // ...

        // Starting suspended thread
        ::ResumeThread(hRequestThread);
      }
    }

    // Checking whether thread is signaled to stop...
    // ...
  }

  // Waiting for all child threads to over...
  // ...
}

Testing this implementation manually gives me the desired results. But when I send multiple requests generated by JMeter I can see that some of them are not handled properly by DWORD CMyServer::ProcessRequest(LPVOID). Looking at log file created by ProcessRequest I determine 10038 WinSock error code (meaning that ::recv call was tried on nonsocket), 10053 error code (Software caused connection abort) or even 10058 error code (Cannot send after socket shutdown). But the 10038th error occurs more often than others mentioned.

It looks like a socket was closed somehow but I close it only after ::recv and ::send have been called in ProcessRequest. I also thought that it can be an issue related to using ::CreateThread instead of ::_beginthreadex but as I can get it could only lead to memory leaks. I don’t have any memory leaks detected by the method described here so I have doubts that it is the reason. All the more, ::CreateThread returns a handle that can be used in ::WaitForMultipleObjects to wait for threads to be over, and I need it to stop my server properly.

Could these errors occur due to a client doesn’t want to wait for response anymore? I am out of ideas, and I will thank you if you tell me what I am missing or doing/understanding wrong. By the way, both my server and JMeter run on the localhost.

Finally, here is my implementation of ProcessRequest method:

DWORD CMyServer::ProcessRequest(LPVOID pInstance)
{
  REQUEST_CONTEXT* pRC = (REQUEST_CONTEXT*)pInstance;
  CMyServer* pTLM = pRC->pTLM;
  SOCKET connectionSocket = pRC->connectionSocket;

  // Retrieving client's request
  const DWORD dwBuffLen = 1 << 15;
  char buffer[dwBuffLen];
  ZeroMemory(buffer, sizeof(buffer));
  if (::recv(connectionSocket, buffer, sizeof(buffer), NULL) == SOCKET_ERROR)
  {
    stringStream ss;
    ss << "Unable to receive client's request with the following error code " << ::WSAGetLastError() << ".";
    pTLM->Log(ss.str(), TLM_LOG_TYPE_ERROR);
    ::SetEvent(pTLM->m_hRequestCompleteEvent);
    return 0;
  }

  string str = "HTTP/1.1 200 OK\nContent-Type: text/plain\n\nHello World!";
  if (::send(connectionSocket, str.c_str(), str.length(), 0) == SOCKET_ERROR)
  {
    stringStream ss;
    ss << "Unable to send response to client with the following error code " << ::WSAGetLastError() << ".";
    pTLM->Log(ss.str(), TLM_LOG_TYPE_ERROR);
    ::SetEvent(pTLM->m_hRequestCompleteEvent);
    return 0;
  }

  ::closesocket(connectionSocket);
  connectionSocket = NULL;

  pTLM->Log(string("Request has been successfully handled."));
  ::SetEvent(pTLM->m_hRequestCompleteEvent);
  return 0;
}

Answer 1

You pass a pointer to the REQUEST_CONTEXT to every newly created thread. However this is an automatic variable, allocated on the stack. Hence its lifetime is limited to its scope. It ends right after you call ResumeThread.

Practically what happens is that the same memory for REQUEST_CONTEXT is used in every loop iteration. Now imagine you accept 2 connections in a short time internal. It’s likely that at the time the first thread starts execution its REQUEST_CONTEXT will already be overwritten. So that you actually have 2 threads serving the same socket.

The easiest fix is to allocate the REQUEST_CONTEXT dynamically. That is, allocate it upon new accept, pass its pointer to the new thread. Then during the thread termination don’t forget to delete it.