Question

Working on an asynchronous network programming for a p2p application, I’m having trouble.
My application has to be a server and a client both. When the server receives a
request it has to broadcast it to k other servers. I’ve thought that the HTTP Server 3 Example of the boost::asio examples could work well, with an implementation of the asynchronous client (as a class) in it.
The client class mentioned above (from the boost::asio client examples) is the following:

  ClientIO::ClientIO(boost::asio::io_service& io_service, tcp::resolver::iterator endpoint_iterator)
:   _io_service(io_service),
      strand_(io_service),
      resolver_(io_service),
  socket_(io_service)
  {
  tcp::endpoint endpoint = *endpoint_iterator;
      socket_.async_connect(endpoint,
      boost::bind(&ClientIO::handle_after_connect, this,
      boost::asio::placeholders::error, ++endpoint_iterator));
  }

  void ClientIO::write(G3P mex)
  {
  _io_service.post(boost::bind(&ClientIO::writeMessage, this, mex));
  }

  void ClientIO::writeMessage(G3P mex)
  {
  bool write_in_progress = !messages_queue_.empty();
  messages_queue_.push_back(mex);
  if (!write_in_progress)
  {
    char* message=NULL;
    boost::system::error_code ec;
    if (messages_queue_.front().opcode == DATA)
    {
      message=(char*)malloc((10800)*sizeof(char));
    }
    else
      message=(char*)malloc(1024*sizeof(char));

    boost::asio::streambuf request;
    std::ostream request_stream(&request);
    serializeMessage(message, messages_queue_.front());
    request_stream   << message;
    boost::asio::async_write(socket_, boost::asio::buffer(message, strlen(message)),
    strand_.wrap(
    boost::bind(&ClientIO::handle_after_write, this,
    boost::asio::placeholders::error)));
      free(message);
  }
  }

  void ClientIO::readMessage()
  {
boost::asio::async_read(socket_, data_,
    boost::bind(&ClientIO::handle_after_read, this,
boost::asio::placeholders::error,
boost::asio::placeholders::bytes_transferred
    ));
  }

  void ClientIO::stop()
  {
  socket_.shutdown(tcp::socket::shutdown_both);
  socket_.close();
  }

  void ClientIO::handle_after_connect(const boost::system::error_code& error,
    tcp::resolver::iterator endpoint_iterator)
  {
  if (error)
  {
    if (endpoint_iterator != tcp::resolver::iterator())
    {
      socket_.close();
      tcp::endpoint endpoint = *endpoint_iterator;
      socket_.async_connect(endpoint,
      boost::bind(&ClientIO::handle_after_connect,this,
      boost::asio::placeholders::error, ++endpoint_iterator));
    }
  }
  else
  {
  }
  }

  void ClientIO::handle_after_read(const boost::system::error_code& error, std::size_t bytes_transferred)
  {
  if (bytes_transferred > 0)
  {
    std::istream response_stream(&data_);
    std::string mex="";
    std::getline(response_stream, mex);
    deserializeMessage(&reply_,mex);
    if (reply_.opcode == REPL)
    {
      cout << "ack received" << endl;
    }
  }
  if (error)
  {
    ERROR_MSG(error.message());
  }
  }

  void ClientIO::handle_after_write(const boost::system::error_code& error)
  {
  if (error)
  {
  //            ERROR_MSG("Error in write: " << error.message());
  }
  else
  {
    messages_queue_.pop_front();
    if (!messages_queue_.empty())
    {
      cout << "[w] handle after write" << endl;
      char* message;
      if (messages_queue_.front().opcode == DATA)
      {
    message=(char*)malloc((10800)*sizeof(char));
      }
      else
    message=(char*)malloc(1024*sizeof(char));
      boost::asio::streambuf request;
      std::ostream request_stream(&request);
      serializeMessage(message, messages_queue_.front());
      request_stream << message;
      boost::asio::async_write(socket_, boost::asio::buffer(message, strlen(message)),
      strand_.wrap(
      boost::bind(&ClientIO::handle_after_write, this,
      boost::asio::placeholders::error)));
    }

    boost::asio::async_read_until(socket_, data_,"\r\n",
            strand_.wrap(
            boost::bind(&ClientIO::handle_after_read, this,
        boost::asio::placeholders::error,
        boost::asio::placeholders::bytes_transferred)));
  }
  }

  ClientIO::~ClientIO()
  {
  cout << "service stopped" << endl;
  }

}

When a new request is received by the server, it starts a new DataManagement class form connection and after some computation, write a queue
to the other servers (here just one) using the class above and at each write has to correspond an ack

client  --write-> server ---write->\ 
                                    |--server1
                 server <--ACK----</ 

To achieve that, I’ve created an io_service instance (io_service_test) as a class variable, instantiating it with the following in the DataManagement constructor:

DataManagement::DataManagement(){
  tcp::resolver resolver(io_service_test);
  tcp::resolver::query query(remotehost, remoteport);
  tcp::resolver::iterator iterator = resolver.resolve(query);
  cluster = new cluster_head::ClusterIO(io_service_test,iterator);
  io_service_test.run_one();
}

Then, after a computation, send the data:

 void DataManagement::sendTuple( . . . ){

  . . . 
  io_service_test.reset();
  io_service_test.run();
  for (size_t i=0; i<ready_queue.size() ;i++)
  {
      cluster->write(fragTuple);
  }
}

The counterpart is the same http proxy3 example modified in the same way (without the client class). The problem is that sometimes everything works good, sometimes it fails and I get a stack trace, sometimes it never stops, or even segmentation faults.
I think that the problem is closed to the io_service management and the life of the class methods, but i can’t figure out.

• Any ideas?
• have you some examples that fits this case, or a dummy class which implement it?

Answer 1

Briefly reviewed the code, I see the following problems.

1. The ClientIO::writeMessage method sends wrong information to recipients, because it
   • allocates memory for message
   • calls boost::asio::async_write, which does not send any data but only puts a request to an internal ASIO’s request’s queue, i.e. the message will be send in sometime. The boost::asio::buffer does not copy the message. It stores only a reference to it.
   • calls free(message). i.e. memory allocated to the message can be overwritten when the queued write request will be executed.
2. A memory leak in ClientIO::handle_after_write. message is allocated but not freed.
3. The boost::asio::async_read method of the ClientIO::readMessage is not wrapped by the strand_.wrap call.

To avoid the issues #1 and #2 is necessary to use something like the shared_const_buffer class of the ASIO’s buffers example.
To fix the issue #3 is necessary to use the strand_.wrap call in the same way as in the boost::asio::async_write calls.