Question

I have multiple daemons (one gateway and multiple service, all running on same node) out of which some of the service daemons need to respond in “soft real time” to the arriving requests on the network, my arch is like i have a gateway daemon which routes the incoming packets based on some protocol tag to the corresponding service daemons. the service daemons process the requests and send the responses back to the gw daemon which puts on the wire. all fine and working but i am not achieving the “soft real time” and seeing a lag.

I plan to improvise on this in below way, sharing the network connection between gateway and the service daemons, i will have a notification scheme by which, when the packets arrive on the connection the gw daemon with out de-queuing the packet from the socket queue looks at the protocol header and “notifies” the corresponding service daemon that “data has arrived”, on receiving the notification the service daemon grabs a binary semaphore and de-queues the data from the socket queue. there will be 2 such semaphores one for writing and the other for reading. when the service daemon needs to send data it grabs the write semaphore and sends the data. when it receives the “data arrival ” notification from the gateway daemon, it grabs the read semaphore and de-queues the data from the socket. On every new connection request the gateway daemon will send the connection to the service daemons using “sendmsg”.

Did any body tried this scheme any time ? do you see any problems with this approach ? pls comment/advise.

Answer 1

If you want to avoid copy overhead you should probably be using splice, rather than trying to share sockets between multiple daemons. That solution is going to be fiendishly difficult to debug and maintain.

I expect (and hope) that your network protocol has a header which makes it easy for the gateway to know where to route a packet to, followed by a payload destined for the service daemon.

In pseudocode the gateway does this:

while (data on socket)
{
    header = read(socket, sizeof(header));

    service_socket = find_service(header);
    splice(socket, NULL, service_socket, NULL, header->payload_length, 0);
}