Question

I have an application with multiple threads, one thread is creating 4 tcp connections, then this thread creates another thread that will handle the receive function for these 4 connections using poll or anything else, and the original thread (the first one) start sending messages to these 4 connections(round-robin). it’s like below pseudo code,

main()
{

    pthread_create( &thread1, NULL, sending_thread, (void*) args);

}

void *sending_thread( void *ptr )
{
    int i=0;
    int *connections;
    connections = create_connections();

    pthread_create( &thread1, NULL, receiving_thread, (void*)&connections);
        while(1) {

        send(connections[i], "test"); 
        i++;
        if (i > 3)
        i=0;             

        }
}

void *receiving_thread( void *ptr )
{
    char *msg; 
        msg = receive(connections[i]);
        save_received_msg_to_disk(msg);

}

My question is How can I check my connections and bring up the disconnected one? for example, let’s say connection1 went down, do I need to create another connection with the same fd, which is connection[1] in this case? or is there other ways to handle this case?

Environment is C/pthread in Linux

Answer 1

Here are some points based on your code and comments.

• While threads do allow some things happen in parallel, they usually have a non-zero cost in terms of more involved design that always boils down to synchronization. Like in your case there’s no easy way of re-establishing one of the connections and sharing it between two threads again.
• I haven’t ever seen an app where input and output streams are completely independent. I can imagine something like a tunneling proxy or a TCP-based VPN where that might make sense, but in general case some higher level protocol still imposes some request-response semantics that again would impose/require arbitration among sending and receiving threads.
• Things are different when you need throughput and when you try to minimize latency. Sleep-waiting on a mutex is usually OK for the former, but is rarely a good idea for the latter. Round-robin writing to several blocking sockets from a single thread hurts both.
• If you have heavy output stream(s) and sparse input, it only makes sense to use something like epoll(7) to detect saturated connections and be notified when they are available again then starve others.

I know this doesn’t answer your question directly, but my rant list did not fit into the comments. Hope this helps a little.

Edit 0:

Here’s the usual setup with epoll(7):

• Make your sockets non-blocking (fcntl(2) with O_NONBLOCK).
• Set epoll_data.fd to your socket descriptor for each potential channel (four sockets in your example). Other options are possible with union epoll_data if you want to keep more complex structures then just socket descriptors.
• Use EPOLLIN and EPOLLET to get edge-triggered behavior, i.e. be woken up when input buffer becomes not empty.
• Only set EPOLLOUT if you get EWOULDBLOCK from a write(2), otherwise do output as usual. Same logic here with EPOLLET to detect output buffer space becoming available.
• Use EPOLLRDHUP to detect other side disconnecting cleanly (for abrupt disconnects you need to handle EPIPE error form write(2)).
• epoll_wait(2) gives you back number of events to iterate through. Do separate checks for input (events & EPOLLIN) and output (events & EPOLLOUT).
• On input read from data.fd (or otherwise associated socket) until you get EWOULDBLOCK.
• On output write until you get EWOULDBLOCK or you don’t have more output data pending (remove EPOLLOUT in that case).

It looks like a lot, but is pretty simple once you get a hang of it.

You can also do non-blocking connect(2), which is probably a good idea if you ever want to re-establish broken streams without hurting others that are still chugging along (connect(2) returns -1 with errno(3) set to EINPROGRESS and you wait for the socket to become writable as above).