Question

In my application, there is a io-thread, that is dedicated for

1. Wrapping data received from the application in a custom protocol
2. Sending the data+custom protocol packet over tcp/ip
3. Receiving data+custom protocol packet over tcp/ip
4. Unwrapping the custom protocol and handing the data to the application.

Application processes the data over a different thread. Additionally, the requirements dictate that the unacknowledged window size should be 1, i.e. there should be only one pending unacknowledged message at anytime. This implies that if io-thread has dispatched a message over the socket, it will not send any more messages, till it hears an ack from the receiver.
Application’s processing thread communicates to io-thread via pipe. Application needs to shut gracefully if someone from linux CLI types ctrl+C.
Thus, given these requirements, i have following options

1. Use PPoll() on socket and pipe descriptors
2. Use Select()
3. Use PSelect()

I have following questions

1. The decision between select() and poll(). My application only deals with less than 50 file descriptors. Is it okay to assume there would be no difference whether i choose select or poll ?

   1. Decision between select() and pselect(). I read the linux documentation and it states about race condition between signals and select(). I dont have experience with signals, so can someone explain more clearly about the race condition and select() ? Does it have something to do with someone pressing ctrl+C on CLI and application not stopping?

   2. Decision between pselect and ppoll() ? Any thoughts on one vs the other

Answer 1

I’d suggest by starting the comparison with select() vs poll(). Linux also provides both pselect() and ppoll(); and the extra const sigset_t * argument to pselect() and ppoll() (vs select() and poll()) has the same effect on each “p-variant”, as it were. If you are not using signals, you have no race to protect against, so the base question is really about efficiency and ease of programming.

Meanwhile there’s already a stackoverflow.com answer here: what are the differences between poll and select.

As for the race: once you start using signals (for whatever reason), you will learn that in general, a signal handler should just set a variable of type volatile sig_atomic_t to indicate that the signal has been detected. The fundamental reason for this is that many library calls are not re-entrant, and a signal can be delivered while you’re “in the middle of” such a routine. For instance, simply printing a message to a stream-style data structure such as stdout (C) or cout (C++) can lead to re-entrancy issues.

Suppose you have code that uses a volatile sig_atomic_t flag variable, perhaps to catch SIGINT, something like this (see also http://pubs.opengroup.org/onlinepubs/007904975/functions/sigaction.html):

volatile sig_atomic_t got_interrupted = 0;
void caught_signal(int unused) {
    got_interrupted = 1;
}
...
    struct sigaction sa;
    sa.sa_handler = caught_signal;
    sigemptyset(&sa.sa_mask);
    sa.sa_flags = SA_RESTART;
    if (sigaction(SIGINT, &sa, NULL) == -1) ... handle error ...
    ...

Now, in the main body of your code, you might want to “run until interrupted”:

    while (!got_interrupted) {
         ... do some work ...
    }

This is fine up until you start needing to make calls that wait for some input/output, such as select or poll. The “wait” action needs to wait for that I/O—but it also needs to wait for a SIGINT interrupt. If you just write:

    while (!got_interrupted) {
        ... do some work ...
        result = select(...); /* or result = poll(...) */
    }

then it’s possible that the interrupt will happen just before you call select() or poll(), rather than afterward. In this case, you did get interrupted—and the variable got_interrupted gets set—but after that, you start waiting. You should have checked the got_interrupted variable before you started waiting, not after.

You can try writing:

    while (!got_interrupted) {
        ... do some work ...
        if (!got_interrupted)
            result = select(...); /* or result = poll(...) */
    }

This shrinks the “race window”, because now you’ll detect the interrupt if it happens while you’re in the “do some work” code; but there is still a race, because the interrupt can happen right after you test the variable, but right before the select-or-poll.

The solution is to make the “test, then wait” sequence “atomic”, using the signal-blocking properties of sigprocmask (or, in POSIX threaded code, pthread_sigmask):

sigset_t mask, omask;
...
while (!got_interrupted) {
    ... do some work ...
    /* begin critical section, test got_interrupted atomically */
    sigemptyset(&mask);
    sigaddset(&mask, SIGINT);
    if (sigprocmask(SIG_BLOCK, &mask, &omask))
        ... handle error ...
    if (got_interrupted) {
        sigprocmask(SIG_SETMASK, &omask, NULL); /* restore old signal mask */
        break;
    }
    result = pselect(..., &omask); /* or ppoll() etc */
    sigprocmask(SIG_SETMASK, &omask, NULL);
    /* end critical section */
}

(the above code is actually not that great, it’s structured for illustration rather than efficiency — it’s more efficient to do the signal mask manipulation slightly differently, and place the “got interrupted” tests differently).

Until you actually start needing to catch SIGINT, though, you need only compare select() and poll() (and if you start needing large numbers of descriptors, some of the event-based stuff like epoll() is more efficient than either one).