Question

I’m working on a project, where a primary server thread needs to dispatch events to a series of worker threads. The work that goes on in the worker threads relies on polling (ie. epoll or kqueue depending on the UNIX system in question) with timeouts on these operations needing to be handles. This means, that a normal conditional variable or semaphore structure is not viable for this dispatch, as it would make one or the other block resulting in an unwanted latency between either handling the events coming from polling or the events originating from the server thread.

So, I’m wondering what the most optimal construct for dispatching such events between threads in a pollable fashion is? Essentially, all that needs to be delivered is a pollable “signal” that tells the worker thread, that it has more events to fetch. I’ve looked at using UNIX pipes (unnamed ones, as it’s internal to the process) which seems like a decent solution given that a single byte can be written to the pipe and read back out when the queue is cleared — but, I’m wondering if this is the best approach available? Or the fastest?

Alternatively, there is the possibility to use signalfd(2) on Linux, but as this is not available on BSD systems, I’d rather like to avoid this construct. I’m also wondering how great the overhead in using system signals actually is?

Answer 1

Jan Hudec’s answer is correct, although I wouldn’t recommend using signals for a few reasons:

• Older versions of glibc emulated pselect and ppoll in a non-atomic fashion, making them basically worthless. Even when you used the mask correctly, signals could get “lost” between the pthread_sigprocmask and select calls, meaning they don’t cause EINTR.
• I’m not sure signalfd is any more efficient than the pipe. (Haven’t tested it, but I don’t have any particular reason to believe it is.)
• signals are generally a pain to get right. I’ve spent a lot of effort on them (see my sigsafe library) and I’d recommend avoiding them if you can.

Since you’re trying to have asynchronous handling portable to several systems, I’d recommend looking at libevent. It will abstract epoll or kqueue for you, and it will even wake up workers on your behalf when you add a new event. See event.c

2058 static inline int
2059 event_add_internal(struct event *ev, const struct timeval *tv,
2060     int tv_is_absolute)
2061 {
...
2189         /* if we are not in the right thread, we need to wake up the loop */
2190         if (res != -1 && notify && EVBASE_NEED_NOTIFY(base))
2191                 evthread_notify_base(base);
...
2196 }

Also,

The worker thread deals with both socket I/O and asynchronous disk I/O, which means that it is optimally always waiting for the event queuing mechanism (epoll/kqueue).

You’re likely to be disappointed here. These event queueing mechanisms don’t really support asynchronous disk I/O. See this recent thread for more details.