Question

I have a single producer multiple consumer program with threads for each role. I am thinking of implementing a circular buffer for tcp on each of the consumers and allow the producer to keep pointers to the circular buffers’ memory then handing out pointer space to the tcp to offload data into.

My problem, how to have consumer threads know when data is in?

I am thinking of busy wait checking the pointer location for something other than a 0; I don’t mind being a cpu hog.

I should mention each thread is cpuset and soft RT by SCHED_FIFO, and of course c implemented.

Answer 1

In my experience, the problem with multiple consumer datastructures is to properly handle concurrency while avoiding issues with false sharing or excessivly wasting CPU cycles.

So if your problems allow it, I would use pipe to create a pipe to each consumer and putting items into these pipes in a round robin fashion. The consumers can then use epoll to watch the file handles. This avoids having to implement and optimize a concurrent datastructure and you won’t burn CPU cycles needlessly. The cost is that you have to go through syscalls.

If you want to do everything yourself with polling to avoid syscalls, you can build a circular buffer but you have to make sure that only one process reads an item at the same time and only after the item has been written. Usually this is done with 4 pointers and proper mutexes.

This article about Xen’s I/O ringbuffers might be of interest.