I have a data feed continuously feeding data packet in. There are 5 threads(A, B, C, D, E) processing the data packages. Note the 5 threads have totally different speed and they generate 5 different features(each thread generate 1 feature) for every incoming data package.
The 5 threads are at different pace: when A has finished analyzing first 10 packages, B might only have finished package 1, package 2, and C might have not even finish a single package at all.
My task is to match the results from 5 threads, and start the final analysis when all the 5 features for the first 10 data package are available.
My question is:
– How to combine the results from different threads making sure the analysis thread is only triggered when a certain amount of result are available?
– I seems that I need a aggregator thread checking the availability of different buffers. I am thinking in terms of lock/condition. How could I implement such a condition involving different buffers?
Totally newbie in multithreading. Any suggestion is welcomed.
I am using GNU C++ with Boost library.
Have yourself an “aggregator” thread: this thread would get its input from the worker threads (through non-blocking thread-safe queues I suggest) and once a “batch” is ready, push it to your “analyzer” thread.
Queues offer the advantage of not blocking any of the workers: the “aggregator” just has to poll the worker queues (through a condition section). You can control the rate of polling to your liking.
This solution goes around the problem of “synchronize all” situations.