Question

I have a multithreaded program which uses pthread mutexes for synchronization on few integers.

Of course it doesn’t scale well.

I read that the best way to share data is to eliminate ALL shared data – but what can I do if I need that overall “task count” (which is determined by shared integer(s)) shouldn’t be larger than this value? Consider that simple divide this integer between threads doesn’t work – one thread can run faster than others and stops when other continue working.

Also, using atomic operations on shared integer doesn’t work too – when thread count is huge, performance degrade because of write sharing.

Update: Its very undesirable to suspend threads, I need not only scalability but also deterministic time of delays in threads.

Answer 1

Give each thread its own counter. (Either in thread-local storage or on a page dedicated to that thread to make sure you don’t have bouncing cache lines.) When you need to provide an “overall task count”, have a control thread read the counters from all the other threads and sum them up in one place.

If you want a perfect count, you could tell all the threads to suspend working. If you don’t mind the count being close but not perfect let them keep running. Since each thread only ever writes its own data it ought to run significantly faster than using a mutex on a shared counter. Integer writes are atomic on nearly all platforms except SPARC; if you don’t care about SPARC, you don’t need to do anything to prevent partial-writes to integers.