Question

I have some questions regarding threads:

1. What is the maximum number of threads allowed for a process before it decreases the performance of the application?
2. If there’s a limit, how can this be changed?
3. Is there an ideal number of threads that should be running in a multi-threaded application? If it depends on what the application is doing, can you cite an example?
4. What are the factors to consider that affects these performance/thread limit?

Answer 1

This is actually a hard set of questions to which there are no absolute answers, but the following should serve as decent approximations:

1. It is a function of your application behavior and your runtime environment, and can only be deduced by experimentation. There is usually a threshold after which your performance actually degrades as you increase the number of threads.

2. Usually, after you find your limits, you have to figure out how to redesign your application such that the cost-per-thread is not as high. (Note that for some domains, you can get better performance by redesigning your algorithm and reducing the number of threads.)

3. There is no general “ideal” number of threads, but you can sometimes find the optimal number of threads for an application on a specific runtime environment. This is usually done by experimentation, and graphing the results of benchmarks while varying the following:

   • Number of threads.
   • Buffer sizes (if the data is not in RAM) incrementing at some reasonable value (e.g., block size, packet size, cache size, etc.)
   • Varying chunk sizes (if you can process the data incrementally).
   • Various tuning knobs for the OS or language runtime.
   • Pinning threads to CPUs to improve locality.

4. There are many factors that affect thread limits, but the most common ones are:

   • Per-thread memory usage (the more memory each thread uses, the fewer threads you can spawn)
   • Context-switching cost (the more threads you use, the more CPU-time is spent switching).
   • Lock contention (if you rely on a lot of coarse grained locking, the increasing the number of threads simply increases the contention.)
   • The threading model of the OS (How does it manage the threads? What are the per-thread costs?)
   • The threading model of the language runtime. (Coroutines, green-threads, OS threads, sparks, etc.)
   • The hardware. (How many CPUs/cores? Is it hyperthreaded? Does the OS loadbalance the threads appropriately, etc.)
   • Etc. (there are many more, but the above are the most important ones.)