Question

Lets make this example simple. In C++ I have a performance critical application and it makes numerous calls to the database which blocks. However i need the data before going on so i do need to wait. I would like to yield and let another task run.

Threads are expensive so i have that in mind. My question is since i need to block how do i keep the application fast? It feels like since the API blocks (lets say i am using fopen or a mysql query call) i i cant exactly call runNextTaskWhileIWait(); It also seems i HAVE TO create threads.

What can i do to keep performance up while i have to deal with blocking calls?

-edit- Is the performance better if put up with it and let it block so it can do its business? or is there something i can do to get better performance?

Answer 1

threads are not expensive if they are used correctly, and when they are the right tool for the job (alternatives exist). typically, threads are ‘expensive’ if you:

• create them too often
• you are distributing the workload incorrectly (are there too many active threads? are your tasks too small? are they all competing for the same resource?).
• too much concurrent communications, or your granularity of locking mechanisms is not optimal.

some considerations:

• is the task appropriate for parallelization? some are not, and some should be made larger.
• how do you create threads? creating a thread is relatively expensive, but you can avoid this in many cases by using a thread pool.
• using your current thread creation model, how many threads could your program create per core? if you create too many, you’re not going to benefit much from multithreading.
• what resource are you competing for where multithreading will help? in this case, it may be the network or disk, but you may continue execution of cpu and memory intensive tasks. be sure you distribute this well if you need execution to be fast (try smaller thread pools which specify the target of the resource they are bound to use). as well, you may want to consider how your tasks read from disk, and if that can be improved.
• similarly, you can localize and marshall tasks/resources to the database, since if it operates in serial.
• in some cases, threads are the wrong answer. creating a thread for a task which is truly synchronous can just slow things down.

assuming you have done everything right so far, my first approach would be:

• localize all database accesses to a small, dedicated thread with a task pool
• to reconsider how your program executes – remove the synchronous (wait) requirement if it’s not too much work.

if you ever take the time to implement a large concurrent program correctly, it will (quite likely) radically change how you approach concurrent problems, and programs in general. a lot of the higher level tools/libraries available today (for c++) target parallelization of existing programs. many also target simple and localized cases (e.g. parallelize this loop). there is a big difference between a program that has been or is being adapted to utilize concurrency and a program which is has been developed for concurrent execution by design, both in flow and performance.

-edit- Is the performance better if put up with it and let it block so it can do its business?

that depends on the task and the frequency. consider how it will distribute. if the implementation you go through enforces serial operation, making 100 requests at once will only slow things down because you are just creating tasks which can require a lot of resources while increasing contest for the given resource. it could just lead to creating many threads and allocations, introduce a lot of locking, and introduce a lot of context switching.

or is there something i can do to get better performance?

ideally, you would understand how the interfaces you use operate, and update your program to flow optimally for concurrent execution. this often requires reading the docs for the apis in question to understand how they execute, profiling, understanding where your existing problems are, and identifying programs/problems which are good candidates for parallelization. learning the mechanisms for concurrency is one step, implementing thread safe programs is another, learning to write optimal high performance concurrent implementations and implementing them well is arguably more difficult than the other two combined. many devs stop at #1 or #2.