I have done strace on my multi-threaded c++ application running on linux
after couple hours of running, none of the threads got run, for about 12 seconds.
I have seen that the unfinished select system call which is called with a timeout was unfinished before the thread was suspended, reported after it resumed that, it took 11.x seconds for the operation to finish. (the timeout is only 900ms)
This is clear indication that the process got starved for a long time.
All threads in the process are created with default scheduling policy(SCHED_OTHER) of linux and default priority.
There are another 5 similar apps running on the same box which are also heavy I/O bound like this app due to heavy data received on the socket. But most of the time, this app is getting scheduled delay. The other apps are created with same sched policy and priority as this i.e. the defaults. why is only this process gets blocked almost all of the time?
Could it be because this process is more I/O intensive as in more busy due to may be higher rates of data? So, the linux dynamic priority adjusting in play here which pushed this process down?
Priority and process scheduling in Linux is only related to CPU time. In fact the Process Scheduler only cares about processes which wait to run on the CPU. Processes/threads which wait for I/O are not handled by the Process Scheduler but the I/O Scheduler.