On Linux, for a C/C++ program using pthreads, is there a way to see how threads were scheduled for the life of a process? I’d like to see which CPU(s) on which every thread runs. I’d like to see when threads were pre-empted (and why).
We have set up a test environment where two identical machines run identical processes. There is a third machine generating “clock” events to which the two machines listen (sent via multicast). The machines’ processes do some stuff at each clock, then send a result to the third machine. The idea is that the third machine eliminates the clock synchronization problem (between the two identical machines). Our expectation is that the results coming back should be at exactly (or almost exactly) the same time. And generally they are. The problem is, we see occasional spikes, where one result is dramatically delayed (by an order of 10x the standard deviation of all results).
We are looking at micro-second level optimization. In this arena, cache misses and thread wake-up times become an issue. It is known that the total number of threads across all processes is greater than the number of CPU cores.
I suspect that these spikes are caused by the occasional “perfect storm” of thread preemption, thread-CPU migration (and therefore cache miss). Within the various processes, there are actually only two or three “important” threads doing time-sensitive work. The rest are ancillary/support threads, of lower priority. In total, the number of important/time-sensitive threads is actually equal to (or less than) the number of cores.
I suspect that the solution to this is to carefully assign important threads to their own core(s), and dump all the support threads onto their own core. But this would require a fair amount of development effort, and I’d like to confirm my suspicions before heading down this road.
strace – simple tool to trace process behavior, but seems can not check the core id.
lttng – need patch the kernel, but more impressive, you can know exactly what each core is doing like context switch/interrupt handling.
Update: As Mathieu Desnoyers mentioned, lttng can be used as Linux kernel modules since 2.6.36, no kernel patching needed now.