Is there any way I can log the CPU activities, like “time-stamp: actual activity” (like context switching, etc.)? It should be like syslog, sar or top, however with more fine grained time values (in nano seconds).
Please let me know if there is a tool which can do this else I am ready for recompiling the Kernel code as well.
I have read a related SO question in this context, where the discussion was mainly related to threads and checked other questions too, however did not get any lead..
Thank you.
[EDIT]: I am looking for a tool(API)/code-change suggestion which can work only at Kernel level, as my context has nothing to do with Application/User space.
[EDIT2]: Brief background: I modified Linux/Android SDIO UART driver and performed benchmarking tests on it (metrics: data transfer speed). The changes which I did indeed improved the performance, however it posed a few questions too. Theoretically the data write time should have been approx 50ns however I observed it to be approx 200ns. In the quest to understand the latency/delay of 150ns, would like to track what CPU does so that I get some cues/answer.
Now, my hopes are on systemtap.
You can’t easily record CPU activities, but you can record your software and kernel activities.
Oprofile can is a sampling profiler that records a stack trace. Can record the stack of both the user-space application and the kernel.
Systemtap is another tool (similar to much touted DTrace) that allows to instrument applications and kernel and get detailed stack traces with function arguments and timestamps.
Using Systemtap you can benchmark functions in the Linux kernel. When analyzing performance on a running system, it is not enough to get min/max/mean/median/stddev because your mean is often going to be too noisy, e.g. the mean of your write function execution time can be 150nsec and stddev is 100nsec, obviously that does not tell you much. Because a picture is worth a thousand words, a good start would be to identify one or more interesting functions in your driver, sample their execution times and then display the sample set as a histogram.
This can be done by inserting probes at function entry and return. The function entry probe saves the current time as entry time. The function return probe takes current time and subtracts the entry time to get the function execution time. Use the function execution time as a subscript/key into an array (Systemtap arrays are hash tables in fact) and increment the sample count associated with that key. After you’ve collected enough samples (you can count them) finish the Systemtap session. On session exit display the array as a histogram, Systemtap provides a function that does it.
I don’t provide any samples because Systemtap has an excellent tutorial.