I understand the difference between “real”,”user” and “sys” when you use the time command on Linux, as explained on this other thread: What do 'real', 'user' and 'sys' mean in the output of time(1)?
Now I am working on a small comparison between the performance of Python, Java and C, and I am wondering which report I should use.
“User+sys” seems to be the more realistic one, but wouldn’t this cause problems when comparing C to Java, for instance, cause the JVM knows how to optimize the code for multi-processors/threads while GCC doesn’t?
Also, wouldn’t “real” be realistic enough if I make sure no other heavy process is running on the background?
The answer will depend on what you mean by “the performance of (Python|Java|C)”. In many cases what a user really cares about is the elapsed wall time, corresponding to
real. Suppose you write some piece of code in a reasonable way in several languages and one of the languages can automatically parallelize it to use your 4 cores. If this makes the user wait less time for a reply, then I say this is a fair comparison. Of course it is valid for that particular machine, the results on a single core machine could be different. If an app causes page faults, then it makes the user wait. For the user it’s no help if you say the app took fewer cycles if they have to wait longer.Any way you measure, be sure to repeat the tests multiple times, as there can be lots of variation between runs. Languages like Java also need a program to run for some time before it reaches top speed, due to JIT compilation (but again: if your program is very short by definition and doesn’t allow the Java Virtual Machine to warp up, then well it’s too bad for Java). Testing performance is very tricky and even experienced developers are prone to misinterpreting results or measuring not what they really intended.