I am operating on the .NET Framework 4.5 using C#
I wish to measure code performance but have the additional complication that I would like to have a comparison between machines (different hardware).
One of the core goals is to come up with benchmarks that compare algorithms (algo X vs algo Y using dataset Z) which is fine if I consistently use the same exact hardware but I would also like to have the option to distribute these performance tests across many machines – which are mostly different.
How can I efficiently measure the performance of a particular machine?
I am currently using the System.Diagnostics.StopWatch class mixed with a Fibonacci suite, trying to measure how many sequences the machine can handle in order to use X ticks/milliseconds. However as you may already know this technique is not very precise.
Is the answer in Stopwatch.Frequency?
Anyone have any better suggestions?
ADDED INFORMATION —–
Example: Compare algorithms that are not multi-threaded (number of cores won’t matter) say like the difference between running a sequential scan and using a red/black tree.
simply: you can’t. Or at least, not reliably.
For example, I once had a discussion with someone about static methods and threading in .NET, and I ran a test of a couple of machines. Running the same executable, the machine with single core outperformed the dual-core machine. Obviously the problem there was the .NET runtime using different internal algorithms to determine safety (ie it put more locks in on the dual-core machine than were needed on the single-core, or the runtime was different between the single-core workstation and the dual-core server). The point being that you can’t even run the same executable to get a different baseline.
So if you ran algorithm X on machines A and B, and record the difference (lets say machine B was twice as fast), you cannot then run algorithm Y on machine A and assume that the algorithm would run twice as fast if executed on machine B.
The complexity of the factors you’d have to take into account would be too great. All you can do is compare the same thing on different machines, if you’re testing machine performance, or different things on the same machine, etc.