I’ve been asked to measure the performance of a fortran program that solves differential equations on a multi-CPU system. My employer insists that I measure FLOP/s (Floating operations per second) and compare the results with benchmarks (LINPACK) but I am not convinced that it’s the way to go, simply because no one can explain to me what a FLOP is.
I did some research on what exactly a FLOP is and I got some pretty contradicting answers. One of the most popular answers I got was ‘1 FLOP = An addition and a multiplication operation’. Is that true? If so, again, physically, what exactly does that mean?
Whatever method I end up using, it has to be scalable. Some of versions of the code solve systems with multi-million unknowns and takes days to execute.
What would be some other, effective, ways of measuring performance in my case (summary of my case being ‘fortran code that does a whole lot of arithmetic calculations over and over again for days on several hundred CPUs)?
It’s a pretty decent measure of performance, as long as you understand exactly what it measures.
FLOPS is, as the name implies FLoating point OPerations per Second, exactly what constitutes a FLOP might vary by CPU. (Some CPU’s can perform addition and multiplication as one operation, others can’t, for example). That means that as a performance measure, it is fairly close to the hardware, which means that 1) you have to know your hardware to compute the ideal FLOPS on the given architecture, and you have to know your algorithm and implementation to figure out how many floating point ops it actually consists of.
In any case, it’s a useful tool for examining how well you utilize the CPU. If you know the CPU’s theoretical peak performance in FLOPS, you can work out how efficiently you use the CPU’s floating point units, which are often one of the hard to utilize efficiently. A program which runs 30% of the FLOPS the CPU is capable of, has room for optimization. One which runs at 70% is probably not going to get much more efficient unless you change the basic algorithm. For math-heavy algorithms like yours, that is pretty much the standard way to measure performance. You could simply measure how long a program takes to run, but that varies wildly depending on CPU. But if your program has a 50% CPU utilization (relative to the peak FLOPS count), that is a somewhat more constant value (it’ll still vary between radically different CPU architectures, but it’s a lot more consistent than execution time).
But knowing that ‘My CPU is capable of X GFLOPS, and I’m only actually achieving a throughput of, say, 20% of that’ is very valuable information in high-performance software. It means that something other than the floating point ops is holding you back, and preventing the FP units from working efficiently. And since the FP units constitute the bulk of the work, that means your software has a problem.
It’s easy to measure ‘My program runs in X minutes’, and if you feel that is unacceptable then sure, you can go ‘I wonder if I can chop 30% off that’, but you don’t know if that is possible unless you work out exactly how much work is being done, and exactly what the CPU is capable of at peak. How much time do you want to spend optimizing this, if you don’t even know whether the CPU is fundamentally capable of running any more instructions per second?
It’s very easy to prevent the CPU’s FP unit from being utilized efficiently, by having too many dependencies between FP ops, or by having too many branches or similar preventing efficient scheduling. And if that is what is holding your implementation back, you need to know that. You need to know that ‘I’m not getting the FP throughput that should be possible, so clearly other parts of my code are preventing FP instructions from being available when the CPU is ready to issue one’.
Why do you need other ways to measure performance? What’s wrong with just working out the FLOPS count as your boss asked you to? 😉