Question

I have a series of functions that are all designed to do the same thing. The same inputs produce the same outputs, but the time that it takes to do them varies by function. I want to determine which one is ‘fastest’, and I want to have some confidence that my measurement is ‘statistically significant’.

Perusing Wikipedia and the interwebs tells me that statistical significance means that a measurement or group of measurements is different from a null hypothesis by a p-value threshold. How would that apply here? What is the null hypothesis between function A being faster than function B?

Once I’ve got that whole setup defined, how do I figure out when to stop measuring? I’ll typically see that a benchmark is run three times, and then the average is reported; why three times and not five or seven? According to this page on Statistical Significance (which I freely admit I do not understand fully), Fisher used 8 as the number of samples that he needed to measure something with 98% confidence; why 8?

Answer 1

You are asking two questions:

1. How do you perform a test of statistical significance that the mean time of function A is greater than the mean time of function B?
2. If you want a certain confidence in your answer, how many samples should you take?

The most common answer to the first question is that you either want to compute a confidence interval or perform a t-test. It’s not different than any other scientific experiment with random variation. To compute the 95% confidence interval of the mean response time for function A simply take the mean and add 1.96 times the standard error to either side. The standard error is the square root of the variance divided by N. That is,

95% CI = mean +/- 1.96 * sqrt(sigma2/N))

where sigma2 is the variance of speed for function A and N is the number of runs you used to calculate mean and variance.

Your second question relates to statistical power analysis and the design of experiments. You describe a sequential setup where you are asking whether to continue sampling. The design of sequential experiments is actually a very tricky problem in statistics, since in general you are not allowed to calculate confidence intervals or p-values and then draw additional samples conditional on not reaching your desired significance. If you wish to do this, it would be wiser to set up a Bayesian model and calculate your posterior probability that speed A is greater than speed B. This, however, is massive overkill.

In a computing environment it is generally pretty trivial to achieve a very small confidence interval both because drawing large N is easy and because the variance is generally small — one function obviously wins.

Given that Wikipedia and most online sources are still horrible when it comes to statistics, I recommend buying Introductory Statistics with R. You will learn both the statistics and the tools to apply what you learn.