Question

I am creating a program that needs to find thousands of individual percentiles, most of which are less than .00005. Currently, to do this I use

0.5 * erfc(-zScore * M_SQRT1_2)

However, this seems to be rounding slightly (throughout the rest of my program I am using doubles and long doubles, so it has to be this). I believe this because at the end, when I add up all the percentiles, I get 1.835468. This tells me that it is rounding as it should add up 1, or at least a number very close to 1. In addition, when I log each individual percentile, I get the same number (let’s say 0.000036) for a few percentiles, and then it goes down to 0.000035. It should be going down each time as each number is further from the mean than the last.

I need a way to find very precise percentiles based on Z-Scores, which this is not giving me as it is rounding too early, at the 6th decimal place.

Answer 1

When you see it jump from 0.000036 to 0.000035, this is because you need to use NSLog(@"Value is: %0.36f", yourPercentile);. You should find that it is not actually rounding at 6 digits, but that was just how it was logged.

Now, your error is coming from the fact that you are using doubles, which do not store precise values very well. First, you need to know how much precision that you need, and then use a type which can handle that level of precision.

Let’s say that you decide that you require 12 digits of precision.

long long is then a good unit to use because it can store 19 digits. When you calculate your original value, you need to multiply the original values by 100,000,000,000 and store them as long long’s. Then do the math that you need using the large values. Eventually, when you get your result, just divide by the same value (or 2 digits less less if you want to see it as a whole number) to get your percentage.