Question

As I understand it, Ruby (1.9.2) floats have a precision of 15 decimal digits. Therefore, I would expect rounding float x to 15 decimal places would equal x. For this calculation this isn’t the case.

x = (0.33 * 10)
x == x.round(15) # => false

Incidentally, rounding to 16 places returns true.

Can you please explain this to me?

Answer 1

Part of the problem is that 0.33 does not have an exact representation in the underlying format, because it cannot be expressed by a series of 1 / 2ⁿ terms. So, when it is multiplied by 10 a number slightly different than 0.33 is being multiplied.

For that matter, 3.3 does not have an exact representation either.

Part One

When numbers don’t have an exact base-10 representation, there will be a remainder when converting the least significant digit for which there was information in the mantissa. This remainder will propagate to the right, possibly forever, but it’s largely meaningless. The apparent randomness of this error is due to the same reason that explains the apparently-inconsistent rounding you and Matchu noticed. That’s in part two.

Part Two

And this information (the right-most bits) is not aligned neatly with the information conveyed by a single decimal digit, so the decimal digit will typically be somewhat smaller than its value would have been if the original precision had been greater.

This is why a conversion might round to 1 at 15 digits and 0.x at 16 digits: because a longer conversion has no value for the bits to the right of the end of the mantissa.