Question

I’m writing a ruby program that uses floats. I’m having trouble with the precision. For example

1.9.3p194 :013 > 113.0 * 0.01
# => 1.1300000000000001

and therefore

1.9.3p194 :018 > 113 * 0.01 == 1.13
# => false

This is exactly the sort of calculation my app needs to get right.

Is this expected? How should I go about handling this?

Answer 1

This is an inherent limitation in floating point numbers (even 0.01 doesn’t have an exact binary floating point representation). You can use the technique provided by Aleksey or, if you want perfect precision, use the BigDecimal class bundled in ruby. It’s more verbose, and slower, but it will give the right results:

require 'bigdecimal'
=> true
1.9.3p194 :003 > BigDecimal.new("113") * BigDecimal("0.01")
=> #<BigDecimal:26cefd8,'0.113E1',18(36)> 
1.9.3p194 :004 > BigDecimal.new("113") * BigDecimal("0.01") == BigDecimal("1.13")
=> true