Home/ Questions/Q 863317
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-15T09:17:23+00:00

here’s a function I wrote to print every digit of a float number in

  • 0

here’s a function I wrote to print every digit of a float number in F#:

let rec TestFloor (fnum:float) =
    let floor = System.Math.Floor(fnum)
    printfn "fnum:%f floor:%f" fnum floor
    if floor > 0.0 then TestFloor((fnum - floor) * 10.0)

Anyway the result is strange, for example:

> TestFloor 1.23;;
fnum:1.230000 floor:1.000000
fnum:2.300000 floor:2.000000
**fnum:3.000000 floor:2.000000**
fnum:10.000000 floor:9.000000
fnum:10.000000 floor:9.000000
fnum:10.000000 floor:9.000000
fnum:10.000000 floor:9.000000
fnum:10.000000 floor:9.000000
fnum:10.000000 floor:9.000000
fnum:10.000000 floor:9.000000
fnum:10.000000 floor:9.000000
fnum:9.999998 floor:9.000000
fnum:9.999982 floor:9.000000
fnum:9.999822 floor:9.000000
...

At the fourth line the floor of 3.0 turned out to be 2.0 which is strange. The following computations are all going wrong. I’m wondering what’s happening there?

Thanks!

Edit 2

@ sepp2k

This is the result after using %.30f:

> TestFloor 1.23;;
fnum:1.230000000000000000000000000000 floor:1.000000000000000000000000000000
fnum:2.300000000000000000000000000000 floor:2.000000000000000000000000000000
**fnum:3.000000000000000000000000000000 floor:2.000000000000000000000000000000**
fnum:9.999999999999980000000000000000 floor:9.000000000000000000000000000000
fnum:9.999999999999820000000000000000 floor:9.000000000000000000000000000000
fnum:9.999999999998220000000000000000 floor:9.000000000000000000000000000000
fnum:9.999999999982240000000000000000 floor:9.000000000000000000000000000000
fnum:9.999999999822360000000000000000 floor:9.000000000000000000000000000000

As you can see at the fourth line fnum is 3.00... and the floor value is 2.00..

Edit 3 – Solved

Thank you all, now I understand what the problem is about.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    This is just one of those rounding issues you get with floating point arithmetic. You will find its representation is 2.99999… probably infinitely recurring. The call to printfn clearly rounds this up to the expected representation, but floor still sees this as 2.99999…

    This is exactly the sort of problem that the decimal type exists to solve, so if we rewrite to use decimal, we obtain the correct result:

    let rec TestFloor dnum =
    let fl = floor dnum
    printfn "fnum:%f floor:%f" dnum fl
    if fl > 0.0M then TestFloor((dnum - fl) * 10.0M)

    This gives:

    > TestFloor 1.23M;;
fnum:1.230000 floor:1.000000
fnum:2.300000 floor:2.000000
fnum:3.000000 floor:3.000000
fnum:0.000000 floor:0.000000
val it : unit = ()

    Of course, you can stick with float, but add a very small tolerance value to ensure that these corner cases are always just a little above the expected value, rather than just a little below, eg:

    let rec TestFloor fnum =
    let fl = floor (fnum + 0.00000000001)
    printfn "fnum:%f floor:%f" fnum fl
    if fl > 0.0 then TestFloor((fnum - fl) * 10.0)

    Which gives the same result as above.

    • 0