Question

In testing as to why my program is not working as intended, I tried typing the calculations that seem to be failing into the immediate window.

Math.Floor(1.0f)
1.0 - correct

However:

200f * 0.005f
1.0

Math.Floor(200f * 0.005f)
0.0 - incorrect

Furthermore:

(float)(200f * 0.005f)
1.0

Math.Floor((float)(200f * 0.005f))
0.0 - incorrect

Probably some float loss is occuring, 0.99963 ≠ 1.00127 for example.

I wouldn’t mind storing less pricise values, but in a non lossy way, for example if there were a numeric type that stored values as integers do, but to only three decimal places, if it could be made performant.

I think probably there is a better way of calculating (n * 0.005f) in regards to such errors.

edit:

TY, a solution:

Math.Floor(200m * 0.005m)

Also, as I understand it, this would work if I didn’t mind changing the 1/200 into 1/256:

Math.Floor(200f * 0.00390625f)

The solution I’m using. It’s the closest I can get in my program and seems to work ok:

float x = ...;
UInt16 n = 200;
decimal d = 1m / n;
... = Math.Floor((decimal)x * d)

Answer 1

Floats represent numbers as fractions with powers of two in the denominator. That is, you can exactly represent 1/2, or 3/4, or 19/256. Since .005 is 1/200, and 200 is not a power of two, instead what you get for 0.005f is the closest fraction that has a power of two on the bottom that can fit into a 32 bit float.

Decimals represent numbers as fractions with powers of ten in the denominator. Like floats, they introduce errors when you try to represent numbers that do not fit that pattern. 1m/333m for example, will give you the closest number to 1/333 that has a power of ten as the denominator and 29 or fewer significant digits. Since 0.005 is 5/1000, and that is a power of ten, 0.005m will give you an exact representation. The price you pay is that decimals are much larger and slower than floats.

You should always always always use decimals for financial calculations, never floats.