Question

I have a method that tests a value is within the range allowed on fields. If it is outside the range returns null and if inside returns the value.

internal float? ExtractMoneyInRangeAndPrecision(string fieldValue, string fieldName, float min, float max, int scale, int lineNumber)
    {          
        float returnValue;

        //Check whether valid float if
        if (float.TryParse(fieldValue, out returnValue))
        {
            //Check whether in range
            if (returnValue >= min && returnValue <= max)
            {
                int decPosition = 0;
                decPosition = fieldValue.IndexOf('.');

                if (
                    (decPosition == -1) ||
                    ((decPosition != -1) && (fieldValue.Substring(decPosition, fieldValue.Length - decPosition).Length -1 <= scale))
                    )
                {
                    return returnValue;
                }
            }
        }            

        return null;
    }

Here is my unit test:

[TestMethod()]
    [DeploymentItem("ImporterEngine.dll")]
    public void ExtractMoneyInRangeAndPrecisionTest_OutsideRange()
    {
        MockSyntaxValidator target = new MockSyntaxValidator("", 0);
        string fieldValue = "1000000";
        string fieldName = "";
        float min = 1;
        float max = 999999.99f;
        int scale = 2;
        int lineNumber = 0;
        float? Int16RangeReturned;

        Int16RangeReturned = target.ExtractMoneyInRangeAndPrecision(fieldValue, fieldName, min, max, scale, lineNumber);

        Assert.IsNull(Int16RangeReturned);
    }

As you can see the max is 999999.99 but when the method takes it in it changes it to 1,000,000

Why is this?

Answer 1

http://en.wikipedia.org/wiki/Floating_point#Accuracy_problems

In short, because of the way floating-point numbers represent real numbers, the number you assign to a float is not always the number you get back out. The value you specify is converted to the nearest value that can be represented in scientific notation with a magnitude determined by a base of 2.

In the case of 999999.99, the nearest number that can be represented as a float with the same number of sig figs is 7.6293945 * 2¹⁷ = 999999.99504, which when rounded to the same sig figs is 1,000,000.00. This may not be the EXACT case, but error like this is inherent in the use of floats.

Do not use floating-point types in situations where the accuracy of the number at a given level of precision is required. Instead, use the decimal type, which will retain the precision of values entered.