Question

For some reason, the following statement evaluates to zero. I assume it’s due to overflow, but all the interim values seem to be well within the limits of a double.

DiffieHellmanKey = (43 ^ 47) - (53 * Fix((43 ^ 47) / 53))

I think it’s overflow because when I execute it with different numbers (below), it results in the correct value of 29.

DiffieHellmanKey = (5 ^ 22) - (53 * Fix((5 ^ 22) / 53))

What gives? Now, back to the original numbers that are giving me overflow. All variables involved are Doubles. It doesn’t even work if I calculate it as a Worksheet formula instead of in VBA:

=(43 ^ 47) - (53 * ROUNDDOWN(((43 ^ 47) / 53), 0))

And if I implement the above example in VBA using the equivalent form (below), I get an incorrect result of -1.75357E+62.

DiffieHellmanKey = (43 ^ 47) - (53 * WorksheetFunction.RoundDown(((43 ^ 47) / 53), 0))

Answer 1

You are kind of right but your problem is not with overflow, but with significant digits.

Numbers in Microsoft Excel can never have more than 15 significant digits, but decimals can be as large as 127.

source: http://msdn.microsoft.com/en-us/library/office/ff838588.aspx

This is what was happening with your original formula:

(43 ^ 47) - (53 * Fix((43 ^ 47) / 53)) simplifies to (43 ^ 47) - fix(43 ^ 47)
Then the (43^47) has about 76 digits long so they are both getting cut down to the same amount and equating to 0.

The largest variable type in VBA is ‘Decimal’ Which only holds 29 digits of significance. You won’t be able to perform math this large using visual basic natively.