Question

I have some problems due to really low numbers used with numpy. It took me several weeks to trace back my constant problems with numerical integration to the fact, that when I add up floats in a function the float64 precision gets lost. Performing the mathematically identic calculation with a product instead of a sum leads to values that are alright.

Here is a code sample and a plot of the results:

from matplotlib.pyplot import *
from numpy import vectorize, arange
import math

def func_product(x):
    return math.exp(-x)/(1+math.exp(x))

def func_sum(x):
    return math.exp(-x)-1/(1+math.exp(x))

#mathematically, both functions are the same

vecfunc_sum = vectorize(func_sum)
vecfunc_product = vectorize(func_product)

x = arange(0.,300.,1.)
y_sum = vecfunc_sum(x)
y_product = vecfunc_product(x)

plot(x,y_sum,    'k.-', label='sum')
plot(x,y_product,'r--',label='product')

yscale('symlog', linthreshy=1E-256)
legend(loc='lower right')
show()

As you can see, the summed values that are quite low are scattered around zero or are exactly zero while the multiplicated values are fine…

Please, could someone help/explain? Thanks a lot!

Answer 1

Floating point precision is pretty sensitive to addition/subtraction due to roundoff error. Eventually, 1+exp(x) gets so big that adding 1 to exp(x) gives the same thing as exp(x). In double precision that’s somewhere around exp(x) == 1e16:

>>> (1e16 + 1) == (1e16)
True
>>> (1e15 + 1) == (1e15)
False

Note that math.log(1e16) is approximately 37 — Which is roughly where things go crazy on your plot.

You can have the same problem, but on different scales:

>>> (1e-16 + 1.) == (1.)
True
>>> (1e-15 + 1.) == (1.)
False

For a vast majority of the points in your regime, your func_product is actually calculating:

exp(-x)/exp(x) == exp(-2*x)

Which is why your graph has a nice slope of -2.

Taking it to the other extreme, you’re other version is calculating (at least approximately):

exp(-x) - 1./exp(x) 

which is approximately

exp(-x) - exp(-x)