Question

I’ve implemented a simple Bayesian classifier, but I’m running into some overflow problems when using it on non-trivial amounts of data.

One strategy I tried in order to keep the numbers small, but still exact, was to keep reducing the numerator and denominator with the greatest common divisor for every part of the equation. This, however, only works when they have a common divisor…

Note, the problem goes both ways, when I keep the denominators and numerators separate for most of the calculation I struggle with integer overflow, when I do most calculations on the fly, using double arithmetic, I’m met with the various problems/limits that really small double values have (as defined by IEEE 754).

As I’m sure some of you here have implemented this algorithm before, how did you deal with these issues? I’d prefer not to pull in arbitrary precision types as they cost too much and I’m sure there exists a solution which doesn’t require them.

Thanks.

Answer 1

Usually the way you handle this is by taking logs and using adds, and then doing an exp if you want to get back into probability space.

p1 * p2 * p3 * … * pn = exp(log(p1) + log(p2) + log(p3) + … log(pn))

You avoid under flows by working in log space.