Question

Problem with arithmetic using logarithms to avoid numerical underflow (take 2)

Having seen the above and having seen softmax normalization I was trying to normalize a vector while avoiding overflow –

that is if I have an array
x[1], x[2] x[3], x[4], ... , x[n]

the normalized form for me has the sum of squares of elements as 1.0
and is obtained by dividing each element by
sqrt(x[1]*x[1]+x[2]*x[2]+...+x[n]*x[n])

now the sum of squares can overflow even if the square root is small enough to fit into a floating point variable, so I imagined one could do something like
s=(2*log(fabs(x[1]))+2*log(fabs(x[2]))+...+2*log(fabs(x[n])))/2

and calculating the elements as

exp(log(fabs(x[1]))-s), ..., exp(log(fabs(x[n]))-s

BUT

The above is incorrect as log(A+B) is not log(A)+log(B) – now is there a way to do vector normalization that avoids overflow better?

Answer 1

You seem to be making the assumption that:

log(x^2 + y^2)

is the same as:

log(x^2) + log(y^2)

However, this isn’t correct, as you can’t simplify the logarithm of a sum like that.