Question

I’m using this article to implement a neural network with backpropagation, but having trouble calculating errors. In a nutshell, my sigmoid function is squashing all my node outputs to 1.0, which then causes the error calculation to return 0:

error = (expected - actual) * (1 - actual) * actual
                                    ^^ this term causes multiply by 0

And so my error is always 0.

I suspect that the problem lies with my sigmoid implementation, which is returning 1.0, rather than asymptotically bounding below 1.0:

# ruby
def sigmoid(x)
  1/(1+Math.exp(-x))
end

Am I correct that sigmoid should never actually reach 1.0, or have I got something else wrong?

Answer 1

In a mathematical context you are correct that sigmoid should never reach 1.0. However in a practical programming context Math.exp(-x) will eventually get so small that the difference between it and 0 is negligible and you will get the 1.0 result. Depending on the range of x, this would not be surprising results.

In order to use the sigmoid approach you should make the sum of the incoming weights at each node approximately one. This will make the output of the sigmoid reasonable and allow your weights to converge quicker.