Question

I got a neural network that is meant to do backpropagation.

I’ve been using the following for weight update on the hidden layer

Δwji = η × δj × yi

where δj is φ'(vj) * ej where vj = Σi wjiyi, and for output layer and φ'(vj) * sum(δk*wkj) for previous layers

However, one of the bias for the hidden units always goes to infinity. What is the meaning of it?

EDIT:

The values going to Infinity turn out to be some input value vj from some neurons. So, the meaning of this is that my NN is just reinforcing the fact that an specific neuron should always fire?

Could it also mean I lack of a variety of examples for those specific inputs when training?

Answer 1

After trial and error turned out the problem was not enough neurons on the hidden layer. I started to see results after adding neurons equal to half of the inputs. Al tough there is no official formula at the ideal number of neurons. It varies for every problem