Question

I really need help implementing a continuous tanh-sigmoid activation function in a very basic neural network. If you could give a basic example that would be great, but if you could change it in my source code I would be extremely grateful! Also, what range should the random weights be initiated with (i.e. what range)?

Answer 1

The weight range depends on what input data range you have. In some implementations the weights can also be negative.

For possible Sigmoid functions, check here (tanh is not the only possibility):

http://en.wikipedia.org/wiki/Sigmoid_function

Tip: You can typically compute the NN with matrix multiplications.

http://www.dtreg.com/mlfn.htm

http://en.wikipedia.org/wiki/Neural_network

P.S.: probably not a good idea to do this in JavaScript.

---

you can either implement it via exp(x) , See: http://www.javascripter.net/faq/mathfunc.htm

          sinh(x)    exp(x) - exp(-x)     exp(2x) - 1
tanh(x) = ------- = ------------------ = -------------
          cosh(x)    exp(x) + exp(-x)     exp(2x) + 1

that gives you:

function tanh(x) {
   e = Math.exp(2*x);
   return (e - 1) / (e + 1) ;
};

another solution is to store a table with the tanh function values in an array, and define a JavaScript function which interpolates the tanh values for x based on the tanh values stored in the array

---

typically people don’t want [-inf…+inf] as the range of the input values, and don’t want [-1…+1] as the range of output values — therefore you might need a different sigmoid function!

you need to take the expected range of input values, and the expected range of output values, and use those to shift the actual sigmoid function, the weight-ranges and the value of the threshhold.

a threshhold of 0.7 or larger is typically used. You need to experiment with that.