I was wondering, MATLAB has a removeconstantrows function that should be applied to feedforward neural network input and target output data. This function removes constant rows from the data. For example if one input vector for a 5-input neural network is [1 1 1 1 1] then it is removed.
Googling, the best explanation I could find is that (paraphrasing) “constant rows are not needed and can be replaced by appropriate adjustments to the biases of the output layer”.
Can someone elaborate?
Who does this adjustment?
From my book, the weight adjustment for simple gradient descent is:
Δ weight_i = learning_rate * local_gradient * input_i
Which means that all weights of a neuron at the first hidden layer are adjusted the same amount. But they ARE adjusted.
I think there is a misundertanding. The “row” is not an input pattern, but a feature, that is i-th component in all patterns. It’s obvious that if some feature does not have big variance on all data set, it does not provide valuable information and does not play a noticable role for network training.
The comparison to a bias is feasible (though I don’t agree, that this applies to output layer (only), bacause it depends on where the constant row is found – if it’s in input data, then it is right as well for the first hidden layer, imho). If you remeber, it’s recommended for each neuron in backpropagation network to have a special bias weight, connected to 1 constant signal. If, for example, a training set contains a row with all 1-th, then this is the same as additional bias. If the constant row has a different value, then the bias will have different effect, but in any case you can simply eliminate this row, and add the constant value of the row into the existing bias.
Disclaimer: I’m not a Matlab user. My background in neural networks comes solely from programming area.