Home/ Questions/Q 3356778
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-18T02:33:14+00:00

Recently I’ve studied the backpropagation network and have done some manual exercise. After that,

  • 0

Recently I’ve studied the backpropagation network and have done some manual exercise.
After that, I came up with a question( maybe doesn’t make sense): is there any thing important in following two different replacement methods:
1. Incremental Training: weights are immediately updated once all the delta Wij’s are known and before presenting the next training vector.
2. Batch Training: delta Wij’s are computed and stored for each exemplar training vector. However, the delta Wij’s are not immediately used to update the weights. Weight updating is done at the end of a training epoch.

I’ve googled for a while but haven’t found any results.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    So what you are referring to is the two modes to perform gradient descent learning. In batch mode, changes to the weight matrix are accumulated over an entire presentation of the training data set (one ‘epoch’); online training updates the weight after presentation of each vector comprising the training set.

    I believe the consensus is that online training is superior because it converges much faster (most studies report no apparent differences in accuracy). (See e.g., Randall Wilson & Tony Martinez, The General Inefficiency of Batch Training for Gradient Descent Learning, In Neural Networks (2003).

    The reason why online training converges faster is that it can follow curves in the error surface over each epoch. The practical significance of this is that you can use a larger learning rate (and therefore converge with fewer cycles through the training data).

    Put another way, the accumulated weight change for batch training increases with the size of the training set. The result is that batch training uses large steps at each iteration, and therefore misses local minima in the error space topology–your solver oscillates rather than converges.

    Batch training is usually the ‘default’ (most often used in ML textbooks, etc.) and there’s nothing wrong with using it as long as it converges within your acceptable time limits. Again, the difference in performance (resolution, or classification accuracy) is small or negligible.

    • 0