Sign Up to our social questions and Answers Engine to ask questions, answer people’s questions, and connect with other people.
Login to our social questions & Answers Engine to ask questions answer people’s questions & connect with other people.
Lost your password? Please enter your email address. You will receive a link and will create a new password via email.
Please briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
Actually these are 3 questions:
Which optimization algorithm should I use to optimize the weights of a multilayer perceptron, if I knew…
1) only the value of the error function? (blackbox)
2) the gradient? (first derivative)
3) the gradient and the hessian? (second derivative)
I heard CMA-ES should work very well for 1) and BFGS for 2) but I would like to know if there are any alternatives and I don’t know wich algorithm to take for 3).
I solved this problem finally: there are some efficient algorithms for optimizing neural networks in reinforcement learning (with fixed topology), e. g. CMA-ES (CMA-NeuroES) or CoSyNE.
The best optimization algorithm for supervised learning seems to be Levenberg-Marquardt (LMA). This is an algorithm that is specifically designed for least square problems. When there are many connections and weights, LMA does not work very well because the required space is huge. In this case I am using Conjugate Gradient (CG).
The hessian matrix does not accelerate optimization. Algorithms that approximate the 2nd derivative are faster and more efficient (BFGS, CG, LMA).
edit: For large scale learning problems often Stochastic Gradient Descent (SGD) outperforms all other algorithms.