Neural Computation, Vol 9, 123-141, Copyright © 1997 by The MIT Press

LETTERS

Partial BFGS Update and Efficient Step-Length Calculation for Three-Layer Neural Networks

Kazumi Saito and Ryohei Nakano

Second-order learning algorithms based on quasi-Newton methods have two problems. First, standard quasi-Newton methods are impractical for large-scale problems because they required N² storage space to maintain an approximation to an inverse Hessian matrix (N is the number of weights). Second, a line search to calculate a reasonably accurate step length is indispensable for these algorithms. In order to provide desirable performance, an efficient and reasonably accurate line search is needed. To overcome these problems, we propose a new second-order learning algorithm. Descent direction is calculated on the basis of a partial Broydon-Fleltcher-Goldfarb-Shanno (BFGS) update with 2Ns memory space (s < N), and a reasonably accurate step length is efficiently calculated as the minimal point of a second-order approximation to the objective function with respect to the step length. Our experiments, which use a parity problem and a speech synthesis problem, have shown that the proposed algorithm outperformed major learning algorithms. Moreover, it turned out that an efficient and accurate step-length calculation plays an important role for the convergence of quasi-Newton algorithms, and a partial BFGS update greatly saves storage space without losing the convergence performance.

This article has been cited by other articles:

				T. Iwata, K. Saito, N. Ueda, S. Stromsten, T. L. Griffiths, and J. B. Tenenbaum Parametric embedding for class visualization. Neural Comput., September 1, 2007; 19(9): 2536 - 2556. [Abstract] [Full Text] [PDF]

				K. Saito and R. Nakano Second-Order Learning Algorithm with Squared Penalty Term Neural Comput., March 1, 2000; 12(3): 709 - 729. [Abstract] [Full Text]