Neural Computation, Vol 10, 2201-2217, Copyright © 1998 by The MIT Press
Online Learning from Finite Training Sets and Robustness to Input Bias
Peter Sollich and David Barber
We analyze online gradient descent learning from finite training sets at
noninfinitesimal learning rates 
. Exact results are obtained for
the time-dependent generalization error of a simple model system: a linear
network with a large number of weights N, trained on
p=
N examples. This allows us to study in
detail the effects of finite training set size on, for example,
the optimal choice of learning rate . We also compare online and
offline learning, for respective optimal settings of at given
final learning time. Online learning turns out to be much more robust to
input bias and actually outperforms offline learning when such bias is
present; for unbiased inputs, online and offline learning perform almost
equally well.
