Neural Computation, Vol 9, 1093-1108, Copyright © 1997 by The MIT Press

LETTERS

Noise Injection: Theoretical Prospects

Yves Grandvalet, Stéphane Canu and Stéphane Boucheron

Noise injection consists of adding noise to the inputs during neural network training. Experimental results suggest that it might improve the generalization ability of the resulting neural network. A justification of this improvement remains elusive: describing analytically the average perturbed cost function is difficult, and controlling the fluctuations of the random perturbed cost function is hard. Hence, recent papers suggest replacing the random perturbed cost by a (deterministic) Taylor approximation of the average perturbed cost function. This article takes a different stance: when the injected noise is gaussian, noise injection is naturally connected to the action of the heat kernel. This provides indications on the relevance domain of traditional Taylor expansions and shows the dependence of the quality of Taylor approximations on global smoothness properties of neural networks under consideration. The connection between noise injection and heat kernel also enables controlling the fluctuations of the random perturbed cost function. Under the global smoothness assumption, tools from gaussian analysis provide bounds on the tail behavior of the perturbed cost. This finally suggests that mixing input perturbation with smoothness-based penalization might be profitable.

This article has been cited by other articles:

				C. P. Unsworth and G. Coghill Excessive noise injection training of neural networks for markerless tracking in obscured and segmented environments. Neural Comput., September 1, 2006; 18(9): 2122 - 2145. [Abstract] [Full Text] [PDF]