HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS		QUICK SEARCH:   [advanced] Author: Keyword(s): Year:  Vol:  Page:

This Article Full Text Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Services Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Lewicki, M. S. Articles by Sejnowski, T. J. Search for Related Content PubMed PubMed Citation Articles by Lewicki, M. S. Articles by Sejnowski, T. J.

Learning Overcomplete Representations

Computer Science Dept. and Center for the Neural Basis of Cognition, Carnegie Mellon Univ., 115 Mellon Inst., 4400 Fifth Ave., Pittsburgh, PA 15213

Terrence J. Sejnowski

Howard Hughes Medical Institute, Computational Neurobiology Laboratory, The Salk Institute, La Jolla, CA 92037, U.S.A.

In an overcomplete basis, the number of basis vectors is greater than the dimensionality of the input, and the representation of an input is not a unique combination of basis vectors. Overcomplete representations have been advocated because they have greater robustness in the presence of noise, can be sparser, and can have greater flexibility in matching structure in the data. Overcomplete codes have also been proposed as a model of some of the response properties of neurons in primary visual cortex. Previous work has focused on finding the best representation of a signal using a fixed overcomplete basis (or dictionary). We present an algorithm for learning an overcomplete basis by viewing it as probabilistic model of the observed data. We show that overcomplete bases can yield a better approximation of the underlying statistical distribution of the data and can thus lead to greater coding efficiency. This can be viewed as a generalization of the technique of independent component analysis and provides a method for Bayesian reconstruction of signals in the presence of noise and for blind source separation when there are more sources than mixtures.

This article has been cited by other articles:

				Z. He, S. Xie, L. Zhang, and A. Cichocki A note on lewicki-sejnowski gradient for learning overcomplete representations. Neural Comput., March 1, 2008; 20(3): 636 - 643. [Abstract] [Full Text] [PDF]

				Y. Xia and M. S. Kamel A cooperative recurrent neural network for solving l(1) estimation problems with general linear constraints. Neural Comput., March 1, 2008; 20(3): 844 - 872. [Abstract] [Full Text] [PDF]

				J. F. Murray and K. Kreutz-Delgado Visual recognition and inference using dynamic overcomplete sparse learning. Neural Comput., September 1, 2007; 19(9): 2301 - 2352. [Abstract] [Full Text] [PDF]

				H. Asari, B. A. Pearlmutter, and A. M. Zador Sparse representations for the cocktail party problem. J. Neurosci., July 12, 2006; 26(28): 7477 - 7490. [Abstract] [Full Text] [PDF]

				X.-L. Zhu, X.-D. Zhang, and J.-M. Ye A generalized contrast function and stability analysis for overdetermined blind separation of instantaneous mixtures. Neural Comput., March 1, 2006; 18(3): 709 - 728. [Abstract] [Full Text] [PDF]

				S. Osindero, M. Welling, and G. E. Hinton Topographic Product Models Applied to Natural Scene Statistics Neural Comput., February 1, 2005; 18(2): 381 - 414. [Abstract] [Full Text] [PDF]

				E. Smith and M. S. Lewicki Efficient Coding of Time-Relative Structure Using Spikes Neural Comput., January 1, 2005; 17(1): 19 - 45. [Abstract] [Full Text] [PDF]

				D. B. Grimes and R. P. N. Rao Bilinear Sparse Coding for Invariant Vision Neural Comput., January 1, 2005; 17(1): 47 - 73. [Abstract] [Full Text] [PDF]

				S.-i. Maeda, W.-J. Song, and S. Ishii Nonlinear and Noisy Extension of Independent Component Analysis: Theory and Its Application to a Pitch Sensation Model Neural Comput., January 1, 2005; 17(1): 115 - 144. [Abstract] [Full Text] [PDF]

				K. Zhang and L.-W. Chan An Adaptive Method for Subband Decomposition ICA Neural Comput., January 1, 2005; 18(1): 191 - 223. [Abstract] [Full Text] [PDF]

				J.-M. Ye, X.-L. Zhu, and X.-D. Zhang Adaptive Blind Separation with an Unknown Number of Sources Neural Comput., August 1, 2004; 16(8): 1641 - 1660. [Abstract] [Full Text] [PDF]

				Y. Li, A. Cichocki, and S.-i. Amari Analysis of Sparse Representation and Blind Source Separation Neural Comput., June 1, 2004; 16(6): 1193 - 1234. [Abstract] [Full Text] [PDF]

				M. S. Caywood, B. Willmore, and D. J. Tolhurst Independent Components of Color Natural Scenes Resemble V1 Neurons in Their Spatial and Color Tuning J Neurophysiol, June 1, 2004; 91(6): 2859 - 2873. [Abstract] [Full Text] [PDF]

				G. de A. Barreto, A. F. R. Araujo, and S. C. Kremer A Taxonomy for Spatiotemporal Connectionist Networks Revisited: The Unsupervised Case Neural Comput., June 1, 2003; 15(6): 1255 - 1320. [Abstract] [Full Text] [PDF]

				S. Takahashi, Y. Anzai, and Y. Sakurai Automatic Sorting for Multi-Neuronal Activity Recorded With Tetrodes in the Presence of Overlapping Spikes J Neurophysiol, April 1, 2003; 89(4): 2245 - 2258. [Abstract] [Full Text] [PDF]

				K. Kreutz-Delgado, J. F. Murray, B. D. Rao, K. Engan, T.-W. Lee, and T. J. Sejnowski Dictionary Learning Algorithms for Sparse Representation Neural Comput., February 1, 2003; 15(2): 349 - 396. [Abstract] [Full Text] [PDF]

				P. A.d.F.R. Hojen-Sorensen, O. Winther, and L. K. Hansen Mean-Field Approaches to Independent Component Analysis Neural Comput., April 1, 2002; 14(4): 889 - 918. [Abstract] [Full Text] [PDF]

				M. Girolami A Variational Method for Learning Sparse and Overcomplete Representations Neural Comput., November 1, 2001; 13(11): 2517 - 2532. [Abstract] [Full Text]

				M. Welling and M. Weber A Constrained EM Algorithm for Independent Component Analysis Neural Comput., March 1, 2001; 13(3): 677 - 689. [Abstract] [Full Text]