HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS		QUICK SEARCH:   [advanced] Author: Keyword(s): Year:  Vol:  Page:

This Article Full Text Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Services Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Bengio, Y. Articles by Ouimet, M. Search for Related Content PubMed PubMed Citation Articles by Bengio, Y. Articles by Ouimet, M.

Learning Eigenfunctions Links Spectral Embedding and Kernel PCA

bengioy{at}iro.umontreal.ca, Département d'Informatique et Recherche Opérationnelle, Centre de Recherches Mathématiques, Université de Montréal, Montréal, Québec, Canada, H3C 3J7

Olivier Delalleau

delallea{at}iro.umontreal.ca, Département d'Informatique et Recherche Opérationnelle, Centre de Recherches Mathématiques, Université de Montréal, Montréal, Québec, Canada, H3C 3J7

Nicolas Le Roux

lerouxni{at}iro.umontreal.ca, Département d'Informatique et Recherche Opérationnelle, Centre de Recherches Mathématiques, Université de Montréal, Montréal, Québec, Canada, H3C 3J7

Jean-François Paiement

paiemeje{at}iro.umontreal.ca, Département d'Informatique et Recherche Opérationnelle, Centre de Recherches Mathématiques, Université de Montréal, Montréal, Québec, Canada, H3C 3J7

Pascal Vincent

vincentp{at}iro.umontreal.ca, Département d'Informatique et Recherche Opérationnelle, Centre de Recherches Mathématiques, Université de Montréal, Montréal, Québec, Canada, H3C 3J7

Marie Ouimet

ouimema{at}iro.umontreal.ca, Département d'Informatique et Recherche Opérationnelle, Centre de Recherches Mathématiques, Université de Montréal, Montréal, Québec, Canada, H3C 3J7

In this letter, we show a direct relation between spectral embedding methods and kernel principal components analysis and how both are special cases of a more general learning problem: learning the principal eigenfunctions of an operator defined from a kernel and the unknown data-generating density. Whereas spectral embedding methods provided only coordinates for the training points, the analysis justifies a simple extension to out-of-sample examples (the Nyström formula) for multidimensional scaling (MDS), spectral clustering, Laplacian eigenmaps, locally linear embedding (LLE), and Isomap. The analysis provides, for all such spectral embedding methods, the definition of a loss function, whose empirical average is minimized by the traditional algorithms. The asymptotic expected value of that loss defines a generalization performance and clarifies what these algorithms are trying to learn. Experiments with LLE, Isomap, spectral clustering, and MDS show that this out-of-sample embedding formula generalizes well, with a level of error comparable to the effect of small perturbations of the training set on the embedding.

This article has been cited by other articles:

				Y. Bengio, M. Monperrus, and H. Larochelle Nonlocal Estimation of Manifold Structure. Neural Comput., October 1, 2006; 18(10): 2509 - 2528. [Abstract] [Full Text] [PDF]