HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS		QUICK SEARCH:   [advanced] Author: Keyword(s): Year:  Vol:  Page:

This Article Full Text Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Services Similar articles in this journal Alert me to new issues of the journal Download to citation manager Google Scholar Articles by Roux, N. L. Articles by Bengio, Y. PubMed Articles by Roux, N. L. Articles by Bengio, Y.

Representational Power of Restricted Boltzmann Machines and Deep Belief Networks

lerouxni{at}iro.umontreal.ca Département Informatique et Recherche Opérationnelle, Université de Montréal, Montréal, Québec, H3C 3J7, Canada

Yoshua Bengio

bengioy{at}iro.umontreal.ca Département Informatique et Recherche Opérationnelle, Université de Montréal, Montréal, Québec, H3C 3J7, Canada

Deep belief networks (DBN) are generative neural network models with many layers of hidden explanatory factors, recently introduced by Hinton, Osindero, and Teh (2006) along with a greedy layer-wise unsupervised learning algorithm. The building block of a DBN is a probabilistic model called a restricted Boltzmann machine (RBM), used to represent one layer of the model. Restricted Boltzmann machines are interesting because inference is easy in them and because they have been successfully used as building blocks for training deeper models. We first prove that adding hidden units yields strictly improved modeling power, while a second theorem shows that RBMs are universal approximators of discrete distributions. We then study the question of whether DBNs with more layers are strictly more powerful in terms of representational power. This suggests a new and less greedy criterion for training RBMs within DBNs.