Neural Computation, Vol 10, 1097-1117, Copyright © 1998 by The MIT Press

LETTERS

Category Learning Through Multi-Modality Sensing

Virginia de Sa and Dana H. Ballard

Humans and other animals learn to form complex categories without receiving a target output, or teaching signal, with each input pattern. In contrast, most computer algorithms that emulate such performance assume the brain is provided with the correct output at the neuronal level or require grossly unphysiological methods of information propagation. Natural environments do not contain explicit labeling signals, but they do contain important information in the form of temporal correlations between sensations to different sensory modalities, and humans are af-fected by this correlational structure (Howells, 1944; McGurk & MacDonald, 1976; MacDonald & McGurk, 1978; Zellner & Kautz, 1990; Durgin & Proffitt, 1996). In this article we describe a simple, unsupervised neu-ral network algorithm that also uses this natural structure. Using only the co-occurring patterns of lip motion and sound signals from a human speaker, the network learns separate visual and auditory speech classifiers that perform comparably to supervised networks.

This article has been cited by other articles:

				M. W. Spratling Cortical region interactions and the functional role of apical dendrites. Behav Cogn Neurosci Rev, September 1, 2002; 1(3): 219 - 228. [Abstract] [PDF]

				Z. Li, K. F. Morris, D. M. Baekey, R. Shannon, and B. G. Lindsey Responses of Simultaneously Recorded Respiratory-Related Medullary Neurons to Stimulation of Multiple Sensory Modalities J Neurophysiol, July 1, 1999; 82(1): 176 - 187. [Abstract] [Full Text] [PDF]