HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS		QUICK SEARCH:   [advanced] Author: Keyword(s): Year:  Vol:  Page:

This Article Full Text Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Services Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Tino, P. Articles by Hammer, B. Search for Related Content PubMed PubMed Citation Articles by Tino, P. Articles by Hammer, B.

Architectural Bias in Recurrent Neural Networks: Fractal Analysis

Peter Tio

P.Tino{at}cs.bham.ac.uk, Aston University, Birmingham B4 7ET, U.K.

Barbara Hammer

hammer{at}informatik.uni-osnabrueck.de, University of Osnabrück, D-49069 Osnabrück, Germany

We have recently shown that when initialized with "small" weights, recurrent neural networks (RNNs) with standard sigmoid-type activation functions are inherently biased toward Markov models; even prior to any training, RNN dynamics can be readily used to extract finite memory machines (Hammer & Tio, 2002; Tio, eransk, & Beuková, 2002a, 2002b). Following Christiansen and Chater (1999), we refer to this phenomenon as the architectural bias of RNNs. In this article, we extend our work on the architectural bias in RNNs by performing a rigorous fractal analysis of recurrent activation patterns. We assume the network is driven by sequences obtained by traversing an underlying finite-state transition diagram—a scenario that has been frequently considered in the past, for example, when studying RNN-based learning and implementation of regular grammars and finite-state transducers. We obtain lower and upper bounds on various types of fractal dimensions, such as box counting and Hausdorff dimensions. It turns out that not only can the recurrent activations inside RNNs with small initial weights be explored to build Markovian predictive models, but also the activations form fractal clusters, the dimension of which can be bounded by the scaled entropy of the underlying driving source. The scaling factors are fixed and are given by the RNN parameters.

This article has been cited by other articles:

				H. Jacobsson Rule Extraction from Recurrent Neural Networks: A Taxonomy and Review Neural Comput., June 1, 2005; 17(6): 1223 - 1263. [Abstract] [Full Text] [PDF]