DescriptionSizeFormat
Author final version397 KBAdobe Acrobat PDFView/Open

Please use this identifier to cite or link to this item: http://hdl.handle.net/1959.13/27498

Title
Incremental training of first order recurrent neural networks to predict a context-sensitive language
Author/Creator
Chalup, Stephan K.;  Blair, Alan D.
Description
In recent years it has been shown that first order recurrent neural networks trained by gradient-descent can learn not only regular but also simple context-free and context-sensitive languages. However, the success rate was generally low and severe instability issues were encountered. The present study examines the hypothesis that a combination of evolutionary hill climbing with incremental learning and a well-balanced training set enables first order recurrent networks to reliably learn context-free and mildly context-sensitive languages. In particular, we trained the networks to predict symbols in string sequences of the context-sensitive language {aⁿbⁿcⁿ; n≥1}. Comparative experiments with and without incremental learning indicated that incremental learning can accelerate and facilitate training. Furthermore, incrementally trained networks generally resulted in monotonic trajectories in hidden unit activation space, while the trajectories of non-incrementally trained networks were oscillating. The non-incrementally trained networks were more likely to generalise.
Relation
Neural Networks Vol. 16, Issue 7, p. 955-972
Publisher Link
http://dx.doi.org/10.1016/S0893-6080(03)00054-6
Date
2003
Publisher
Pergamon / Elsevier Ltd.
Keyword(s)
mildly context-sensitive language;  incremental learning;  simple recurrent neural network;  evolutionary algorithm;  hidden unit dynamics
Resource Type
journal article
Identifier
http://hdl.handle.net/1959.13/27498
Identifier
ISSN:0893-6080
Reviewed
Reviewed
 
Image Thumbnail
30 Visitors 52 Hits 9 Downloads
Save/E-mail Citation
Citation Format
E-mail Address
Subject
"Neural Networks"
 
OR