GAL: NETWORKS THAT GROW WHEN THEY LEARN AND SHRINK WHEN THEY FORGET

Abstract

Learning when limited to modification of some parameters has a limited scope; capability to modify the system structure is also needed to get a wider range of the learnable. In the case of artificial neural networks, learning by iterative adjustment of synaptic weights can only succeed if the network designer predefines an appropriate network structure, i.e. the number of hidden layers, units, and the size and shape of their receptive and projective fields. This paper advocates the view that the network structure should not, as is usually done, be determined by trial-and-error but should be computed by the learning algorithm. Incremental learning algorithms can modify the network structure by addition and/or removal of units and/or links. A survey of current connectionist literature is given on this line of thought. “Grow and Learn” (GAL) is a new algorithm that learns an association at one shot due to its being incremental and using a local representation. During the so-called “sleep” phase, units that were previously stored but which are no longer necessary due to recent modifications are removed to minimize network complexity. The incrementally constructed network can later be finetuned off-line to improve performance. Another method proposed that greatly increases recognition accuracy is to train a number of networks and vote over their responses. The algorithm and its variants were tested on recognition of handwritten numerals and seem promising especially in terms of learning speed. This makes the algorithm attractive for on-line learning tasks, e.g. in robotics. The biological plausibility of incremental learning is also discussed briefly.