NILRNN: A Neocortex-Inspired Locally Recurrent Neural Network for Unsupervised Feature Learning in Sequential Data

Abstract

AbstractUnsupervised feature learning refers to the problem of learning useful feature extraction functions from unlabeled data. Despite the great success of deep learning networks in this task in recent years, both for static and for sequential data, these systems can in general still not compete with the high performance of our brain at learning to extract useful representations from its sensory input. We propose the Neocortex-Inspired Locally Recurrent Neural Network: a new neural network for unsupervised feature learning in sequential data that brings ideas from the structure and function of the neocortex to the well-established fields of machine learning and neural networks. By mimicking connection patterns in the feedforward circuits of the neocortex, our system tries to generalize some of the ideas behind the success of convolutional neural networks to types of data other than images. To evaluate the performance of our system at extracting useful features, we have trained different classifiers using those and other learnt features as input and we have compared the obtained accuracies. Our system has shown to outperform other shallow feature learning systems in this task, both in terms of the accuracies achieved and in terms of how fast the classification task is learnt. The results obtained confirm our system as a state-of-the-art shallow feature learning system for sequential data, and suggest that extending it to or integrating it into deep architectures may lead to new successful networks that are competent at dealing with complex sequential tasks.