Author:
Rostami Amirhossein,Vogginger Bernhard,Yan Yexin,Mayr Christian G.
Abstract
IntroductionIn recent years, the application of deep learning models at the edge has gained attention. Typically, artificial neural networks (ANNs) are trained on graphics processing units (GPUs) and optimized for efficient execution on edge devices. Training ANNs directly at the edge is the next step with many applications such as the adaptation of models to specific situations like changes in environmental settings or optimization for individuals, e.g., optimization for speakers for speech processing. Also, local training can preserve privacy. Over the last few years, many algorithms have been developed to reduce memory footprint and computation.MethodsA specific challenge to train recurrent neural networks (RNNs) for processing sequential data is the need for the Back Propagation Through Time (BPTT) algorithm to store the network state of all time steps. This limitation is resolved by the biologically-inspired E-prop approach for training Spiking Recurrent Neural Networks (SRNNs). We implement the E-prop algorithm on a prototype of the SpiNNaker 2 neuromorphic system. A parallelization strategy is developed to split and train networks on the ARM cores of SpiNNaker 2 to make efficient use of both memory and compute resources. We trained an SRNN from scratch on SpiNNaker 2 in real-time on the Google Speech Command dataset for keyword spotting.ResultWe achieved an accuracy of 91.12% while requiring only 680 KB of memory for training the network with 25 K weights. Compared to other spiking neural networks with equal or better accuracy, our work is significantly more memory-efficient.DiscussionIn addition, we performed a memory and time profiling of the E-prop algorithm. This is used on the one hand to discuss whether E-prop or BPTT is better suited for training a model at the edge and on the other hand to explore architecture modifications to SpiNNaker 2 to speed up online learning. Finally, energy estimations predict that the SRNN can be trained on SpiNNaker2 with 12 times less energy than using a NVIDIA V100 GPU.
Funder
Electronic Components and Systems for European Leadership
Horizon 2020 Framework Programme
Deutsche Forschungsgemeinschaft
Bundesministerium für Bildung und Forschung
Reference51 articles.
1. “Tensorflow: a system for large-scale machine learning,”;Abadi;12th USENIX Symposium on Operating Systems Design and Implementation,2016
2. “Low precision local learning for hardware-friendly neuromorphic visual recognition,”;Acharya;ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP),2022
3. Using social and behavioural science to support COVID-19 pandemic response;Bavel;Nat. Hum. Behav,2020
4. Long short-term memory and learning-to-learn in networks of spiking neurons;Bellec;arXiv:1803.09574 [cs, q-bio,2018
5. A solution to the learning dilemma for recurrent networks of spiking neurons;Bellec;Nat. Commun
Cited by
4 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献