Mixed-precision quantization-aware training for photonic neural networks
-
Published:2023-08-05
Issue:29
Volume:35
Page:21361-21379
-
ISSN:0941-0643
-
Container-title:Neural Computing and Applications
-
language:en
-
Short-container-title:Neural Comput & Applic
Author:
Kirtas ManosORCID, Passalis Nikolaos, Oikonomou Athina, Moralis-Pegios Miltos, Giamougiannis George, Tsakyridis Apostolos, Mourgias-Alexandris George, Pleros Nikolaos, Tefas Anastasios
Abstract
AbstractThe energy demanding nature of deep learning (DL) has fueled the immense attention for neuromorphic architectures due to their ability to operate in a very high frequencies in a very low energy consumption. To this end, neuromorphic photonics are among the most promising research directions, since they are able to achieve femtojoule per MAC efficiency. Although electrooptical substances provide a fast and efficient platform for DL, they also introduce various noise sources that impact the effective bit resolution, introducing new challenges to DL quantization. In this work, we propose a quantization-aware training method that gradually performs bit reduction to layers in a mixed-precision manner, enabling us to operate lower-precision networks during deployment and further increase the computational rate of the developed accelerators while keeping the energy consumption low. Exploiting the observation that intermediate layers have lower-precision requirements, we propose to gradually reduce layers’ bit resolutions, by normally distributing the reduction probability of each layer. We experimentally demonstrate the advantages of mixed-precision quantization in both performance and inference time. Furthermore, we experimentally evaluate the proposed method in different tasks, architectures, and photonic configurations, highlighting its immense capabilities to reduce the average bit resolution of DL models while significantly outperforming the evaluated baselines.
Funder
H2020 Industrial Leadership Aristotle University of Thessaloniki
Publisher
Springer Science and Business Media LLC
Subject
Artificial Intelligence,Software
Reference64 articles.
1. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436–444 2. Brown TB, Mann B, Ryder N, Subbiah M, Kaplan J, Dhariwal P, Neelakantan A, Shyam P, Sastry G, Askell A, Agarwal S, Herbert-Voss A, Krueger G, Henighan T, Child R, Ramesh A, Ziegler DM, Wu J, Winter C, Hesse C, Chen M, Sigler E, Litwin M, Gray S, Chess B, Clark J, Berner C, McCandlish S, Radford A, Sutskever I, Amodei D (2020) Language models are few-shot learners. https://doi.org/10.48550/ARXIV.2005.14165. arXiv:org/abs/2005.14165 3. Ramesh A, Pavlov M, Goh G, Gray S, Voss C, Radford A, Chen M, Sutskever I (2021) Zero-shot text-to-image generation. https://doi.org/10.48550/ARXIV.2102.12092. arXiv:org/abs/2102.12092 4. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser L, Polosukhin I (2017) Attention is all you need. https://doi.org/10.48550/ARXIV.1706.03762. arXiv:org/abs/1706.03762 5. Silver D, Hubert T, Schrittwieser J, Antonoglou I, Lai M, Guez A, Lanctot M, Sifre L, Kumaran D, Graepel T, Lillicrap T, Simonyan K, Hassabis D (2017) Mastering chess and shogi by self-play with a general reinforcement learning algorithm. https://doi.org/10.48550/ARXIV.1712.01815. arXiv:org/abs/1712.01815
Cited by
1 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献
|
|