Affiliation:
1. Department of Electrical Engineering and Computer Science University of Michigan Ann Arbor MI 48109 USA
2. Applied Materials Inc. Santa Clara CA 95054 USA
Abstract
AbstractThe constant drive to achieve higher performance in deep neural networks (DNNs) has led to the proliferation of very large models. Model training, however, requires intensive computation time and energy. Memristor‐based compute‐in‐memory (CIM) modules can perform vector‐matrix multiplication (VMM) in place and in parallel, and have shown great promises in DNN inference applications. However, CIM‐based model training faces challenges due to non‐linear weight updates, device variations, and low‐precision. In this work, a mixed‐precision training scheme is experimentally implemented to mitigate these effects using a bulk‐switching memristor‐based CIM module. Low‐precision CIM modules are used to accelerate the expensive VMM operations, with high‐precision weight updates accumulated in digital units. Memristor devices are only changed when the accumulated weight update value exceeds a pre‐defined threshold. The proposed scheme is implemented with a system‐onchip of fully integrated analog CIM modules and digital sub‐systems, showing fast convergence of LeNet training to 97.73%. The efficacy of training larger models is evaluated using realistic hardware parameters and verifies that CIM modules can enable efficient mix‐precision DNN training with accuracy comparable to full‐precision software‐trained models. Additionally, models trained on chip are inherently robust to hardware variations, allowing direct mapping to CIM inference chips without additional re‐training.
Funder
National Science Foundation
Subject
Mechanical Engineering,Mechanics of Materials,General Materials Science
Reference54 articles.
1. a)J.Devlin M.‐W.Chang K.Lee K.Toutanova Proc. 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies Ithaca New York 2019 1 4171‐4186;
2. Deep Learning for Computer Vision: A Brief Review
3. Highly accurate protein structure prediction with AlphaFold
Cited by
2 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献