SMaLL: Software for rapidly instantiating Machine Learning Libraries-Reference-Cited by-同舟云学术

SMaLL: Software for rapidly instantiating Machine Learning Libraries

Published:2023-07-12 Issue: Volume: Page:
ISSN:1539-9087
Container-title:ACM Transactions on Embedded Computing Systems
language:en
Short-container-title:ACM Trans. Embed. Comput. Syst.

Author:

Sridhar Upasana¹,Tukanov Nicholai¹,Binder Elliott¹,Low Tze Meng¹,McMillan Scott²,Schatz Martin D.³

Affiliation:

1. ECE, Carnegie Mellon University, USA

2. Software Engineering Institute, Carnegie Mellon University, USA

3. Meta, USA

Abstract

Interest in deploying deep neural network (DNN) inference on edge devices has resulted in an explosion of the number and types of hardware platforms that machine learning (ML) libraries must support. High-level programming interfaces, such as TensorFlow, can be readily ported across different devices; however, maintaining performance when porting the low-level implementation is more nuanced. High-performance inference implementations require an effective mapping of the high-level interface to the target hardware platform. Commonly, this mapping may use optimizing compilers to generate code at compile time or high-performance vendor libraries that have been specialized to the target platform. Both approaches rely on expert knowledge across levels to produce an efficient mapping. This makes supporting new architectures difficult and time-consuming. In this work, we present a DNN library framework, SMaLL, that is easily extensible to new architectures. The framework uses a unified loop structure and shared, cache-friendly data format across all intermediate layers, eliminating the time and memory overheads incurred by data transformation between layers. Each layer is implemented by specifying its dimensions and a kernel , the key computing operation of that layer. The unified loop structure and kernel abstraction allows the reuse of code across layers and computing platforms. New architectures only require a few hundred of lines in the kernel to be redesigned. To show the benefits of our approach, we have developed software that supports a range of layer types and computing platforms; this software is easily extensible for rapidly instantiating high-performance DNN libraries. An evaluation of the portability of our framework is shown by instantiating end-to-end networks from the MLPerf:tiny benchmark suite on five ARM platforms and one x86 platform (an AMD Zen 2). We also show that the end-to-end performance is comparable to or better than ML frameworks such as TensorFlow, TVM, and LibTorch.

Publisher

Association for Computing Machinery (ACM)

Subject

Hardware and Architecture,Software

Link

https://dl.acm.org/doi/pdf/10.1145/3607870

Reference45 articles.

1. José E. Moreira Kit Barton Steven Battle Peter Bergner Ramon Bertran Puneeth Bhat Pedro Caldeira David Edelsohn Gordon C. Fossum Brad Frey Nemanja Ivanovic Chip Kerchner Vincent Lim Shakti Kapoor Tulio Machado Filho Silvia Melitta Mueller Brett Olsson Satish Sadasivam Baptiste Saleil Bill Schmidt Rajalakshmi Srinivasaraghavan Shricharan Srivatsan Brian W. Thompto Andreas Wagner and Nelson Wu. 2021. A matrix math facility for Power ISA(TM) processors. CoRR abs/2104.03142(2021). arXiv:2104.03142 https://arxiv.org/abs/2104.03142 José E. Moreira Kit Barton Steven Battle Peter Bergner Ramon Bertran Puneeth Bhat Pedro Caldeira David Edelsohn Gordon C. Fossum Brad Frey Nemanja Ivanovic Chip Kerchner Vincent Lim Shakti Kapoor Tulio Machado Filho Silvia Melitta Mueller Brett Olsson Satish Sadasivam Baptiste Saleil Bill Schmidt Rajalakshmi Srinivasaraghavan Shricharan Srivatsan Brian W. Thompto Andreas Wagner and Nelson Wu. 2021. A matrix math facility for Power ISA(TM) processors. CoRR abs/2104.03142(2021). arXiv:2104.03142 https://arxiv.org/abs/2104.03142

2. Nicholai Tukanov , Rajalakshmi Srinivasaraghavan , José E Moreira , and Tze Meng Low . 2022 . Modeling Matrix Engines for Portability and Performance. In 2022 IEEE International Parallel and Distributed Processing Symposium (IPDPS). IEEE, 1173–1183 . Nicholai Tukanov, Rajalakshmi Srinivasaraghavan, José E Moreira, and Tze Meng Low. 2022. Modeling Matrix Engines for Portability and Performance. In 2022 IEEE International Parallel and Distributed Processing Symposium (IPDPS). IEEE, 1173–1183.

3. Nvidia Corporation . NVIDIA A100 Tensor Core GPU Architecture . Nvidia Corporation . https://images.nvidia.com/aem-dam/en-zz/Solutions/data-center/nvidia-ampere-architecture-whitepaper.pdf Nvidia Corporation. NVIDIA A100 Tensor Core GPU Architecture. Nvidia Corporation. https://images.nvidia.com/aem-dam/en-zz/Solutions/data-center/nvidia-ampere-architecture-whitepaper.pdf

4. Albert Reuther Peter Michaleas Michael Jones Vijay Gadepally Siddharth Samsi and Jeremy Kepner. 2020. Survey of Machine Learning Accelerators. CoRR abs/2009.00993(2020). arXiv:2009.00993 https://arxiv.org/abs/2009.00993 Albert Reuther Peter Michaleas Michael Jones Vijay Gadepally Siddharth Samsi and Jeremy Kepner. 2020. Survey of Machine Learning Accelerators. CoRR abs/2009.00993(2020). arXiv:2009.00993 https://arxiv.org/abs/2009.00993

5. Martín Abadi Ashish Agarwal Paul Barham Eugene Brevdo Zhifeng Chen Craig Citro Greg S. Corrado Andy Davis Jeffrey Dean Matthieu Devin Sanjay Ghemawat Ian Goodfellow Andrew Harp Geoffrey Irving Michael Isard Yangqing Jia Rafal Jozefowicz Lukasz Kaiser Manjunath Kudlur Josh Levenberg Dandelion Mané Rajat Monga Sherry Moore Derek Murray Chris Olah Mike Schuster Jonathon Shlens Benoit Steiner Ilya Sutskever Kunal Talwar Paul Tucker Vincent Vanhoucke Vijay Vasudevan Fernanda Viégas Oriol Vinyals Pete Warden Martin Wattenberg Martin Wicke Yuan Yu and Xiaoqiang Zheng. 2015. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. (2015). https://www.tensorflow.org/ Software available from tensorflow.org. Martín Abadi Ashish Agarwal Paul Barham Eugene Brevdo Zhifeng Chen Craig Citro Greg S. Corrado Andy Davis Jeffrey Dean Matthieu Devin Sanjay Ghemawat Ian Goodfellow Andrew Harp Geoffrey Irving Michael Isard Yangqing Jia Rafal Jozefowicz Lukasz Kaiser Manjunath Kudlur Josh Levenberg Dandelion Mané Rajat Monga Sherry Moore Derek Murray Chris Olah Mike Schuster Jonathon Shlens Benoit Steiner Ilya Sutskever Kunal Talwar Paul Tucker Vincent Vanhoucke Vijay Vasudevan Fernanda Viégas Oriol Vinyals Pete Warden Martin Wattenberg Martin Wicke Yuan Yu and Xiaoqiang Zheng. 2015. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. (2015). https://www.tensorflow.org/ Software available from tensorflow.org.