Software-hardware co-design for fast and scalable training of deep learning recommendation models-Reference-Cited by-同舟云学术

Software-hardware co-design for fast and scalable training of deep learning recommendation models

Published:2022-06-11 Issue: Volume: Page:
ISSN:
Container-title:Proceedings of the 49th Annual International Symposium on Computer Architecture
language:
Short-container-title:

Author:

Mudigere Dheevatsa¹,Hao Yuchen¹,Huang Jianyu¹,Jia Zhihao²,Tulloch Andrew¹,Sridharan Srinivas¹,Liu Xing¹,Ozdal Mustafa¹,Nie Jade¹,Park Jongsoo¹,Luo Liang¹,Yang Jie (Amy)¹,Gao Leon¹,Ivchenko Dmytro¹,Basant Aarti¹,Hu Yuxi¹,Yang Jiyan¹,Ardestani Ehsan K.¹,Wang Xiaodong¹,Komuravelli Rakesh¹,Chu Ching-Hsiang¹,Yilmaz Serhat¹,Li Huayu¹,Qian Jiyuan¹,Feng Zhuobo¹,Ma Yinbin¹,Yang Junjie¹,Wen Ellie¹,Li Hong¹,Yang Lin¹,Sun Chonglin¹,Zhao Whitney¹,Melts Dimitry¹,Dhulipala Krishna¹,Kishore KR¹,Graf Tyler¹,Eisenman Assaf¹,Matam Kiran Kumar¹,Gangidi Adi¹,Chen Guoqiang Jerry¹,Krishnan Manoj¹,Nayak Avinash¹,Nair Krishnakumar¹,Muthiah Bharath¹,khorashadi Mahmoud¹,Bhattacharya Pallab¹,Lapukhov Petr¹,Naumov Maxim¹,Mathews Ajit¹,Qiao Lin¹,Smelyanskiy Mikhail¹,Jia Bill¹,Rao Vijay¹

Affiliation:

1. Meta Platforms

2. Carnegie Mellon University

Publisher

ACM

Link

https://dl.acm.org/doi/pdf/10.1145/3470496.3533727

Reference71 articles.

1. [n.d.]. NVIDIA Collective Communications Library (NCCL) https://developer.nvidia.com/nccl. [n.d.]. NVIDIA Collective Communications Library (NCCL) https://developer.nvidia.com/nccl.

2. [n.d.]. OCP Open rack standard (v2) https://www.opencompute.org/wiki/Open_Rack/SpecsAndDesigns#RACK_Standards. [n.d.]. OCP Open rack standard (v2) https://www.opencompute.org/wiki/Open_Rack/SpecsAndDesigns#RACK_Standards.

3. Martín Abadi Ashish Agarwal Paul Barham Eugene Brevdo Zhifeng Chen Craig Citro Greg S. Corrado Andy Davis Jeffrey Dean Matthieu Devin Sanjay Ghemawat Ian Goodfellow Andrew Harp Geoffrey Irving Michael Isard Yangqing Jia Rafal Jozefowicz Lukasz Kaiser Manjunath Kudlur Josh Levenberg Dandelion Mané Rajat Monga Sherry Moore Derek Murray Chris Olah Mike Schuster Jonathon Shlens Benoit Steiner Ilya Sutskever Kunal Talwar Paul Tucker Vincent Vanhoucke Vijay Vasudevan Fernanda Viégas Oriol Vinyals Pete Warden Martin Wattenberg Martin Wicke Yuan Yu and Xiaoqiang Zheng. 2015. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. https://www.tensorflow.org/ Software available from tensorflow.org. Martín Abadi Ashish Agarwal Paul Barham Eugene Brevdo Zhifeng Chen Craig Citro Greg S. Corrado Andy Davis Jeffrey Dean Matthieu Devin Sanjay Ghemawat Ian Goodfellow Andrew Harp Geoffrey Irving Michael Isard Yangqing Jia Rafal Jozefowicz Lukasz Kaiser Manjunath Kudlur Josh Levenberg Dandelion Mané Rajat Monga Sherry Moore Derek Murray Chris Olah Mike Schuster Jonathon Shlens Benoit Steiner Ilya Sutskever Kunal Talwar Paul Tucker Vincent Vanhoucke Vijay Vasudevan Fernanda Viégas Oriol Vinyals Pete Warden Martin Wattenberg Martin Wicke Yuan Yu and Xiaoqiang Zheng. 2015. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. https://www.tensorflow.org/ Software available from tensorflow.org.

4. Bilge Acun Matthew Murphy Xiaodong Wang Jade Nie Carole-Jean Wu and Kim Hazelwood. 2020. Understanding Training Efficiency of Deep Learning Recommendation Models at Scale. arXiv:2011.05497 [cs.AR] Bilge Acun Matthew Murphy Xiaodong Wang Jade Nie Carole-Jean Wu and Kim Hazelwood. 2020. Understanding Training Efficiency of Deep Learning Recommendation Models at Scale. arXiv:2011.05497 [cs.AR]

5. Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell Sandhini Agarwal Ariel Herbert-Voss Gretchen Krueger Tom Henighan Rewon Child Aditya Ramesh Daniel M. Ziegler Jeffrey Wu Clemens Winter Christopher Hesse Mark Chen Eric Sigler Mateusz Litwin Scott Gray Benjamin Chess Jack Clark Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever and Dario Amodei. 2020. Language Models are Few-Shot Learners. arXiv:2005.14165 [cs.CL] Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell Sandhini Agarwal Ariel Herbert-Voss Gretchen Krueger Tom Henighan Rewon Child Aditya Ramesh Daniel M. Ziegler Jeffrey Wu Clemens Winter Christopher Hesse Mark Chen Eric Sigler Mateusz Litwin Scott Gray Benjamin Chess Jack Clark Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever and Dario Amodei. 2020. Language Models are Few-Shot Learners. arXiv:2005.14165 [cs.CL]

Cited by 56 articles. 订阅此论文施引文献订阅此论文施引文献，注册后可以免费订阅5篇论文的施引文献，订阅后可以查看论文全部施引文献

1. Rail-only: A Low-Cost High-Performance Network for Training LLMs with Trillion Parameters;2024 IEEE Symposium on High-Performance Interconnects (HOTI);2024-08-21

2. RDMA over Ethernet for Distributed Training at Meta Scale;Proceedings of the ACM SIGCOMM 2024 Conference;2024-08-04

3. Breaking Barriers: Expanding GPU Memory with Sub-Two Digit Nanosecond Latency CXL Controller;Proceedings of the 16th ACM Workshop on Hot Topics in Storage and File Systems;2024-07-08

4. MAD-Max Beyond Single-Node: Enabling Large Machine Learning Model Acceleration on Distributed Systems;2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA);2024-06-29

5. ElasticRec: A Microservice-based Model Serving Architecture Enabling Elastic Resource Scaling for Recommendation Models;2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA);2024-06-29