Overlap Communication with Dependent Computation via Decomposition in Large Deep Learning Models-Reference-Cited by-同舟云学术

Overlap Communication with Dependent Computation via Decomposition in Large Deep Learning Models

Published:2022-12-19 Issue: Volume: Page:
ISSN:
Container-title:Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1
language:
Short-container-title:

Author:

Wang Shibo¹,Wei Jinliang¹,Sabne Amit¹,Davis Andy¹,Ilbeyi Berkin¹,Hechtman Blake¹,Chen Dehao²,Murthy Karthik Srinivasa¹,Maggioni Marcello¹,Zhang Qiao¹,Kumar Sameer¹,Guo Tongfei¹,Xu Yuanzhong¹,Zhou Zongwei¹

Affiliation:

1. Google, USA

2. Waymo, USA

Publisher

ACM

Link

https://dl.acm.org/doi/pdf/10.1145/3567955.3567959

Reference55 articles.

1. 2020. Google breaks AI performance records in MLPerf with world’s fastest training supercomputer. https://cloud.google.com/blog/products/ai-machine-learning/google-breaks-ai-performance-records-in-mlperf-with-worlds-fastest-training-supercomputer 2020. Google breaks AI performance records in MLPerf with world’s fastest training supercomputer. https://cloud.google.com/blog/products/ai-machine-learning/google-breaks-ai-performance-records-in-mlperf-with-worlds-fastest-training-supercomputer

2. 2021. MLPerf Training v1.1. https://mlcommons.org/en/training-normal-11/ 2021. MLPerf Training v1.1. https://mlcommons.org/en/training-normal-11/

3. 2021. XLA: Optimizing Compiler for TensorFlow. https://www.tensorflow.org/xla 2021. XLA: Optimizing Compiler for TensorFlow. https://www.tensorflow.org/xla

4. 2022. NVIDIA H100 Tensor Core GPU Architecture. https://www.hpctech.co.jp/catalog/gtc22-whitepaper-hopper_v1.01.pdf 2022. NVIDIA H100 Tensor Core GPU Architecture. https://www.hpctech.co.jp/catalog/gtc22-whitepaper-hopper_v1.01.pdf

5. 2022. XLA DynamicSlice Semantics. https://www.tensorflow.org/xla/operation_semantics##dynamicslice 2022. XLA DynamicSlice Semantics. https://www.tensorflow.org/xla/operation_semantics##dynamicslice

Cited by 9 articles. 订阅此论文施引文献订阅此论文施引文献，注册后可以免费订阅5篇论文的施引文献，订阅后可以查看论文全部施引文献

1. Scaling Down to Scale Up: A Cost-Benefit Analysis of Replacing OpenAI's LLM with Open Source SLMs in Production;2024 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS);2024-05-05

2. TCCL: Discovering Better Communication Paths for PCIe GPU Clusters;Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3;2024-04-27

3. T3: Transparent Tracking & Triggering for Fine-grained Overlap of Compute & Collectives;Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2;2024-04-27

4. Heet: Accelerating Elastic Training in Heterogeneous Deep Learning Clusters;Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2;2024-04-27

5. CUTE: A scalable CPU-centric and Ultra-utilized Tensor Engine for convolutions;Journal of Systems Architecture;2024-04