Overlap Communication with Dependent Computation via Decomposition in Large Deep Learning Models-Reference-Cited by-同舟云学术

Overlap Communication with Dependent Computation via Decomposition in Large Deep Learning Models

Published:2022-12-19 Issue: Volume: Page:
ISSN:
Container-title:Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1
language:
Short-container-title:

Author:

Wang Shibo¹,Wei Jinliang¹,Sabne Amit¹,Davis Andy¹,Ilbeyi Berkin¹,Hechtman Blake¹,Chen Dehao²,Murthy Karthik Srinivasa¹,Maggioni Marcello¹,Zhang Qiao¹,Kumar Sameer¹,Guo Tongfei¹,Xu Yuanzhong¹,Zhou Zongwei¹

Affiliation:

1. Google, USA

2. Waymo, USA

Publisher

ACM

Link

https://dl.acm.org/doi/pdf/10.1145/3567955.3567959

Reference55 articles.

1. 2020. Google breaks AI performance records in MLPerf with world’s fastest training supercomputer. https://cloud.google.com/blog/products/ai-machine-learning/google-breaks-ai-performance-records-in-mlperf-with-worlds-fastest-training-supercomputer 2020. Google breaks AI performance records in MLPerf with world’s fastest training supercomputer. https://cloud.google.com/blog/products/ai-machine-learning/google-breaks-ai-performance-records-in-mlperf-with-worlds-fastest-training-supercomputer

2. 2021. MLPerf Training v1.1. https://mlcommons.org/en/training-normal-11/ 2021. MLPerf Training v1.1. https://mlcommons.org/en/training-normal-11/

3. 2021. XLA: Optimizing Compiler for TensorFlow. https://www.tensorflow.org/xla 2021. XLA: Optimizing Compiler for TensorFlow. https://www.tensorflow.org/xla

4. 2022. NVIDIA H100 Tensor Core GPU Architecture. https://www.hpctech.co.jp/catalog/gtc22-whitepaper-hopper_v1.01.pdf 2022. NVIDIA H100 Tensor Core GPU Architecture. https://www.hpctech.co.jp/catalog/gtc22-whitepaper-hopper_v1.01.pdf

5. 2022. XLA DynamicSlice Semantics. https://www.tensorflow.org/xla/operation_semantics##dynamicslice 2022. XLA DynamicSlice Semantics. https://www.tensorflow.org/xla/operation_semantics##dynamicslice

Cited by 2 articles. 订阅此论文施引文献订阅此论文施引文献，注册后可以免费订阅5篇论文的施引文献，订阅后可以查看论文全部施引文献

1. Calculon: a methodology and tool for high-level co-design of systems and large language models;Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis;2023-11-11

2. Tale of Two Cs: Computation vs. Communication Scaling for Future Transformers on Future Hardware;2023 IEEE International Symposium on Workload Characterization (IISWC);2023-10-01