Is Network the Bottleneck of Distributed Training?-Reference-Cited by-同舟云学术

Is Network the Bottleneck of Distributed Training?

Published:2020-08-10 Issue: Volume: Page:
ISSN:
Container-title:Proceedings of the Workshop on Network Meets AI & ML
language:
Short-container-title:

Author:

Zhang Zhen¹,Chang Chaokun²,Lin Haibin²,Wang Yida²,Arora Raman¹,Jin Xin¹

Affiliation:

1. Johns Hopkins University

2. Amazon Web Services

Publisher

ACM

Link

https://dl.acm.org/doi/pdf/10.1145/3405671.3405810

Reference28 articles.

1. "AI and Compute." https://openai.com/blog/ai-and-compute/. "AI and Compute." https://openai.com/blog/ai-and-compute/.

2. "GPUs Power Five of World's Top Seven Supercomputers." https://www.hpcwire.com/2018/06/25/gpus-power-five-of-worlds-top-seven-supercomputers/. "GPUs Power Five of World's Top Seven Supercomputers." https://www.hpcwire.com/2018/06/25/gpus-power-five-of-worlds-top-seven-supercomputers/.

3. "Cloud TPU." https://cloud.google.com/tpu/. "Cloud TPU." https://cloud.google.com/tpu/.

4. Y. Peng Y. Zhu Y. Chen Y. Bao B. Yi C. Lan C. Wu and C. Guo "A generic communication scheduler for distributed dnn training acceleration " in ACM SOSP 2019. Y. Peng Y. Zhu Y. Chen Y. Bao B. Yi C. Lan C. Wu and C. Guo "A generic communication scheduler for distributed dnn training acceleration " in ACM SOSP 2019.

5. Y. Lin S. Han H. Mao Y. Wang and W. J. Dally "Deep gradient compression: Reducing the communication bandwidth for distributed training " arXiv preprint arXiv:1712.01887 2017. Y. Lin S. Han H. Mao Y. Wang and W. J. Dally "Deep gradient compression: Reducing the communication bandwidth for distributed training " arXiv preprint arXiv:1712.01887 2017.

Cited by 40 articles. 订阅此论文施引文献订阅此论文施引文献，注册后可以免费订阅5篇论文的施引文献，订阅后可以查看论文全部施引文献

1. Understanding Communication Characteristics of Distributed Training;Proceedings of the 8th Asia-Pacific Workshop on Networking;2024-08-03

2. SDCC: software-defined collective communication for distributed training;Science China Information Sciences;2024-07-31

3. PCNP: A RoCEv2 congestion control using precise CNP;Computer Networks;2024-06

4. Accelerating Lossy and Lossless Compression on Emerging BlueField DPU Architectures;2024 IEEE International Parallel and Distributed Processing Symposium (IPDPS);2024-05-27

5. Bandwidth Characterization of DeepSpeed on Distributed Large Language Model Training;2024 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS);2024-05-05