Hi-Speed DNN Training with Espresso: Unleashing the Full Potential of Gradient Compression with Near-Optimal Usage Strategies-Reference-Cited by-同舟云学术

Hi-Speed DNN Training with Espresso: Unleashing the Full Potential of Gradient Compression with Near-Optimal Usage Strategies

Published:2023-05-08 Issue: Volume: Page:
ISSN:
Container-title:Proceedings of the Eighteenth European Conference on Computer Systems
language:
Short-container-title:

Author:

Wang Zhuang¹^ORCID,Lin Haibin²^ORCID,Zhu Yibo³^ORCID,Ng T. S. Eugene⁴^ORCID

Affiliation:

1. Computer Science Department, Rice University, Houston, Texas, United States of America

2. ByteDance Inc., Newark, California, USA

3. ByteDance Inc., Bellevue, Washington, United States of America

4. Rice University, Houston, Texas, United States of America

Funder

National Science Foundation

Publisher

ACM

Link

https://dl.acm.org/doi/pdf/10.1145/3552326.3567505

Reference79 articles.

1. 2021. NVIDIA NCCL. https://developer.nvidia.com/NCCL. 2021. NVIDIA NCCL. https://developer.nvidia.com/NCCL.

2. 2022. MLaaS in the Wild: Workload Analysis and Scheduling in Large-Scale Heterogeneous GPU Clusters. In 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI 22) . USENIX Association, Renton, WA. https://www.usenix.org/conference/nsdi22/presentation/weng 2022. MLaaS in the Wild: Workload Analysis and Scheduling in Large-Scale Heterogeneous GPU Clusters. In 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI 22). USENIX Association, Renton, WA. https://www.usenix.org/conference/nsdi22/presentation/weng

3. Martín Abadi , Paul Barham , Jianmin Chen , Zhifeng Chen , Andy Davis , Jeffrey Dean , Matthieu Devin , Sanjay Ghemawat , Geoffrey Irving , Michael Isard , Manjunath Kudlur , Josh Levenberg , Rajat Monga , Sherry Moore , Derek G. Murray , Benoit Steiner , Paul Tucker , Vijay Vasudevan , Pete Warden , Martin Wicke , Yuan Yu , and Xiaoqiang Zheng . 2016 . Tensorflow: A system for large-scale machine learning . In USENIX Symposium on Operating Systems Design and Implementation (OSDI). 265--283 . Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. Tensorflow: A system for large-scale machine learning. In USENIX Symposium on Operating Systems Design and Implementation (OSDI). 265--283.

4. Saurabh Agarwal , Hongyi Wang , Shivaram Venkataraman , and Dimitris Papailiopoulos . 2021. On the Utility of Gradient Compression in Distributed Training Systems. arXiv preprint arXiv:2103.00543 ( 2021 ). Saurabh Agarwal, Hongyi Wang, Shivaram Venkataraman, and Dimitris Papailiopoulos. 2021. On the Utility of Gradient Compression in Distributed Training Systems. arXiv preprint arXiv:2103.00543 (2021).

5. Alham Fikri Aji and Kenneth Heafield. 2017. Sparse communication for distributed gradient descent. Alham Fikri Aji and Kenneth Heafield. 2017. Sparse communication for distributed gradient descent.

Cited by 6 articles. 订阅此论文施引文献订阅此论文施引文献，注册后可以免费订阅5篇论文的施引文献，订阅后可以查看论文全部施引文献

1. Understanding Silent Data Corruption in Processors for Mitigating its Effects;ACM Transactions on Architecture and Code Optimization;2024-09-02

2. Sparse Gradient Communication with AlltoAll for Accelerating Distributed Deep Learning;Proceedings of the 53rd International Conference on Parallel Processing;2024-08-12

3. SqueezeNIC: Low-Latency In-NIC Compression for Distributed Deep Learning;Proceedings of the 2024 SIGCOMM Workshop on Networks for AI Computing;2024-08-04

4. CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving;Proceedings of the ACM SIGCOMM 2024 Conference;2024-08-04

5. Accelerating Model Training in Multi-cluster Environments with Consumer-grade GPUs;Proceedings of the ACM SIGCOMM 2024 Conference;2024-08-04