Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compression-Reference-Cited by-同舟云学术

Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compression

Published:2023-01-27 Issue: Volume: Page:
ISSN:
Container-title:Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2
language:
Short-container-title:

Author:

Song Jaeyong¹,Yim Jinkyu²,Jung Jaewon¹,Jang Hongsun²,Kim Hyung-Jin³,Kim Youngsok¹,Lee Jinho²

Affiliation:

1. Yonsei University, South Korea

2. Seoul National University, South Korea

3. Samsung Electronics, South Korea

Publisher

ACM

Link

https://dl.acm.org/doi/pdf/10.1145/3575693.3575712

Reference89 articles.

1. Saurabh Agarwal Hongyi Wang Shivaram Venkataraman and Dimitris Papailiopoulos. 2022. On the Utility of Gradient Compression in Distributed Training Systems. In MLSys. Saurabh Agarwal Hongyi Wang Shivaram Venkataraman and Dimitris Papailiopoulos. 2022. On the Utility of Gradient Compression in Distributed Training Systems. In MLSys.

2. Takuya Akiba Shuji Suzuki and Keisuke Fukuda. 2017. Extremely Large Minibatch SGD: Training Resnet-50 on Imagenet in 15 Minutes. arXiv preprint arXiv:1711.04325. Takuya Akiba Shuji Suzuki and Keisuke Fukuda. 2017. Extremely Large Minibatch SGD: Training Resnet-50 on Imagenet in 15 Minutes. arXiv preprint arXiv:1711.04325.

3. Aida Amini Saadia Gabriel Shanchuan Lin Rik Koncel-Kedziorski Yejin Choi and Hannaneh Hajishirzi. 2019. MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms. In NAACL. 2357–2367. Aida Amini Saadia Gabriel Shanchuan Lin Rik Koncel-Kedziorski Yejin Choi and Hannaneh Hajishirzi. 2019. MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms. In NAACL. 2357–2367.

4. Debraj Basu , Deepesh Data , Can Karakus, and Suhas Diggavi. 2019 . Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification and Local Computations. In NeurIPS. Debraj Basu, Deepesh Data, Can Karakus, and Suhas Diggavi. 2019. Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification and Local Computations. In NeurIPS.

5. Jeremy Bernstein Yu-Xiang Wang Kamyar Azizzadenesheli and Anima Anandkumar. 2018. signSGD: compressed optimisation for non-convex problems. In ICML. Jeremy Bernstein Yu-Xiang Wang Kamyar Azizzadenesheli and Anima Anandkumar. 2018. signSGD: compressed optimisation for non-convex problems. In ICML.

Cited by 9 articles. 订阅此论文施引文献订阅此论文施引文献，注册后可以免费订阅5篇论文的施引文献，订阅后可以查看论文全部施引文献

1. Scaling Down to Scale Up: A Cost-Benefit Analysis of Replacing OpenAI's LLM with Open Source SLMs in Production;2024 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS);2024-05-05

2. PrimePar: Efficient Spatial-temporal Tensor Partitioning for Large Transformer Model Training;Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3;2024-04-27

3. ScheMoE: An Extensible Mixture-of-Experts Distributed Training System with Tasks Scheduling;Proceedings of the Nineteenth European Conference on Computer Systems;2024-04-22

4. SoCFlow: Efficient and Scalable DNN Training on SoC-Clustered Edge Servers;Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1;2024-04-17

5. Smart-Infinity: Fast Large Language Model Training using Near-Storage Processing on a Real System;2024 IEEE International Symposium on High-Performance Computer Architecture (HPCA);2024-03-02