ScheMoE: An Extensible Mixture-of-Experts Distributed Training System with Tasks Scheduling-Reference-Cited by-同舟云学术

ScheMoE: An Extensible Mixture-of-Experts Distributed Training System with Tasks Scheduling

Published:2024-04-22 Issue: Volume:1 Page:236-249
ISSN:
Container-title:Proceedings of the Nineteenth European Conference on Computer Systems
language:
Short-container-title:

Author:

Shi Shaohuai¹^ORCID,Pan Xinglin²^ORCID,Wang Qiang¹^ORCID,Liu Chengjian³^ORCID,Ren Xiaozhe⁴^ORCID,Hu Zhongzhe⁴^ORCID,Yang Yu⁴^ORCID,Li Bo⁵^ORCID,Chu Xiaowen⁶^ORCID

Affiliation:

1. Harbin Institute of Technology, Shenzhen

2. The Hong Kong University of Science and Technology (Guangzhou)

3. Shenzhen Technology University

4. Huawei Central Research Institute, Huawei Technologies

5. The Hong Kong University of Science and Technology

6. The Hong Kong University of Science and Technology (Guangzhou) and The Hong Kong University of Science and Technology

Funder

National Science Foundation of China (NSFC)

Hong Kong RGC GRF

Shenzhen Science and Technology Program

School-Enterprise Cooperation Project of Shenzhen Technology University

Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies

Hong Kong RIF

Hong Kong RGC CRF

Publisher

ACM

Link

https://dl.acm.org/doi/pdf/10.1145/3627703.3650083

Reference55 articles.

1. 2022-07-13. Doubling all2all Performance with NVIDIA Collective Communication Library 2.12. https://developer.nvidia.com/blog/doubling-all2all-performance-with-nvidia-collective-communication-library-2-12/ (2022-07-13).

2. DeepSpeed- Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale

3. Yixin Bao, Yanghua Peng, Yangrui Chen, and Chuan Wu. 2020. Preemptive All-reduce Scheduling for Expediting Distributed DNN Training. In IEEE INFOCOM 2020-IEEE Conference on Computer Communications. IEEE, 626--635.

4. Gradient amplification: An efficient way to train deep neural networks

5. Findings of the 2014 Workshop on Statistical Machine Translation