A Survey on Auto-Parallelism of Large-Scale Deep Learning Training-Reference-Cited by-同舟云学术

A Survey on Auto-Parallelism of Large-Scale Deep Learning Training

Author:

Liang Peng¹^ORCID,Tang Yu¹^ORCID,Zhang Xiaoda²,Bai Youhui²,Su Teng²^ORCID,Lai Zhiquan¹^ORCID,Qiao Linbo¹^ORCID,Li Dongsheng¹^ORCID

Affiliation:

1. State Key Laboratory of Parallel and Distributed Processing, National University of Defense Technology, Changsha, China

2. Huawei Technologies Co. Ltd., Shenzhen, China

Funder

National Natural Science Foundation of China

Publisher

Institute of Electrical and Electronics Engineers (IEEE)

Subject

Computational Theory and Mathematics,Hardware and Architecture,Signal Processing

Link

Reference102 articles.

1. Attention is all you need;vaswani;Proc Adv Neural Inf Process Syst,2017

2. TeraPipe: Token-level pipeline parallelism for training large-scale language models;li;Proc 38th Int Conf Mach Learn,2021

3. Exploring hidden dimensions in accelerating convolutional neural networks;jia;Proc 35th Int Conf Mach Learn,2018

4. Sequence parallelism: Making 4D parallelism possible;li,2021

5. End-to-end adaptive distributed training on PaddlePaddle;ao,2021

Cited by 8 articles. 订阅此论文施引文献订阅此论文施引文献，注册后可以免费订阅5篇论文的施引文献，订阅后可以查看论文全部施引文献

4. Distributed Training Optimization for DCU;2024 5th International Conference on Information Science, Parallel and Distributed Systems (ISPDS);2024-05-31