Merak: An Efficient Distributed DNN Training Framework With Automated 3D Parallelism for Giant Foundation Models-Reference-Cited by-同舟云学术

Merak: An Efficient Distributed DNN Training Framework With Automated 3D Parallelism for Giant Foundation Models

Published:2023-05 Issue:5 Volume:34 Page:1466-1478
ISSN:1045-9219
Container-title:IEEE Transactions on Parallel and Distributed Systems
language:
Short-container-title:IEEE Trans. Parallel Distrib. Syst.

Author:

Lai Zhiquan¹^ORCID,Li Shengwei¹^ORCID,Tang Xudong¹,Ge Keshi¹^ORCID,Liu Weijie¹^ORCID,Duan Yabo¹,Qiao Linbo¹^ORCID,Li Dongsheng¹^ORCID

Affiliation:

1. National Laboratory for Parallel and Distributed Processing, College of Computer, National University of Defense Technology, Changsha, Hunan, China

Funder

National Key R&D Program of China

National Natural Science Foundation of China

Publisher

Institute of Electrical and Electronics Engineers (IEEE)

Subject

Computational Theory and Mathematics,Hardware and Architecture,Signal Processing

Link

http://xplorestaging.ieee.org/ielx7/71/10075651/10049507.pdf?arnumber=10049507

Reference52 articles.

1. PyTorch: An imperative style, high-performance deep learning library;paszke;Proc Adv Neural Inf Process Syst,2019

2. End-to-end adaptive distributed training on paddlepaddle;ao,2021

3. DeepSpeed

4. Transformers: State-of-the-Art Natural Language Processing

5. Openwebtext corpus;gokaslan,2019

Cited by 22 articles. 订阅此论文施引文献订阅此论文施引文献，注册后可以免费订阅5篇论文的施引文献，订阅后可以查看论文全部施引文献

1. DELTA: Memory-Efficient Training via Dynamic Fine-Grained Recomputation and Swapping;ACM Transactions on Architecture and Code Optimization;2024-08-20

2. Proactive Caching With Distributed Deep Reinforcement Learning in 6G Cloud-Edge Collaboration Computing;IEEE Transactions on Parallel and Distributed Systems;2024-08

3. Distributed Foundation Models for Multi-Modal Learning in 6G Wireless Networks;IEEE Wireless Communications;2024-06

4. Distributed Training Optimization for DCU;2024 5th International Conference on Information Science, Parallel and Distributed Systems (ISPDS);2024-05-31

5. Construction of Raining System Based on 3D Virtual Simulation;2024 Second International Conference on Data Science and Information System (ICDSIS);2024-05-17