Efficient large-scale language model training on GPU clusters using megatron-LM-Reference-Cited by-同舟云学术

Efficient large-scale language model training on GPU clusters using megatron-LM

Published:2021-11-13 Issue: Volume: Page:
ISSN:
Container-title:Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis
language:
Short-container-title:

Author:

Narayanan Deepak¹,Shoeybi Mohammad²,Casper Jared²,LeGresley Patrick²,Patwary Mostofa²,Korthikanti Vijay²,Vainbrand Dmitri²,Kashinkunti Prethvi²,Bernauer Julie²,Catanzaro Bryan²,Phanishayee Amar³,Zaharia Matei⁴

Affiliation:

1. NVIDIA and Microsoft Research

2. NVIDIA

3. Microsoft Research

4. Stanford University

Publisher

ACM

Link

https://dl.acm.org/doi/pdf/10.1145/3458817.3476209

Reference47 articles.

1. Applications of GPT-3. https://openai.com/blog/gpt-3-apps/. Applications of GPT-3. https://openai.com/blog/gpt-3-apps/.

2. DeepSpeed: Extreme-Scale Model Training for Everyone. https://www.microsoft.com/en-us/research/blog/deepspeed-extreme-scale-model-training-for-everyone/. DeepSpeed: Extreme-Scale Model Training for Everyone. https://www.microsoft.com/en-us/research/blog/deepspeed-extreme-scale-model-training-for-everyone/.

3. DeepSpeed Repository. https://www.deepspeed.ai/. DeepSpeed Repository. https://www.deepspeed.ai/.

4. GitHub Copilot. https://copilot.github.com/. GitHub Copilot. https://copilot.github.com/.

5. Microsoft Translates Spoken Text to Code. https://techcrunch.com/2021/05/25/microsoft-uses-gpt-3-to-let-you-code-in-natural-language/. Microsoft Translates Spoken Text to Code. https://techcrunch.com/2021/05/25/microsoft-uses-gpt-3-to-let-you-code-in-natural-language/.

Cited by 167 articles. 订阅此论文施引文献订阅此论文施引文献，注册后可以免费订阅5篇论文的施引文献，订阅后可以查看论文全部施引文献

1. Proteus: Simulating the Performance of Distributed DNN Training;IEEE Transactions on Parallel and Distributed Systems;2024-10

2. Swift: Expedited Failure Recovery for Large-Scale DNN Training;IEEE Transactions on Parallel and Distributed Systems;2024-09

3. BC4LLM: A perspective of trusted artificial intelligence when blockchain meets large language models;Neurocomputing;2024-09

4. Rail-only: A Low-Cost High-Performance Network for Training LLMs with Trillion Parameters;2024 IEEE Symposium on High-Performance Interconnects (HOTI);2024-08-21

5. Holmes: Towards Distributed Training Across Clusters with Heterogeneous NIC Environment;Proceedings of the 53rd International Conference on Parallel Processing;2024-08-12