ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scale-Reference-Cited by-同舟云学术

ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scale

Published:2023-04 Issue: Volume: Page:
ISSN:
Container-title:2023 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS)
language:
Short-container-title:

Author:

Won William¹,Heo Taekyung¹,Rashidi Saeed¹,Sridharan Srinivas²,Srinivasan Sudarshan³,Krishna Tushar¹

Affiliation:

1. Georgia Institute of Technology Atlanta,GA,USA

2. Meta, Menlo Park,CA,USA

3. Intel,Bangalore,Karnataka,India

Publisher

IEEE

Link

http://xplorestaging.ieee.org/ielx7/10158063/10158070/10158106.pdf?arnumber=10158106

Reference80 articles.

1. Impact of RoCE Congestion Control Policies on Distributed Training of DNNs

2. Scalable Distributed Training of Recommendation Models: An ASTRA-SIM + NS3 case-study with TCP/IP transport

3. GShard: Scaling giant models with conditional computation and automatic sharding;lepikhin;Proceedings of the 9th International Conference on Learning Representations (ICLR),2021

4. Efficient large-scale language model training on GPU clusters using megatron-LM

Cited by 9 articles. 订阅此论文施引文献订阅此论文施引文献，注册后可以免费订阅5篇论文的施引文献，订阅后可以查看论文全部施引文献

1. Towards a Standardized Representation for Deep Learning Collective Algorithms;2024 IEEE Symposium on High-Performance Interconnects (HOTI);2024-08-21

2. POSTER: Hybrid-Granularity Network Load balancing for Distributed AI Model Training;Proceedings of the ACM SIGCOMM 2024 Conference: Posters and Demos;2024-08-04

3. MAD-Max Beyond Single-Node: Enabling Large Machine Learning Model Acceleration on Distributed Systems;2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA);2024-06-29

4. The Case For Data Centre Hyperloops;2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA);2024-06-29

5. LLMCompass: Enabling Efficient Hardware Design for Large Language Model Inference;2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA);2024-06-29