EmbRace: Accelerating Sparse Communication for Distributed Training of Deep Neural Networks

Author:

Li Shengwei1,Lai Zhiquan1,Li Dongsheng1,Zhang Yiming2,Ye Xiangyu1,Duan Yabo1

Affiliation:

1. National University of Defense Technology, China

2. National University of Defense Technology, China and Xiamen University, China

Funder

National Key R&D Program of China

Publisher

ACM

Reference43 articles.

1. 2019. Gloo: a collective communications library.https://github.com/facebookincubator/gloo 2019. Gloo: a collective communications library.https://github.com/facebookincubator/gloo

2. 2019. NVIDIA Collective Communications Library (NCCL). https://developer.nvidia.com/nccl 2019. NVIDIA Collective Communications Library (NCCL). https://developer.nvidia.com/nccl

3. QSGD: Communication-efficient SGD via gradient quantization and encoding;Alistarh Dan;Advances in Neural Information Processing Systems,2017

4. Preemptive All-reduce Scheduling for Expediting Distributed DNN Training

5. Ondřej Bojar , Christian Buck , Christian Federmann , Barry Haddow , Philipp Koehn , Johannes Leveling , Christof Monz , Pavel Pecina , Matt Post , Herve Saint-Amand , 2014 . Findings of the 2014 workshop on statistical machine translation. In Proceedings of the ninth workshop on statistical machine translation. 12–58 . Ondřej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, 2014. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the ninth workshop on statistical machine translation. 12–58.

Cited by 2 articles. 订阅此论文施引文献 订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献

1. Prophet: Fine-grained Load Balancing for Parallel Training of Large-scale MoE Models;2023 IEEE International Conference on Cluster Computing (CLUSTER);2023-10-31

2. Parallel intelligent computing: development and challenges;SCIENTIA SINICA Informationis;2023-08-01

同舟云学术

1.学者识别学者识别

2.学术分析学术分析

3.人才评估人才评估

"同舟云学术"是以全球学者为主线,采集、加工和组织学术论文而形成的新型学术文献查询和分析系统,可以对全球学者进行文献检索和人才价值评估。用户可以通过关注某些学科领域的顶尖人物而持续追踪该领域的学科进展和研究前沿。经过近期的数据扩容,当前同舟云学术共收录了国内外主流学术期刊6万余种,收集的期刊论文及会议论文总量共计约1.5亿篇,并以每天添加12000余篇中外论文的速度递增。我们也可以为用户提供个性化、定制化的学者数据。欢迎来电咨询!咨询电话:010-8811{复制后删除}0370

www.globalauthorid.com

TOP

Copyright © 2019-2024 北京同舟云网络信息技术有限公司
京公网安备11010802033243号  京ICP备18003416号-3