Affiliation:
1. University of Illinois at Urbana-Champaign
Abstract
Modern supercomputers with torus networks allow each node to simultaneously pass messages on all of its links. However, most collective algorithms are designed to only use one link at a time. In this work, we present novel multiported algorithms for the scatter, gather, all-gather, and reduce-scatter operations. Our algorithms can be combined to create multiported reduce, all-reduce, and broadcast algorithms. Several of these algorithms involve a new technique where we relax the MPI message-ordering constraints to achieve high performance and restore the correctordering using an additional stage of redundant communication.
According to our models, on an
n
-dimensional torus, our algorithms should allow for nearly a 2
n
-fold improvement in communication performance compared to known, single-ported torus algorithms. In practice, we have achieved nearly 6x better performance on a 32k-node 3-dimensional torus.
Funder
Department of Energy
National Science Foundation
Publisher
Association for Computing Machinery (ACM)
Subject
Computational Theory and Mathematics,Computer Science Applications,Hardware and Architecture,Modeling and Simulation,Software
Cited by
2 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献
1. In-network Allreduce with Multiple Spanning Trees on PolarFly;Proceedings of the 35th ACM Symposium on Parallelism in Algorithms and Architectures;2023-06-17
2. Communication Optimization Technology Based on Network Dynamic Performance Model;Mathematical Problems in Engineering;2020-10-15