Performance Implication of Tensor Irregularity and Optimization for Distributed Tensor Decomposition
-
Published:2023-06-20
Issue:2
Volume:10
Page:1-27
-
ISSN:2329-4949
-
Container-title:ACM Transactions on Parallel Computing
-
language:en
-
Short-container-title:ACM Trans. Parallel Comput.
Author:
Miao Zheng1ORCID,
Calhoun Jon C.2ORCID,
Ge Rong2ORCID,
Li Jiajia3ORCID
Affiliation:
1. Hangzhou Dianzi University, China
2. Clemson University, USA
3. North Carolina State University, USA
Abstract
Tensors are used by a wide variety of applications to represent multi-dimensional data; tensor decompositions are a class of methods for latent data analytics, data compression, and so on. Many of these applications generate large tensors with irregular dimension sizes and nonzero distribution. CANDECOMP/PARAFAC decomposition (
Cpd
) is a popular low-rank tensor decomposition for discovering latent features. The increasing overhead on memory and execution time of
Cpd
for large tensors requires distributed memory implementations as the only feasible solution. The sparsity and irregularity of tensors hinder the improvement of performance and scalability of distributed memory implementations. While previous works have been proved successful in
Cpd
for tensors with relatively regular dimension sizes and nonzero distribution, they either deliver unsatisfactory performance and scalability for irregular tensors or require significant time overhead in preprocessing. In this work, we focus on medium-grained tensor distribution to address their limitation for irregular tensors. We first thoroughly investigate through theoretical and experimental analysis. We disclose that the main cause of poor
Cpd
performance and scalability is the imbalance of multiple types of computations and communications and their tradeoffs; and sparsity and irregularity make it challenging to achieve their balances and tradeoffs. Irregularity of a sparse tensor is categorized based on two aspects: very different dimension sizes and a non-uniform nonzero distribution. Typically, focusing on optimizing one type of load imbalance causes other ones more severe for irregular tensors. To address such challenges, we propose irregularity-aware distributed
Cpd
that leverages the sparsity and irregularity information to identify the best tradeoff between different imbalances with low time overhead. We materialize the idea with two optimization methods: the prediction-based grid configuration and matrix-oriented distribution policy, where the former forms the global balance among computations and communications, and the latter further adjusts the balances among computations. The experimental results show that our proposed irregularity-aware distributed
Cpd
is more scalable and outperforms the medium- and fine-grained distributed implementations by up to 4.4 × and 11.4 × on 1,536 processors, respectively. Our optimizations support different sparse tensor formats, such as compressed sparse fiber (CSF), coordinate (COO), and Hierarchical Coordinate (HiCOO), and gain good scalability for all of them.
Funder
U.S. National Science Foundation Principles and Practice of Scalable Systems (PPoSS) program and by U.S. Department of Energy and Pacific Northwest National Laboratory
U.S. National Science Foundation
Publisher
Association for Computing Machinery (ACM)
Subject
Computational Theory and Mathematics,Computer Science Applications,Hardware and Architecture,Modeling and Simulation,Software
Reference40 articles.
1. Martín Abadi et al. 2015. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. (2015) arXiv preprint arXiv:1603.04467.
2. Improving Medium-Grain Partitioning for Scalable Sparse Tensor Decomposition
3. Phillip Alpatov, Greg Baker, H. Carter Edwards, John Gunnels, Greg Morrow, James Overfelt, and Robert van de Geijn. 1997. PLAPACK Parallel linear algebra package design overview. In Proceedings of the ACM/IEEE Conference on Supercomputing. IEEE, 29–29.
4. General Memory-Independent Lower Bound for MTTKRP