End-to-end I/O Monitoring on Leading Supercomputers
-
Published:2023-01-11
Issue:1
Volume:19
Page:1-35
-
ISSN:1553-3077
-
Container-title:ACM Transactions on Storage
-
language:en
-
Short-container-title:ACM Trans. Storage
Author:
Yang Bin1ORCID, Xue Wei2ORCID, Zhang Tianyu3ORCID, Liu Shichao3ORCID, Ma Xiaosong4ORCID, Wang Xiyang5ORCID, Liu Weiguo3ORCID
Affiliation:
1. Shandong University, National Supercomputing Center in Wuxi, Jinan, China 2. Tsinghua, Beijing, University, Beijing, China 3. Shandong University, Jinan, China 4. Qatar Computing Research Institute, HBKU, Doha, Qatar 5. National Supercomputing Center in Wuxi, Wuxi, China
Abstract
This paper offers a solution to overcome the complexities of production system I/O performance monitoring. We present Beacon, an end-to-end I/O resource monitoring and diagnosis system for the 40960-node Sunway TaihuLight supercomputer, currently the fourth-ranked supercomputer in the world. Beacon simultaneously collects and correlates I/O tracing/profiling data from all the compute nodes, forwarding nodes, storage nodes, and metadata servers. With mechanisms such as aggressive online and offline trace compression and distributed caching/storage, it delivers scalable, low-overhead, and sustainable I/O diagnosis under production use. With Beacon’s deployment on TaihuLight for more than three years, we demonstrate Beacon’s effectiveness with real-world use cases for I/O performance issue identification and diagnosis. It has already successfully helped center administrators identify obscure design or configuration flaws, system anomaly occurrences, I/O performance interference, and resource under- or over-provisioning problems. Several of the exposed problems have already been fixed, with others being currently addressed. Encouraged by Beacon’s success in I/O monitoring, we extend it to monitor interconnection networks, which is another contention point on supercomputers. In addition, we demonstrate Beacon’s generality by extending it to other supercomputers. Both Beacon codes and part of collected monitoring data are released.
1
Funder
National Key R&D Program of China National Natural Science Foundation of China
Publisher
Association for Computing Machinery (ACM)
Subject
Hardware and Architecture
Reference96 articles.
1. Anthony Agelastos, Benjamin Allan, Jim Brandt, Paul Cassella, Jeremy Enos, Joshi Fullop, Ann Gentile, Steve Monk, Nichamon Naksinehaboon, Jeff Ogden, Mahesh Rajan, Michael Showerman, Joel Stevenson, Narate Taerat, and Tom Tucker. 2014. The lightweight distributed metric service: A scalable infrastructure for continuous monitoring of large scale computing systems and applications. In International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, New Orleans, 154–165. 2. John Bent, Garth Gibson, Gary Grider, Ben McClelland, Paul Nowoczynski, James Nunez, Milo Polte, and Meghan Wingate. 2009. PLFS: A checkpoint filesystem for parallel applications. In International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, Portland, 1–12. 3. Lawrence Berkeley and ANL. 2017. TOKIO: Total knowledge of I/O. http://www.nersc.gov/research-and-development/tokio. 4. Lustre: A scalable, high performance file system;Braam Peter J.;Cluster File Systems, Inc,2002 5. Jim Brandt, Ann Gentile, Jackson Mayo, Philippe Pebay, Diana Roe, David Thompson, and Matthew Wong. 2009. Resource monitoring and management with OVIS to enable HPC in cloud computing environments. In International Symposium on Parallel and Distributed Processing. IEEE, Rome, 1–8.
Cited by
10 articles.
订阅此论文施引文献
订阅此论文施引文献，注册后可以免费订阅5篇论文的施引文献，订阅后可以查看论文全部施引文献
|
|