Author:
Kim Sunggon,Sim Alex,Wu Kesheng,Byna Suren,Son Yongseok
Abstract
AbstractLarge-scale high performance computing (HPC) systems typically consist of many thousands of CPUs and storage units used by hundreds to thousands of users simultaneously. Applications from large numbers of users have diverse characteristics, such as varying computation, communication, memory, and I/O intensity. A good understanding of the performance characteristics of each user application is important for job scheduling and resource provisioning. Among these performance characteristics, I/O performance is becoming increasingly important as data sizes rapidly increase and large-scale applications, such as simulation and model training, are widely adopted. However, predicting I/O performance is difficult because I/O systems are shared among all users and involve many layers of software and hardware stack, including the application, network interconnect, operating system, file system, and storage devices. Furthermore, updates to these layers and changes in system management policy can significantly alter the I/O behavior of applications and the entire system. To improve the prediction of the I/O performance on HPC systems, we propose integrating information from several different system logs and developing a regression-based approach to predict the I/O performance. Our proposed scheme can dynamically select the most relevant features from the log entries using various feature selection algorithms and scoring functions, and can automatically select the regression algorithm with the best accuracy for the prediction task. The evaluation results show that our proposed scheme can predict the write performance with up to 90% prediction accuracy and the read performance with up to 99% prediction accuracy using the real logs from the Cori supercomputer system at NERSC.
Funder
Seoultech
U.S. Department of Energy
National Research Foundation of Korea
Publisher
Springer Science and Business Media LLC
Subject
Information Systems and Management,Computer Networks and Communications,Hardware and Architecture,Information Systems
Reference53 articles.
1. Abadi M et al. Tensorflow: a system for large-scale machine learning. In: 12th $$\{$$USENIX$$\}$$ Symposium on Operating Systems Design and Implementation ($$\{$$OSDI$$\}$$ 16); 2016. p. 265–83.
2. Agarwal M, Singhvi D, Malakar P, Byna S. Active learning-based automatic tuning and prediction of parallel i/o performance. In: 2019 IEEE/ACM Fourth International Parallel Data Systems Workshop (PDSW), IEEE; 2019. p. 20–9.
3. Behzad B et al. Improving parallel I/O autotuning with performance modeling. In: Proceedings of the 23rd International Symposium on High-Performance Parallel and Distributed Computing, Association for Computing Machinery, New York, NY, USA; 2014. p. 253–56. https://doi.org/10.1145/2600212.2600708.
4. Behzad B et al. Pattern-driven parallel I/O tuning. In: Proceedings of the 10th Parallel Data Storage Workshop, ACM, New York, NY, USA; 2015. p. 43–48. https://doi.org/10.1145/2834976.2834977.
5. Benesty J, et al. Pearson correlation coefficient. In: Davis GM, editor., et al., Noise reduction in speech processing. Heidelberg: Springer; 2009. p. 1–4.
Cited by
1 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献
1. Capturing Periodic I/O Using Frequency Techniques;2024 IEEE International Parallel and Distributed Processing Symposium (IPDPS);2024-05-27