From Missteps to Milestones: A Journey to Practical Fail-Slow Detection-Reference-Cited by-同舟云学术

From Missteps to Milestones: A Journey to Practical Fail-Slow Detection

Published:2023-11 Issue:4 Volume:19 Page:1-28
ISSN:1553-3077
Container-title:ACM Transactions on Storage
language:en
Short-container-title:ACM Trans. Storage

Author:

Lu Ruiming¹^ORCID,Xu Erci²^ORCID,Zhang Yiming³^ORCID,Zhu Fengyi⁴^ORCID,Zhu Zhaosheng⁴^ORCID,Wang Mengtian⁴^ORCID,Zhu Zongpeng⁴^ORCID,Xue Guangtao¹^ORCID,Shu Jiwu³^ORCID,Li Minglu⁵^ORCID,Wu Jiesheng⁴^ORCID

Affiliation:

1. Shanghai Jiao Tong University, China

2. Alibaba Inc. and Shanghai Jiao Tong University, China

3. Xiamen University, China

4. Alibaba Inc., China

5. Shanghai Jiao Tong University and Zhejiang Normal University, China

Abstract

The newly emerging “fail-slow” failures plague both software and hardware where the victim components are still functioning yet with degraded performance. To address this problem, this article presents Perseus , a practical fail-slow detection framework for storage devices. Perseus leverages a light regression-based model to quickly pinpoint and analyze fail-slow failures at the granularity of drives. Within a 10-month close monitoring on 248K drives, Perseus managed to find 304 fail-slow cases. Isolating them can reduce the (node-level) 99.99th tail latency by 48%. We assemble a large-scale fail-slow dataset (including 41K normal drives and 315 verified fail-slow drives) from our production traces, based on which we provide root cause analysis on fail-slow drives covering a variety of ill-implemented scheduling, hardware defects, and environmental factors. We have released the dataset to the public for fail-slow study.

Funder

NSFC

Alibaba Innovation Research

National Key R&D Program of China

Program of Hunan Postdoc Innovation

Program of Shanghai Academic Research Leader

Publisher

Association for Computing Machinery (ACM)

Subject

Hardware and Architecture

Link

https://dl.acm.org/doi/pdf/10.1145/3617690

Reference51 articles.

1. (n.d.). S.M.A.R.T. (Self-Monitoring Analysis and Reporting Technology). https://en.wikipedia.org/wiki/S.M.A.R.T.

2. Principal component analysis;Abdi Hervé;WIREs Computational Statistics,2010

3. Ramnatthan Alagappan, Aishwarya Ganesan, Yuvraj Patel, Thanumalayan Sankaranarayana Pillai, Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau. 2016. Correlated crash vulnerabilities. In Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI). https://www.usenix.org/conference/osdi16/technical-sessions/presentation/alagappan

4. Jacob Alter, Ji Xue, Alma Dimnaku, and Evgenia Smirni. 2019. SSD failures in the field: Symptoms, causes, and prediction models. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC). Article 75, 14 pages. DOI:10.1145/3295500.3356172

5. Behnaz Arzani, Selim Ciraci, Luiz Chamon, Yibo Zhu, Hongqiang (Harry) Liu, Jitu Padhye, Boon Thau Loo, and Geoff Outhred. 2018. 007: Democratically finding the cause of packet drops. In Proceedings of the 15th USENIX Symposium on Networked Systems Design and Implementation (NSDI). https://www.usenix.org/conference/nsdi18/presentation/arzani