Review of Metrics and Assignment of Confidence Intervals for Health Management of Gas Turbine Engines

Abstract

The development and evaluation of new diagnostic systems requires statistically-based methods to measure performance. Various metrics are in use by developers and users of diagnostic systems. Current metrics practices are reviewed, including receiver operating characteristics, confusion matrices, Kappa coefficients and various entropy techniques. A set of metrics is then proposed for assessment of diverse gas path diagnostic systems. The use of bootstrap statistics to compare metric results is developed, and demonstrated for a set of hypothetical data sets with a range of relevant characteristics. The bootstrap technique allows the expected range of the metric to be assessed without assuming a probability distribution. A method is proposed to develop confidence intervals for the calculated metrics. The application of a confidence interval could prevent a good diagnostic technique being discarded because of a lower value metric in one test instance. The strengths and weaknesses of the various metrics with derived confidence intervals are discussed. Recommendations are made for further work.