A new accuracy metric under three classes when subclasses are involved and its confidence interval estimation

Abstract

“Compound multi‐class classification” refers to the setting where three or more main classes are involved and at least one of the main classes have multiple subclasses. A common practice in evaluating biomarker performance under “compound multi‐class classification” is “subclasses pooling.” In this article, we first explore the downsides of accuracy metrics based on pooled data. Then we propose a new accuracy measure proper for “compound multi‐class classification” with three ordinal main classes, namely “volume under compound surface ().” The proposed evaluates the accuracy of a biomarker appropriately by identifying main classes without requiring specification of an ordering for marker values of subclasses within each main class. For confidence interval estimation of , both parametric and nonparametric methods are studied, and simulation studies are carried out to assess coverage probabilities. A subset of Alzheimer's Disease Neuroimaging Initiative study dataset is analyzed.