Evaluating binary classifiers: extending the Efficiency Index

Abstract

AbstractAn “efficiency index” (EI) for the evaluation of binary classifiers was recently characterised, where EI is the ratio of classifier accuracy to inaccuracy. The purpose of this study was to further develop EI by substituting balanced accuracy and unbiased accuracy in place of accuracy, and their respective complements in place of inaccuracy, to construct balanced EI and unbiased EI measures. Additional investigations, using the dataset of a prospective pragmatic test accuracy study of a cognitive screening instrument, explored use of the log method to calculate confidence intervals for the various EI formulations; the dependence of EI formulations on prevalence; and comparison of EI formulations with analogous formulations based on the Identification Index (II), a previously described metric which is also based on accuracy and inaccuracy, where II is accuracy minus inaccuracy. EI formulations are shown to have advantages over II formulations, in particular their boundary values (0 and ∞) mean that negative values never occur, unlike the case for II, and the inflection point of 1 demarcates likelihood of correct versus incorrect classification.