Receiver operating characteristic plot and area under the curve with binary classifiers: pragmatic analysis of cognitive screening instruments

Abstract

Aim: To examine whether receiver operating characteristic plots and area under the curve (AUC) values may be potentially misleading when assessing cognitive screening instruments as binary predictors rather than as categorical or continuous scales. Materials & methods: AUC was calculated using different methods (rank-sum, diagnostic odds ratio) using data from test accuracy studies of two binary classifiers of cognitive status (applause sign, attended with sign), a screener producing categorical data (Codex), and a continuous scale screening test (Mini-Addenbrooke’s Cognitive Examination). Results: For all screeners, AUC calculated using diagnostic odds ratio method was greater than using rank-sum method. When Codex and Mini-Addenbrooke’s Cognitive Examination were analyzed as binary (single fixed threshold) tests, AUC using rank-sum method was lower than when screeners were analyzed as categorical or continuous scales, respectively. Conclusion: If cognitive screeners producing categorical or continuous measures are dichotomized, calculated AUC may be an underestimate, thus affecting screening test accuracy.