Comparing the Accuracy of Diagnostic Tests When Disease Is Characterized by an Ordinal Scale

Abstract

Abstract In diagnostic medicine, the true disease status of a patient is often represented on an ordinal scale—for example, cancer stage (0, I, II, III, or IV) or coronary artery disease severity measured using the Coronary Artery Disease Reporting and Data System (CAD-RADS) scale (none, minimal, mild, moderate, severe, or occluded). With advances in quantitation of diagnostic images and in artificial intelligence (AI), both supervised and unsupervised algorithms are being developed to help physicians correctly grade disease. Most of the diagnostic accuracy literature deals with binary disease status (disease present or absent); however, tests diagnosing ordinal-scaled diseases should not be reduced to a binary status just to simplify diagnostic accuracy testing. In this paper, we propose different characterizations of ordinal-scale accuracy for different clinical use scenarios, along with methods for comparing tests. In the simplest scenario, just the proportion of correct grades is considered; other scenarios address the magnitude and direction of misgrading; and at the other extreme, a weighted accuracy measure with weights based on the relative costs of different types of misgrading is presented. The various scenarios are illustrated using a coronary artery disease example where the accuracy of AI algorithms in providing patients with the correct CAD-RADS grade is assessed.