The Measurement of Performance in Probabilistic Diagnosis

Abstract

Attention is focused on one important aspect of good performance in probabilistic diagnosis, the »reliability« (external validity) of probabilistic assertions: a diagnostic alternative claimed to be 90% certain, say, must occur neither more nor less than nine times out of ten on the average. Statistical measures are offered by which departures from such perfect reliability can be estimated, and statistical tests are developed in order to test the hypothesis of perfect reliability. The specific reliability defects looked for include overconfident diagnoses and so-called size bias (common diseases being over diagnosed). Reliability is contrasted with discriminatory power and other performance aspects. The illustrative data derive from a study of computer-aided diagnosis of the acute abdomen.