Understanding the Influence of Patient Factors on Accuracy and Decision-Making in a Diagnostic Accuracy Study with Multiple Raters—A Case Study from Dentistry

Abstract

In diagnostic accuracy studies, the test of interest is typically applied only once in each patient. This paper illustrates some possibilities that arise when diagnoses are carried out by a sufficiently large number of multiple raters. In a dental study, sixty-one examiners were asked to diagnose 49 tooth areas with different grades of tissue loss (minor, moderate, and advanced) to decide whether dentine was exposed (positive status) or not (negative status). The true status was determined by histology (reference). For each tooth, the rate of correct decisions reflecting the difficulty to diagnose this tooth and the positive rate reflecting the perception of the tooth by the raters was computed. Meta-analytical techniques were used to assess the inter-tooth variation and the influence of tooth-specific factors on difficulty or perception, respectively. A huge variation in diagnostic difficulty and perception could be observed. Advanced tissue loss made diagnoses more difficult. The background colour and tissue loss were associated with perception and may hint to cues used by the raters. The use of multiple raters in a diagnostic accuracy study allows detailed investigations which make it possible to obtain further insights into the decision-making process of the raters.