Evaluation of diagnostic accuracy in free-response detection-localization tasks using ROC tools

Abstract

Diagnostic systems designed to detect possibly multiple lesions per patient (e.g. multiple polyps during CT colonoscopy) are often evaluated in “free-response” studies that allow for diagnostic responses unconstrained in their number and locations. Analysis of free-response studies requires extensions of the traditional receiver operating characteristic (ROC) analysis, which are termed free-response ROC (FROC) methodology. Despite substantial developments in this area, FROC tools and approaches are much more cumbersome than traditional ROC methods. Alternative approaches that use well-known ROC tools (e.g. ROI-ROC) require defining and physically delineating regions of interest (ROI) and combine FROC data within ROIs. We propose an approach that allows analyzing FROC data using conventional ROC tools without delineating the actual ROIs or reducing data. The design parameters of FROC study are used to make FROC data analyzable using ROC tools and to calibrate the corresponding FROC and ROC curves on both conceptual and numerical levels. Differences in the performance indices of the nonparametric FROC and the new approach are shown to be asymptotically negligible and typically rather small in practice. Data from a large multi-reader study of colon cancer detection are used to illustrate the new approach.