A Public Benchmark for Human Performance in FCD Detection

Abstract

This study aims to report human performance in the detection of Focal Cortical Dysplasias (FCDs), localized regions of malformed cerebral cortex, using a public dataset. Additionally, it defines a subset of this data as a representative testset to establish a baseline benchmark for the evaluation of automatic FCD detection approaches. The performance of 28 human readers was analyzed using 85 publicly available cases. Performance was measured based on the overlap between predicted regions of interest (ROIs) and ground truth lesion masks. The testset was chosen to consist of 15 subjects most predictive for human performance and 13 subjects identified by at most 3 readers. Expert readers achieved an average detection rate of 68%, compared to 45% for non-experts and 27% for laypersons. Neuroradiologists detected the highest percentage of lesions (64%), while psychiatrists detected the least (34%). Neurosurgeons had the highest ROI sensitivity (0.70), and psychiatrists had the highest ROI precision (0.78). In the testset, expert detection rate was 49%. Reporting human performance in FCD detection provides a baseline for assessing the effectiveness of automatic detection methods in a clinically relevant context. The representative testset will serve as an indicator for the clinical usefulness of computer-aided FCD detection approaches.