EVALUATION OF VGC ANALYZER BY COMPARISON WITH GOLD STANDARD ROC SOFTWARE AND ANALYSIS OF SIMULATED VISUAL GRADING DATA

Abstract

Abstract The purpose of the present work was to evaluate the use of resampling statistical methods for analysis of visual grading data—implemented in the software VGC Analyzer—by comparing the reanalyzed results from previously performed visual grading studies with the results calculated by gold standard receiver operating characteristic (ROC) methodology, Obuchowski-Rockette (OR)-Dorfman–Berbaum–Metz (DBM) multiple-readers and multiple-case (MRMC) and by analysis of simulated visual grading data where the true distribution was presumed to be known. The reanalysis was performed on two multiple-reader studies with non-paired data and paired data, respectively. The simulation study was performed by simulating a large number of visual grading characteristics (VGC) studies and by analyzing the statistical distribution of null hypothesis (H0) rejection rate. The comparison with OR-DBM MRMC showed good agreement when analyzing non-paired data for both fixed-reader and random-reader settings for the calculated area under the curve values and the confidence intervals (CIs). For paired data analysis, VGC Analyzer showed significantly lower CIs compared with the ROC software. This effect was also illustrated by the simulation study, where the VGC Analyzer, in general, showed good accuracy for simulated studies with stable statistical basis. For simulated studies with unstable statistics, the accuracy in the H0 rejection rate decreased. The present study has shown that resampling methodology can be used to accurately perform the statistical analysis of a VGC study, although the resampling technique used makes the method sensitive to small data sets.