Multiclass risk models for ovarian malignancy: an illustration of prediction uncertainty due to the choice of algorithm

Abstract

ABSTRACTOBJECTIVETo compare performance and probability estimates of six algorithms to estimate the probabilities that an ovarian tumor is benign, borderline malignant, stage I primary invasive, stage II-IV primary invasive, or secondary metastatic.MATERIALS AND METHODSModels were developed on 5909 patients (recruited 1999-2012) and validated on 3199 patients (2012-2015). Nine clinical and ultrasound predictors were used. Outcome was based on histology following surgery within 120 days after the ultrasound examination. We developed models using multinomial logistic regression (MLR), Ridge MLR, random forest (RF), XGBoost, neural networks (NN), and support vector machines (SVM).RESULTSBenign tumors were most common (62%), secondary metastatic tumors least common (5%). XGBoost, RF, NN and MLR had similar performance: c-statistics for benign versus any type of malignant tumors were 0.92, multiclass c-statistics 0.54-0.55, average Estimated Calibration Indexes 0.03-0.07, and Net Benefits at the 10% malignancy risk threshold 0.33-0.34. Despite poorer discrimination and calibration performance for Ridge MLR and in particular SVM, Net Benefits were similar for all models. The estimated probabilities often differed strongly between models. For example, the probability of a benign tumor differed by more than 20 percentage points in 29% of the patients, and by more than 30 percentage points in 16% of the patients.DISCUSSIONSeveral regression and machine learning models had very good and similar performance in terms of discrimination, calibration and clinical utility. Nevertheless, individual probabilities often varied substantially.CONCLUSIONMachine learning did not outperform MLR. The choice of algorithm can strongly affect probabilities given to a patient.