External validation of a deep-learning mandibular ORN prediction model trained on 3D radiation distribution maps

Abstract

AbstractBackground and purposeMandibular osteoradionecrosis (ORN) is a severe side effect affecting patients undergoing radiation therapy for head and neck cancer. Variations in the bone’s vascularization and composition across the mandible may influence the susceptibility to ORN. Recently, deep learning-based models have been introduced for predicting mandibular ORN using radiation dose distribution maps to incorporate spatial information. These studies, however, only feature internal validation on a holdout subset of the data used for training.Materials and methodsThis study externally validated a 3D DenseNet-40 (DN40) ORN prediction model on an independent dataset. Model performance was evaluated in terms of discrimination and calibration, with Platt scaling applied for improved external calibration. The DN40 model’s discriminative ability on the external dataset was compared to a Random Forest model on corresponding dose-volume histogram (DVH) data.ResultsThe overall model performance was worse at external validation than at internal validation, with Platt scaling improving balance between recall and specificity but not significantly improving the overall calibration. Although the discrimination ability of the DN40 model was slightly lower at external validation (AUROC 0.63 vs. 0.69), this was statistically comparable to that of a DVH-based RF model for the same dataset (p-value 0.667).ConclusionsOur results suggest that, in addition to potential model overfitting issues, dosimetric data distribution differences between the two datasets could explain the low generalisability of the DN40 ORN prediction model. Future work will involve a larger and more diverse cohort.