Performance assessment of the metastatic spinal tumor frailty index using machine learning algorithms: limitations and future directions

Abstract

OBJECTIVE Frailty is recognized as an important consideration in patients with cancer who are undergoing therapies, including spine surgery. The definition of frailty in the context of spinal metastases is unclear, and few have studied such markers and their association with postoperative outcomes and survival. Using national databases, the metastatic spinal tumor frailty index (MSTFI) was developed as a tool to predict outcomes in this specific patient population and has not been tested with external data. The purpose of this study was to test the performance of the MSTFI with institutional data and determine whether machine learning methods could better identify measures of frailty as predictors of outcomes. METHODS Electronic health record data from 479 adult patients admitted to the Massachusetts General Hospital for metastatic spinal tumor surgery from 2010 to 2019 formed a validation cohort for the MSTFI to predict major complications, in-hospital mortality, and length of stay (LOS). The 9 parameters of the MSTFI were modeled in 3 machine learning algorithms (lasso regularization logistic regression, random forest, and gradient-boosted decision tree) to assess clinical outcome prediction and determine variable importance. Prediction performance of the models was measured by computing areas under the receiver operating characteristic curve (AUROCs), calibration, and confusion matrix metrics (positive predictive value, sensitivity, and specificity) and was subjected to internal bootstrap validation. RESULTS Of 479 patients (median age 64 years [IQR 55–71 years]; 58.7% male), 28.4% had complications after spine surgery. The in-hospital mortality rate was 1.9%, and the mean LOS was 7.8 days. The MSTFI demonstrated poor discrimination for predicting complications (AUROC 0.56, 95% CI 0.50–0.62) and in-hospital mortality (AUROC 0.69, 95% CI 0.54–0.85) in the validation cohort. For postoperative complications, machine learning approaches showed a greater advantage over the logistic regression model used to develop the MSTFI (AUROC 0.62, 95% CI 0.56–0.68 for random forest vs AUROC 0.56, 95% CI 0.50–0.62 for logistic regression). The random forest model had the highest positive predictive value (0.53, 95% CI 0.43–0.64) and the highest negative predictive value (0.77, 95% CI 0.72–0.81), with chronic lung disease, coagulopathy, anemia, and malnutrition identified as the most important predictors of postoperative complications. CONCLUSIONS This study highlights the challenges of defining and quantifying frailty in the metastatic spine tumor population. Further study is required to improve the determination of surgical frailty in this specific cohort.