Which supervised machine learning algorithm can best predict achievement of minimum clinically important difference in neck pain after surgery in patients with cervical myelopathy? A QOD study

Abstract

OBJECTIVE The purpose of this study was to evaluate the performance of different supervised machine learning algorithms to predict achievement of minimum clinically important difference (MCID) in neck pain after surgery in patients with cervical spondylotic myelopathy (CSM). METHODS This was a retrospective analysis of the prospective Quality Outcomes Database CSM cohort. The data set was divided into an 80% training and a 20% test set. Various supervised learning algorithms (including logistic regression, support vector machine, decision tree, random forest, extra trees, gaussian naïve Bayes, k–nearest neighbors, multilayer perceptron, and extreme gradient boosted trees) were evaluated on their performance to predict achievement of MCID in neck pain at 3 and 24 months after surgery, given a set of predicting baseline features. Model performance was assessed with accuracy, F1 score, area under the receiver operating characteristic curve, precision, recall/sensitivity, and specificity. RESULTS In total, 535 patients (46.9%) achieved MCID for neck pain at 3 months and 569 patients (49.9%) achieved it at 24 months. In each follow-up cohort, 501 patients (93.6%) were satisfied at 3 months after surgery and 569 patients (100%) were satisfied at 24 months after surgery. Of the supervised machine learning algorithms tested, logistic regression demonstrated the best accuracy (3 months: 0.76 ± 0.031, 24 months: 0.773 ± 0.044), followed by F1 score (3 months: 0.759 ± 0.019, 24 months: 0.777 ± 0.039) and area under the receiver operating characteristic curve (3 months: 0.762 ± 0.027, 24 months: 0.773 ± 0.043) at predicting achievement of MCID for neck pain at both follow-up time points, with fair performance. The best precision was also demonstrated by logistic regression at 3 (0.724 ± 0.058) and 24 (0.780 ± 0.097) months. The best recall/sensitivity was demonstrated by multilayer perceptron at 3 months (0.841 ± 0.094) and by extra trees at 24 months (0.817 ± 0.115). Highest specificity was shown by support vector machine at 3 months (0.952 ± 0.013) and by logistic regression at 24 months (0.747 ± 0.18). CONCLUSIONS Appropriate selection of models for studies should be based on the strengths of each model and the aims of the studies. For maximally predicting true achievement of MCID in neck pain, of all the predictions in this balanced data set the appropriate metric for the authors’ study was precision. For both short- and long-term follow-ups, logistic regression demonstrated the highest precision of all models tested. Logistic regression performed consistently the best of all models tested and remains a powerful model for clinical classification tasks.