Predicting Opportunities for Improvement in Trauma Care Using Machine Learning

Abstract

2 Abstract2.1ImportanceIdentifying opportunities for improvement (OFI), errors in care with adverse outcomes, through mortality and morbidity conferences is essential for improving trauma quality. To screen patients for such conferences, trauma quality improvement programs rely on labor-intensive human reviews and audit filters that exhibit high false positive rates.2.2ObjectiveThis study was conducted to develop machine learning models that predicts OFI in trauma care and compare the performances of these models to those of commonly used audit filters.2.3DesignIn this registry-based cohort study, we developed eight binary classification models using different machine learning methods with 17 predictors. Development used data from 2013 to 2022, and performance was measured between 2017 and 2022 using a add-one-year-in expanding window approach. We used two calibration strategies: 95% sensitivity (High sensitivity) and optimizing the area under the curve (Balanced). A bootstrap estimated confidence intervals.2.4SettingThe setting is a level one equivalent trauma center with bimonthly mortality and morbidity conferences for identifying OFIs; a combination of human review of individual patient cases and audit filters is used to screen patients for these conferences.2.5ParticipantsA total of 8220 adult trauma patients were screened for OFI. All patients prompted trauma team activation or were later found to have an injury severity score greater than 9.2.6Main outcome measuresOutcome measures were the models and audit filter performances, measured as discrimination, calibration, true positive and false positive rates.2.7ResultsOFI were identified in 496 (6%) patients. The best performing model was XGBoost (High sensitivity: [auc:0.75, sens:0.904, FPR: 0.599], and Balanced: [auc:0.75, sens:0.502, FPR: 0.186]) followed by Random Forest (High sensitivity: auc:0.733, sens:0.888, FPR: 0.617), and Balanced: [auc:0.733, sens:0.519, FPR: 0.222]). All machine learning models showed higher AUC and lower FPRs compared to Audit filters (auc:0.616, sens:0.903, FPR: 0.671).2.8Conclusion and RelevanceMachine learning models generally outperformed audit filters in predicting OFI among adult trauma patients, balancing and reducing overall screening burden for trauma quality improvement programs while potentially identifying new OFI types.1Key pointQuestion:How does the performance of machine learning models compare to that of audit filters when screening for opportunities for improvement (OFI), errors in care with adverse outcomes, among adult trauma patients?Findings:Our registry-based cohort study including 8,220 patients showed that machine learning models outperform audit filters, exhibiting greater area under the curve values and reduced false-positive rates. Compared to audit filters, these models can be calibrated to balance sensitivity against overall screening burden.Meaning:Machine learning models have the potential to reduce false positives when screening for OFI in adult trauma patients and thereby enhancing trauma quality programs.