Performance Characteristics of a Machine-Learning Tool to Predict 7-Day Hospital Readmissions

Abstract

OBJECTIVES To develop an institutional machine-learning (ML) tool that utilizes demographic, socioeconomic, and medical information to stratify risk for 7-day readmission after hospital discharge; assess the validity and reliability of the tool; and demonstrate its discriminatory capacity to predict readmissions. PATIENTS AND METHODS We performed a combined single-center, cross-sectional, and prospective study of pediatric hospitalists assessing the face and content validity of the developed readmission ML tool. The cross-sectional analyses used data from questionnaire Likert scale responses regarding face and content validity. Prospectively, we compared the discriminatory capacity of provider readmission risk versus the ML tool to predict 7-day readmissions assessed via area under the receiver operating characteristic curve analyses. RESULTS Overall, 80% (15 of 20) of hospitalists reported being somewhat to very confident with their ability to accurately predict readmission risk; 53% reported that an ML tool would influence clinical decision-making (face validity). The ML tool variable exhibiting the highest content validity was history of previous 7-day readmission. Prospective provider assessment of risk of 413 discharges showed minimal agreement with the ML tool (κ = 0.104 [95% confidence interval 0.028–0.179]). Both provider gestalt and ML calculations poorly predicted 7-day readmissions (area under the receiver operating characteristic curve: 0.67 vs 0.52; P = .11). CONCLUSIONS An ML tool for predicting 7-day hospital readmissions after discharge from the general pediatric ward had limited face and content validity among pediatric hospitalists. Both provider and ML-based determinations of readmission risk were of limited discriminatory value. Before incorporating similar tools into real-time discharge planning, model calibration efforts are needed.