Comparing the predictive ability of a commercial artificial intelligence early warning system with physician judgement for clinical deterioration in hospitalised general internal medicine patients: a prospective observational study

Abstract

ObjectiveOur study compares physician judgement with an automated early warning system (EWS) for predicting clinical deterioration of hospitalised general internal medicine patients.DesignProspective observational study of clinical predictions made at the end of the daytime work-shift for an academic general internal medicine floor team compared with the risk assessment from an automated EWS collected at the same time.SettingInternal medicine teaching wards at a single tertiary care academic medical centre in the USA.ParticipantsIntern physicians working on the internal medicine wards and an automated EWS (Rothman Index by PeraHealth).OutcomeClinical deterioration within 24 hours including cardiac or pulmonary arrest, rapid response team activation or unscheduled intensive care unit transfer.ResultsWe collected predictions for 1874 patient days and saw 35 clinical deteriorations (1.9%). The area under the receiver operating curve (AUROC) for the EWS was 0.73 vs 0.70 for physicians (p=0.571). A linear regression model combining physician and EWS predictions had an AUROC of 0.75, outperforming physicians (p=0.016) and the EWS (p=0.05).ConclusionsThere is no significant difference in the performance of the EWS and physicians in predicting clinical deterioration at 24 hours on an inpatient general medicine ward. A combined model outperformed either alone. The EWS and physicians identify partially overlapping sets of at-risk patients suggesting they rely on different cues or decision rules for their predictions.Trial registration numberNCT02648828.