Automated Calculator for the Pediatric Sequential Organ Failure Assessment Score: Development and External Validation in a Single-Center 7-Year Cohort, 2015–2021

Abstract

OBJECTIVES: The pediatric Sequential Organ Failure Assessment (pSOFA) score summarizes severity of organ dysfunction and can be used to predict in-hospital mortality. Manual calculation of the pSOFA score is time-consuming and prone to human error. An automated method that is open-source, flexible, and scalable for calculating the pSOFA score directly from electronic health record data is desirable. DESIGN: Single-center, retrospective cohort study. SETTING: Quaternary 40-bed PICU. PATIENTS: All patients admitted to the PICU between 2015 and 2021 with ICU stay of at least 24 hours. INTERVENTIONS: None. MEASUREMENTS AND MAIN RESULTS: We used 77 records to evaluate the automated score. The automated algorithm had an overall accuracy of 97%. The algorithm calculated the respiratory component of two cases incorrectly. An expert human annotator had an initial accuracy of 75% at the patient level and 95% at the component level. An untrained human annotator with general clinical research experience had an overall accuracy of 16% and component-wise accuracy of 67%. Weighted kappa for agreement between the automated method and the expert annotator’s initial score was 0.92 (95% CI, 0.88–0.95), and between the untrained human annotator and the automated score was 0.50 (95% CI, 0.36–0.61). Data from 9146 patients (in-hospital mortality 3.6%) were included to validate externally the discriminability of the automated pSOFA score. The admission-day pSOFA score had an area under the receiver operating characteristic curve of 0.79 (95% CI, 0.77–0.82). CONCLUSIONS: The developed automated algorithm calculates pSOFA score with high accuracy and is more accurate than a trained expert rater and nontrained data abstracter. pSOFA’s performance for predicting in-hospital mortality was lower in our cohort than it was for the originally derived score.