An Ensemble approach for Ensemble-Modelled Cardiac Surgery Risk Evaluation, Data Usage and Clinical Interpretability

Abstract

Abstract Risk stratification plays a major role in the clinical decision-making process, patient consent and clinical governance analysis. However, the calibration of current risk scores (e.g., European System for Cardiac Operative Risk Evaluation (EuroSCORE), The Society of Thoracic Surgeons (STS) risk score) has been shown to deteriorate over time – a process known as calibration drift. The introduction of new clinical scores with different variable sets typically result in disparate datasets due to different levels of missingness. This is a barrier to the full insight and predictive capability of datasets across all potentially available time ranges. Little is known about the use of ensemble learning with ensemble metrics to mitigate the effects of calibration drift and changing risk across siloed datasets and time. In this study, we evaluated the effect of various combinations of Machine Learning (ML) models in improving model performance. The National Adult Cardiac Surgery Audit dataset was used (January 1996 to March 2019, 647,726 patients). We trained six different base learner models including Logistic Regression, Neuronetwork, Random Forest (RF), Weighted Support Vector Machine, Xgboost and Bayesian Update, based on two different variable sets of either Logistic EuroScore (LogES) or EuroScore II (ES II), partitioned by the time of score adoption (1996–2011 and 2012–2019). These base learner models are ensembled using nine different combinations to produce homogeneous or heterogeneous ensembles. Discrimination, calibration, clinical effectiveness and overall accuracy were assessed using an ensemble metric, referred to as clinical effectiveness metric (CEM). Xgboost homogenous ensemble (HE) was the highest performing model (CEM 0.725) with AUC (0.8327; 95% Confidence Interval (CI) 0.8323–0.8329) followed by Random Forest HE (CEM 0.723; AUC 0.8325; 95%CI 0.8320–0.8326). Across different heterogenous ensembles, significantly better performance was obtained by combining siloed datasets across time (CEM 0.720) than building ensembles of either 1996–2011 (t-test adjusted, p = 1.67e-6) or 2012–2019 (t-test adjusted, p = 1.35e-193) datasets alone. Both homogenous and heterogenous ML ensembles performed significantly better than traditional recalibration method (Bayesian Update). Combining the metrics covering all four aspects of discrimination, calibration, clinical usefulness and overall accuracy into a single ensemble metric improved the efficiency of cognitive decision-making. Xgboost/Random Forest homogenous ensembling and a highly heterogeneous ensemble approach showed high performance across multifaceted aspects of ML performance and were superior to traditional recalibration methods. Time-dependent ensemble combination of variables, having differing qualities according to time of score adoption, enabled previously siloed data to be combined, leading to increased power, clinical interpretability of variables and usage of data. For models to guide clinicians in individual decisions, performance exceeding these multifaceted benchmarks is necessary.