Reliability and reproducibility of clinical phenotypes developed during the first wave of COVID-19: A validation study in critically ill patients from the second and third wave

Abstract

Abstract Background: During the first wave of the COVID-19 pandemic, different clinical phenotypes were published. However, none of them have been validated in subsequent waves, so their current validity is unknown. The aim of the study is to validate the unsupervised cluster model developed during the first pandemic wave in a cohort of critically ill patients from the second and third pandemic waves. Methods: Retrospective, multicentre, observational study of critically ill patients with confirmed COVID-19 disease and acute respiratory failure admitted from 74 Intensive Care Units (ICU) in Spain. To validate our original phenotypes model, we assigned a phenotype to each patient of the validation cohort using the same medoids, the same number of clusters (n= 3), the same number of variables (n= 25) and the same discretisation used in the development cohort. The performance of the classification was determined by Silhouette analysis and general linear modelling. The prognostic models were validated, and their performance was measured using accuracy test and area under curve (AUC)ROC. Results: The database included a total of 2,033 patients (mean age 63[53-92] years, 1643(70.5%) male, median APACHE II score (12[9-16]) and SOFA score (4[3-6]) points. The ICU mortality rate was 27.2%. Although the application of unsupervised cluster analysis classified patients in the validation population into 3 clinical phenotypes. Phenotype A (n=1,206 patients, 59.3%), phenotype B (n=618 patients, 30.4%) and phenotype C (n=506 patients, 24.3%), the characteristics of patients within each phenotype were significantly different from the original population. Furthermore, the silhouette coefficients were close to or below zero and the inclusion of phenotype classification in a regression model did not improve the model performance (accuracy =0.78, AUC=0.78) with respect to a standard model (accuracy = 0.79, AUC=0.79) or even worsened when the model was applied to patients within each phenotype (accuracy = 0.80, AUC 0.77 for Phenotype A, accuracy=0.73, AUC= 0.67 for phenotype B and accuracy= 0.66 , AUC= 0.76 for phenotype C ) Conclusion: Models developed using machine learning techniques during the first pandemic wave cannot be applied with adequate performance to patients admitted in subsequent waves without prior validation. Trial Registration: The study was retrospectively registered (NCT 04948242) on June 30, 2021