AI learns racial information from the values of vital signs

Abstract

AbstractBackgroundBias in medical practice is multifaceted, including treatment variations across race-ethnicity, unconscious bias in healthcare providers’ attitudes, and bias in clinical scores. However, far less is known about the potential racial bias in routinely collected, essential information in clinical decision-making, namely vital signs.Research questionDo vital signs embed racial information that can be learned by AI algorithms?Study Design and MethodsRetrospective cohort study of critically ill patients between 2014 and 2015 from the multi-centre eICU-CRD critical care database involving 335 Intensive Care Units (ICU) based in 208 US hospitals, containing 200,859 patient admissions. We extracted 10,763 critical care admissions of patients aged 18 and over, alive during the first 24 hours after admission to ICU with recorded self-reported race as well as at least two measurement of heart rate, oxygen saturation, respiratory rate, and blood pressure. Pairs of racial subgroups were matched based on age, gender, admission diagnosis and APACHE IV scores. Traditional machine learning algorithms, including XGBoost and Logistic regression were used to predict self-reported race using values of vital signs as an input.ResultsAI models derived from only six vital signs can predict patients’ self-reported race with an AUC of 0.74 (± 0.022) between White and Black patients. Technologies used to measure oxygen saturation are a significant source of self-reported racial information (AUC of 0.72 ± 0.028), in addition to blood pressure measurements (AUC of 0.63 ± 0.035). Care delivery practices do not present a significant source of racial information (AUC of 0.57 ± 0.019). However, even when controlling for these known factors, self-reported race can still be learned from vital signs, whose origin we cannot currently explain.InterpretationVital signs embed racial information that can be learned by AI algorithms, posing a significant risk to equitable clinical decision-making. Mitigating measures might be challenging, considering fundamental role of vital signs.