The Impact of Longitudinal Data‐Completeness of Electronic Health Record Data on the Prediction Performance of Clinical Risk Scores

Abstract

The impact of electronic health record (EHR) discontinuity (i.e., receiving care outside of a given EHR system) on EHR‐based risk prediction is unknown. We aimed to assess the impact of EHR‐continuity on the performance of clinical risk scores. The study cohort consisted of patients aged ≥ 65 years with ≥ 1 EHR encounter in the 2 networks in Massachusetts (MA; 2007/1/1–2017/12/31, internal training and validation dataset), and one network in North Carolina (NC; 2007/1/1–2016/12/31, external validation dataset) that were linked with Medicare claims data. Risk scores were calculated using EHR data alone vs. linked EHR‐claims data (not subject to misclassification due to EHR‐discontinuity): (i) combined comorbidity score (CCS), (ii) claim‐based frailty score (CFI), (iii) CHAD2DS2‐VASc, and (iv) Hypertension, Abnormal renal/liver function, Stroke, Bleeding, Labile, Elderly, and Drugs (HAS‐BLED). We assessed the performance of CCS and CFI predicting death, CHAD2DS2‐VASc predicting ischemic stroke, and HAS‐BLED predicting bleeding by area under receiver operating characteristic curve (AUROC), stratified by quartiles of predicted EHR‐continuity (Q1‐4). There were 319,740 patients in the MA systems and 125,380 in the NC system. In the external validation dataset, AUROC for EHR‐based CCS predicting 1‐year risk of death was 0.583 in Q1 (lowest) EHR‐continuity group, which increased to 0.739 in Q4 (highest) EHR‐continuity group. The corresponding improvement in AUROC was 0.539 to 0.647 for CFI, 0.556 to 0.637 for CHAD2DS2‐VASc, and 0.517 to 0.556 for HAS‐BLED. The AUROC in Q4 EHR‐continuity group based on EHR alone approximates that based on EHR‐claims data. The prediction performance of four clinical risk scores was substantially worse in patients with lower vs. high EHR‐continuity.