Developing and validating a pancreatic cancer risk model for the general population using multi-institutional electronic health records from a federated network

Abstract

AbstractPurposePancreatic Duct Adenocarcinoma (PDAC) screening can enable detection of early-stage disease and long-term survival. Current guidelines are based on inherited predisposition; only about 10% of PDAC cases meet screening eligibility criteria. Electronic Health Record (EHR) risk models for the general population hold out the promise of identifying a high-risk cohort to expand the currently screened population. Using EHR data from a multi-institutional federated network, we developed and validated a PDAC risk prediction model for the general US population.MethodsWe developed Neural Network (NN) and Logistic Regression (LR) models on structured, routinely collected EHR data from 55 US Health Care Organizations (HCOs). Our models used sex, age, frequency of clinical encounters, diagnoses, lab tests, and medications, to predict PDAC risk 6-18 months before diagnosis. Model performance was assessed using Receiver Operating Characteristic (ROC) curves and calibration plots. Models were externally validated using location, race, and temporal validation, with performance assessed using Area Under the Curve (AUC). We further simulated model deployment, evaluating sensitivity, specificity, Positive Predictive Value (PPV) and Standardized Incidence Ratio (SIR). We calculated SIR based on the SEER data of the general population with matched demographics.ResultsThe final dataset included 63,884 PDAC cases and 3,604,863 controls between the ages 40 and 97.4 years. Our best performing NN model obtained an AUC of 0.829 (95% CI: 0.821 to 0.837) on the test set. Calibration plots showed good agreement between predicted and observed risks. Race-based external validation (trained on four races, tested on the fifth) AUCs of NN were 0.836 (95% CI: 0.797 to 0.874), 0.838 (95% CI: 0.821 to 0.855), 0.824 (95% CI: 0.819 to 0.830), 0.842 (95% CI: 0.750 to 0.934), and 0.774 (95% CI: 0.771 to 0.777) for AIAN, Asian, Black, NHPI, and White, respectively. Location-based external validation (trained on three locations, tested on the fourth) AUCs of NN were 0.751 (95% CI: 0.746 to 0.757), 0.749 (95% CI: 0.745 to 0.753), 0.752 (95% CI: 0.748 to 0.756), and 0.722 (95% CI: 0.713 to 0.732) for Midwest, Northeast, South, and West, respectively. Average temporal external validation (trained on data prior to certain dates, tested on data after a date) AUC of NN was 0.784 (95% CI: 0.763 to 0.805). Simulated deployment on the test set, with a mean follow up of 2.00 (SD 0.39) years, demonstrated an SIR range between 2.42-83.5 for NN, depending on the chosen risk threshold. At an SIR of 5.44, which exceeds the current threshold for inclusion into PDAC screening programs, NN sensitivity was 35.5% (specificity 95.6%), which is 3.5 times the sensitivity of those currently being screened with an inherited predisposition to PDAC. At a chosen high-risk threshold with a lower SIR, specificity was about 85%, and both models exhibited sensitivities above 50%.ConclusionsOur models demonstrate good accuracy and generalizability across populations from diverse geographic locations, races, and over time. At comparable risk levels these models can predict up to three times as many PDAC cases as current screening guidelines. These models can therefore be used to identify high-risk individuals, overlooked by current guidelines, who may benefit from PDAC screening or inclusion in an enriched group for further testing such as biomarker testing. Our integration with the federated network provided access to data from a large, geographically and racially diverse patient population as well as a pathway to future clinical deployment.