Testing an informatics consulting service for systematic bias using negative control reference sets

Abstract

Abstract Background We used negative control reference sets to estimate whether automated statistical methods can lead to unbiased effect estimates in the context of a clinical informatics consult. Methods We used clinical data from two national databases and one regional academic medical center. We used treatment-comparator-outcome triads defined by the Observational Health Data Sciences and Informatics network as negative control reference sets. For each set, we estimated the hazard ratio for the outcome between populations exposed to treatment vs. comparator medication in each dataset via a new-user cohort design. We estimated hazard ratios both unadjusted and adjusted via demographic and propensity score matching. Results Unadjusted estimates showed systematic bias in all three databases, with expected absolute systematic error (EASE) up to 0.19. In contrast, bias was minimal after propensity score adjustment (EASE range, -0.04 to 0.04) and propensity score matching yielded low mean squared error. After empirical calibration, the false positive rates were as expected (type one error rate of close to 0.05). Conclusions Data-driven propensity score matching has been shown to produce estimates consistent with manual confounder adjustment, but it is not known whether such methods are consistent with true population values. Through the use of negative controls, where the true association is known to be null, we have shown that automated confounder adjustment can produce estimates that are free of systematic bias in the context of clinical informatics consulting.