Best of intent, worst of both worlds: why sequentially combining epidemiological designs does not improve signal detection in vaccine safety surveillance

Abstract

AbstractBackgroundVaccine safety surveillance commonly includes a serial testing approach with a sensitive method for “signal generation” and specific method for “signal validation.” Whether serially combining epidemiological designs improves both sensitivity and specificity is unknown.MethodsWe assessed the overall performance of serial testing using three administrative claims and one electronic health record database. We compared Type I and II errors before and after empirical calibration for historical comparator, SCCS, and the serial combination of those designs against six vaccine exposure groups with 93 negative control and 279 imputed positive control outcomes.ResultsHistorical comparator mostly had lower Type II error than SCCS. SCCS had lower Type I error than the historical comparator. Before empirical calibration, serial combination increased specificity and decreased sensitivity. Type II errors mostly exceeded 50%. After empirical calibration, Type I errors returned to nominal; sensitivity was lowest when the methods were combined.ConclusionWe recommend against the serial approach in vaccine safety surveillance. While serial combination produced fewer false positive signals compared to the most specific method, it generated more false negative signals compared to the most sensitive method. Using the noisy historical comparator in front of SCCS deteriorated overall performance in evaluating safety signals.Key MessagesUsing the serial approach in vaccine safety surveillance did not improve overall performance: specificity increased but sensitivity decreased.Without empirical calibration, Type II errors exceeded 50%; after empirical calibration, Type I error rates returned to nominal with negligible change to Type II error rates.While prior research has suggested high sensitivity of the historical comparator method in distinguishing true safety signals, there were cases when self-controlled case series was more sensitive.Vaccine safety surveillance is becoming increasingly important, so monitoring systems should closely consider the utility and sequence of epidemiological designs.