Comparison of statistical methods used to meta-analyse results from interrupted time series studies: an empirical study

Abstract

Abstract Background The Interrupted Time Series (ITS) is a robust design for evaluating public health and policy interventions or exposures when randomisation is infeasible. Several statistical methods are available for the analysis and meta-analysis of ITS studies. We sought to empirically compare available methods when applied to real-world ITS data. Methods We sourced ITS data from published meta-analyses to create an online data repository. Each dataset was re-analysed using two ITS estimation methods. The level- and slope-change effect estimates (and standard errors) were calculated and combined using fixed-effect and four random-effects meta-analysis methods. We examined differences in meta-analytic level- and slope-change estimates, their 95% confidence intervals, p-values, and estimates of heterogeneity across the statistical methods. Results Of 40 eligible meta-analyses, data from 17 meta-analyses including 283 ITS studies were obtained and analysed. We found that on average, the meta-analytic effect estimates, their standard errors and between-study variances were not sensitive to meta-analysis method choice, irrespective of the ITS analysis method. However, confidence interval widths and p-values for the meta-analytic effect estimates varied depending on the choice of confidence interval method and ITS analysis method. Conclusions The meta-analysis effect estimates, their standard errors and between-study variance estimates were minimally impacted by ITS analysis and meta-analysis method choice. However, the confidence interval widths and p-values could vary according to the statistical method, which may impact interpretations and conclusions of a meta-analysis. This empirical study, in conjunction with evidence from numerical simulation, allows for a more complete understanding of which methods should be used in different scenarios.