Assessing the robustness of COVID-19 vaccine efficacy trials: systematic review and meta-analysis, January 2023

Abstract

Background Vaccines play a crucial role in the response to COVID-19 and their efficacy is thus of great importance. Aim To assess the robustness of COVID-19 vaccine efficacy (VE) trial results using the fragility index (FI) and fragility quotient (FQ) methodology. Methods We conducted a Cochrane and PRISMA-compliant systematic review and meta-analysis of COVID-19 VE trials published worldwide until 22 January 2023. We calculated the FI and FQ for all included studies and assessed their associations with selected trial characteristics using Wilcoxon rank sum tests and Kruskal–Wallis H tests. Spearman correlation coefficients and scatter plots were used to quantify the strength of correlation of FIs and FQs with trial characteristics. Results Of 6,032 screened records, we included 40 trials with 54 primary outcomes, comprising 909,404 participants with a median sample size per outcome of 13,993 (interquartile range (IQR): 8,534–25,519). The median FI and FQ was 62 (IQR: 22–123) and 0.50% (IQR: 0.24–0.92), respectively. FIs were positively associated with sample size (p < 0.001), and FQs were positively associated with type of blinding (p = 0.023). The Spearman correlation coefficient for FI with sample size was moderately strong (0.607), and weakly positive for FI and FQ with VE (0.138 and 0.161, respectively). Conclusions This was the largest study on trial robustness to date. Robustness of COVID-19 VE trials increased with sample size and varied considerably across several other important trial characteristics. The FI and FQ are valuable complementary parameters for the interpretation of trial results and should be reported alongside established trial outcome measures.