Vibration of effect in more than 16 000 pooled analyses of individual participant data from 12 randomised controlled trials comparing canagliflozin and placebo for type 2 diabetes mellitus: multiverse analysis

Abstract

ObjectiveTo evaluate the impact of conducting all possible pooled analyses across different combinations of randomised controlled trials and endpoints.DesignMultiverse analysis, consisting of numerous pooled analyses of individual participant data.SettingIndividual patient data from 12 randomised controlled trials comparing canagliflozin treatment with placebo, shared on the Yale University Open Data Access project (https://yoda.yale.edu/) platform, up to 16 April 2021.Participants15 094 people with type 2 diabetes mellitus.Main outcome measuresPooled analyses estimated changes in serum glycated haemoglobin (HbA1c), major adverse cardiovascular events, and serious adverse events at weeks 12, 18, 26, and 52. The distribution of effect estimates was calculated for all possible combinations, and the direction and magnitude of the first and 99th centiles of effect estimates were compared.ResultsAcross 16 332 distinct pooled analyses comparing canagliflozin with placebo for changes in HbA1c, standardised effect estimates were in favour of canagliflozin treatment at both the first centile (−0.75%) and 99th centile (−0.48%); 15 994 (97.93%) analyses showed significant results (P<0.05) in favour of canagliflozin. For major adverse cardiovascular events, estimated hazard ratios were 0.20 at the first centile and 0.90 at the 99th centile; 2705 of 8144 analyses (33.21%) were significant, all of which were in favour of canagliflozin treatment. For serious adverse events, estimated hazard ratios were 0.59 at the first centile and 1.14 at the 99th centile; 5793 of 16 332 (35.47%) analyses were significant, with 5754 in favour of canagliflozin and 39 in favour of placebo.ConclusionResults from pooled analyses can be subject to vibration of effects and should be critically appraised, especially regarding the risk for selection and availability bias in individual participant data retrieved.