A re‐analysis of 150 women's health trials to investigate how the Bayesian approach may offer a solution to the misinterpretation of statistical findings

Abstract

AbstractObjectiveTo investigate whether a Bayesian interpretation might help prevent misinterpretation of statistical findings and support authors to differentiate evidence of no effect from statistical uncertainty.DesignA Bayesian re‐analysis to determine posterior probabilities of clinically important effects (e.g., a large effect is set at a 4 percentage point difference and a trivial effect to be within a 0.5 percentage point difference). Posterior probabilities greater than 95% are considered as strong statistical evidence, and less than 95% as inconclusive.Sample150 major women's health trials with binary outcomes.Main Outcome MeasuresPosterior probabilities of large, moderate, small and trivial effects.ResultsUnder frequentist methods, 48 (32%) were statistically significant (p‐value ≤ 0.05) and 102 (68%) statistically non‐significant. The frequentist and Bayesian point estimates and confidence intervals showed strong concordance. Of the statistically non‐significant trials (n = 102), the Bayesian approach classified the majority (94, 92%) as inconclusive, neither able to confirm or refute effectiveness. A small number of statistically non‐significant findings (8, 8%) were classified as having strong statistical evidence of an effect.ConclusionsWhilst almost all trials report confidence intervals, in practice most statistical findings are interpreted on the basis of statistical significance, mostly concluding evidence of no effect. Findings here suggest the majority are likely uncertain. A Bayesian approach could help differentiate evidence of no effect from statistical uncertainty.