Statistical significance and publication reporting bias in abstracts of reproductive medicine studies

Abstract

Abstract STUDY QUESTION What were the frequency and temporal trends of reporting P-values and effect measures in the abstracts of reproductive medicine studies in 1990–2022, how were reported P-values distributed, and what proportion of articles that present with statistical inference reported statistically significant results, i.e. ‘positive’ results? SUMMARY ANSWER Around one in six abstracts reported P-values alone without effect measures, while the prevalence of effect measures, whether reported alone or accompanied by P-values, has been increasing, especially in meta-analyses and randomized controlled trials (RCTs); the reported P-values were frequently observed around certain cut-off values, notably at 0.001, 0.01, or 0.05, and among abstracts present with statistical inference (i.e. P-value, CIs, or significant terms), a large majority (77%) reported at least one statistically significant finding. WHAT IS KNOWN ALREADY Publishing or reporting only results that show a ‘positive’ finding causes bias in evaluating interventions and risk factors and may incur adverse health outcomes for patients.   Despite efforts to minimize publication reporting bias in medical research, it remains unclear whether the magnitude and patterns of the bias have changed over time. STUDY DESIGN, SIZE, DURATION We studied abstracts of reproductive medicine studies from 1990 to 2022. The reproductive medicine studies were published in 23 first-quartile journals under the category of Obstetrics and Gynaecology and Reproductive Biology in Journal Citation Reports and 5 high-impact general medical journals (The Journal of the American Medical Association, The Lancet, The BMJ, The New England Journal of Medicine, and PLoS Medicine). Articles without abstracts, animal studies, and non-research articles, such as case reports or guidelines, were excluded. PARTICIPANTS/MATERIALS, SETTING, METHODS Automated text-mining was used to extract three types of statistical significance reporting, including P-values, CIs, and text description. Meanwhile, abstracts were text-mined for the presence of effect size metrics and Bayes factors. Five hundred abstracts were randomly selected and manually checked for the accuracy of automatic text extraction. The extracted statistical significance information was then analysed for temporal trends and distribution in general as well as in subgroups of study designs and journals. MAIN RESULTS AND THE ROLE OF CHANCE A total of 24 907 eligible reproductive medicine articles were identified from 170 739 screened articles published in 28 journals. The proportion of abstracts not reporting any statistical significance inference halved from 81% (95% CI, 76–84%) in 1990 to 40% (95% CI, 38–44%) in 2021, while reporting P-values alone remained relatively stable, at 15% (95% CI, 12–18%) in 1990 and 19% (95% CI, 16–22%) in 2021. By contrast, the proportion of abstracts reporting effect measures alone increased considerably from 4.1% (95% CI, 2.6–6.3%) in 1990 to 26% (95% CI, 23–29%) in 2021. Similarly, the proportion of abstracts reporting effect measures together with P-values showed substantial growth from 0.8% (95% CI, 0.3–2.2%) to 14% (95% CI, 12–17%) during the same timeframe. Of 30 182 statistical significance inferences, 56% (n = 17 077) conveyed statistical inferences via P-values alone, 30% (n = 8945) via text description alone such as significant or non-significant, 9.3% (n = 2820) via CIs alone, and 4.7% (n = 1340) via both CI and P-values. The reported P-values (n = 18 417), including both a continuum of P-values and dichotomized P-values, were frequently observed around common cut-off values such as 0.001 (20%), 0.05 (16%), and 0.01 (10%). Of the 13 200 reproductive medicine abstracts containing at least one statistical inference, 77% of abstracts made at least one statistically significant statement. Among articles that reported statistical inference, a decline in the proportion of making at least one statistically significant inference was only seen in RCTs, dropping from 71% (95% CI, 48–88%) in 1990 to 59% (95% CI, 42–73%) in 2021, whereas the proportion in the rest of study types remained almost constant over the years. Of abstracts that reported P-value, 87% (95% CI, 86–88%) reported at least one statistically significant P-value; it was 92% (95% CI, 82–97%) in 1990 and reached its peak at 97% (95% CI, 93–99%) in 2001 before declining to 81% (95% CI, 76–85%) in 2021. LIMITATIONS, REASONS FOR CAUTION First, our analysis focused solely on reporting patterns in abstracts but not full-text papers; however, in principle, abstracts should include condensed impartial information and avoid selective reporting. Second, while we attempted to identify all types of statistical significance reporting, our text mining was not flawless. However, the manual assessment showed that inaccuracies were not frequent. WIDER IMPLICATIONS OF THE FINDINGS There is a welcome trend that effect measures are increasingly reported in the abstracts of reproductive medicine studies, specifically in RCTs and meta-analyses. Publication reporting bias remains a major concern. Inflated estimates of interventions and risk factors could harm decisions built upon biased evidence, including clinical recommendations and planning of future research. STUDY FUNDING/COMPETING INTEREST(S) No funding was received for this study. B.W.M. is supported by an NHMRC Investigator grant (GNT1176437); B.W.M. reports research grants and travel support from Merck and consultancy from Merch and ObsEva. W.L. is supported by an NHMRC Investigator Grant (GNT2016729). Q.F. reports receiving a PhD scholarship from Merck. The other author has no conflict of interest to declare. TRIAL REGISTRATION NUMBER N/A.