The Fragility of Statistical Significance in Sham Orthopaedic Surgery: A Systematic Review of Randomized Controlled Trials

Abstract

Objectives: The purpose of this study was to determine the stability of statistical findings among sham surgery randomized controlled trials (RCTs) in orthopaedic surgery using fragility analysis. Methods: PubMed systematic review was conducted to include studies reporting dichotomous outcomes pertaining to sham surgery. The final review included eight RCTs involving only partial meniscectomies and vertebroplasties from 2009 to 2020. With a fixed sample size with dichotomous outcome measures (events versus non-events), the Total Fragility Index (TFI), which is composed of the fragility index (FI) and reverse fragility index (RFI), was calculated by altering the ratio of events to non-events in an iterative fashion until results were reversed from significant to nonsignificant findings (FI) or vice versa (RFI). The TFI, FI, and RFI were divided by their sample sizes to obtain the respective total fragility quotient, fragility quotient (FQ), and reverse fragility quotient. Median fragility indices and quotients were reported for all studies. Results: The eight RCTs included 50 dichotomous outcomes involving either partial meniscectomies or vertebroplasties, with a median TFI and total fragility quotient of 5 [interquartile range (IQR) 4 to 6] and 0.035 (IQR 0.028 to 0.048), respectively, indicating that a median of five total patients or 3.5 per 100 patients would need to experience a different outcome to reverse significant or insignificant findings for each of the eight trials. Among the 8 statistically significant (P < 0.05) outcome events (16%), the respective FI and FQ were 2 (IQR 1 to 5) and 0.018 (IQR 0.010 to 0.044). Among the 42 statistically insignificant outcome events (84%), the respective RFI and reverse fragility quotient were 5 (IQR 4 to 6) and 0.04 (IQR 0.034 to 0.048). The median number of patients lost to follow-up was 1.5 (IQR 0.5 to 2). Conclusion: The unstable findings in partial meniscectomy and vertebroplasty sham surgical RCTs undermine their study conclusions and recommendations. We recommend using fragility analysis in future sham surgical RCTs to contextualize statistical findings. Level of evidence: Level IV; Systematic Review.