Disagreements in risk of bias assessment for randomized controlled trials in hypertension-related Cochrane reviews

Abstract

Abstract Background The inter-reviewer reliability of the risk of bias (RoB) assessment lacked agreement in previous studies. It is important to analyse these disagreements to improve the repeatability of RoB assessment. The objective of the study was to evaluate the frequency and reasons for disagreements in RoB assessments for randomised controlled trials (RCTs) that were included in multiple Cochrane reviews in the field of hypertension. Methods A cross-sectional study was employed. We retrieved any RCTs that had been included in multiple Cochrane reviews in the field of hypertension from ARCHIE. The results of the RoB assessments were extracted, and the distributions of agreements and possible reasons for disagreement were analyzed. Results Twenty-six Cochrane reviews were included in this study. A total of 78 RCTs appeared in more than one Cochrane review. The level of agreement ranged from domain to domain. “Blinding of outcome assessment” showed a reasonably high level of agreement (94.9%), while “incomplete outcome data”, “selective outcome reporting” and “other sources of bias” showed moderate levels of agreement (74.6%, 79.2% and 75.6%, respectively). However, the domains of “allocation concealment”, “random sequence generation” and “blinding of participants and personnel” showed low levels of agreement (24.4%, 23.5%, and 47.4%, respectively). In the domains of “allocation concealment” and “blinding of participants and personnel”, the agreement group had higher proportion of publication year ≤ 1996 than the disagreement group (P = 0.008 and P < 0.001, respectively). In the “blinding of participants and personnel”, the impact factor was higher in the agreement group (P < 0.001). By analyzing the support text, we found that the most likely reason for disagreement was extracting different information from the same RCT. Conclusion For Cochrane reviews in the field of hypertension using the 2011 version of the RoB tool, there was a large disagreement in the RoB assessment. It is suggested that the results of RoB assessments in systematic reviews that used the 2011 version of the RoB tool need to be interpreted with caution. More accurate information from RCTs needs to be collected when we synthesize clinical evidence.