The Effectiveness of the Capacity Building and Mentorship Program in Improving Evidence-Based Decision-making in the Amhara Region, Northwest Ethiopia: Difference-in-Differences Study

Abstract

Background Weak health information systems (HISs) hobble countries’ abilities to effectively manage and distribute their resources to match the burden of disease. The Capacity Building and Mentorship Program (CBMP) was implemented in select districts of the Amhara region of Ethiopia to improve HIS performance; however, evidence about the effectiveness of the intervention was meager. Objective This study aimed to determine the effectiveness of routine health information use for evidence-based decision-making among health facility and department heads in the Amhara region, Northwest Ethiopia. Methods The study was conducted in 10 districts of the Amhara region: five were in the intervention group and five were in the comparison group. We employed a quasi-experimental study design in the form of a pretest-posttest comparison group. Data were collected from June to July 2020 from the heads of departments and facilities in 36 intervention and 43 comparison facilities. The sample size was calculated using the double population formula, and we recruited 172 participants from each group. We applied a difference-in-differences analysis approach to determine the effectiveness of the intervention. Heterogeneity of program effect among subgroups was assessed using a triple differences method (ie, difference-in-difference-in-differences [DIDID] method). Thus, the β coefficients, 95% CIs, and P values were calculated for each parameter, and we determined that the program was effective if the interaction term was significant at P<.05. Results Data were collected using the endpoint survey from 155 out of 172 (90.1%) participants in the intervention group and 166 out of 172 (96.5%) participants in the comparison group. The average level of information use for the comparison group was 37.3% (95% CI 31.1%-43.6%) at baseline and 43.7% (95% CI 37.9%-49.5%) at study endpoint. The average level of information use for the intervention group was 52.2% (95% CI 46.2%-58.3%) at baseline and 75.8% (95% CI 71.6%-80.0%) at study endpoint. The study indicated that the net program change over time was 17% (95% CI 5%-28%; P=.003). The subgroup analysis also indicated that location showed significant program effect heterogeneity, with a DIDID estimate equal to 0.16 (95% CI 0.026-0.29; P=.02). However, sex, age, educational level, salary, and experience did not show significant heterogeneity in program effect, with DIDID estimates of 0.046 (95% CI –0.089 to 0.182), –0.002 (95% CI –0.015 to 0.009), –0.055 (95% CI –0.190 to 0.079), –1.63 (95% CI –5.22 to 1.95), and –0.006 (95% CI –0.017 to 0.005), respectively. Conclusions The CBMP was effective at enhancing the capacity of study participants in using the routine HIS for decision-making. We noted that urban facilities had benefited more than their counterparts. The intervention has been shown to produce positive outcomes and should be scaled up to be used in other districts. Moreover, the mentorship modalities for rural facilities should be redesigned to maximize the benefits. Trial Registration Pan African Clinical Trials Registry PACTR202001559723931; https://tinyurl.com/3j7e5ka5