Developing the Interrater Reliability of the Modified EMDR Fidelity Checklist

Abstract

Although treatment fidelity measures for eye movement desensitization and reprocessing (EMDR) have been cited in past research, none have been subject to any empirical investigation of reliability. This three-phase study aimed to quantify the interrater reliability of a measure of EMDR treatment fidelity. First, two raters refined the reprocessing section of the EMDR Fidelity Checklist (Leeds, 2016) by developing a descriptive item-by-item scoring system to improve interpretation and reliability. The resultant checklist was piloted on recordings of five EMDR session recordings from the Laugharne et al. (2016) study. The checklist was then revised. Next, the raters used the checklist to assess 15 other recorded EMDR sessions from the same study. The intraclass correlations (ICCs) were in the excellent range for all subscales and total session scores (i.e., >0.75), with an exception of the Desensitization subscale, ICC = 0.69 (0.08, 0.90). Finally, individual items in that subscale were evaluated, finding that five items did not contribute to the ICC. When these were removed/revised, the ICC for this subscale moved into the excellent range, ICC = 0.81(0.43, 0.94). The findings of this study indicate that this checklist may be a reliable measure of treatment fidelity for single reprocessing EMDR sessions with the possible exception of the Body Scan phase. Future research using the checklist with raters who were not involved in checklist development is needed to confirm the generalizability of these findings.