Defining and Evaluating Fidelity at the Program Level in Psychosocial Treatments

Abstract

One challenge to research on the implementation of effective psychosocial treatments is how to define and measure fidelity at the program level. The purpose of this study was to evaluate an approach to defining, measuring, and observing over time fidelity at the program level for Multisystemic Therapy (MST). For this study, program fidelity was conceptualized as a program’s performance on several key areas identified in the literature as contributing to sustainability. A composite index, the MST Program Performance Index (PPI), was developed using data from the MST quality assurance system. The PPI included indicators of treatment adherence, treatment completion, program operations, program capacity, clinical supervisor leadership, and stakeholder relationships. A PPI score was calculated for 496 MST teams every 6 months for a two year period, during which time, the teams served over 25,000 young people. The predictive validity of the PPI score was supported by both client- and team-level outcomes. Specifically, youth treated by teams with lower PPI scores were more likely to be rearrested during their course of treatment. Likewise, teams with lower PPI scores were more likely to close during the time period covered by the study. Analysis of scores over the two year period showed substantial within-team variability in the PPI; however, the scores did not follow a linear pattern of change. In summary, preliminary evidence suggests the PPI may be a useful tool to index program-level fidelity for comparative purposes and as an additional tool for a decision support system.