Sources of Variability in Estimates of Predictive Validity

Abstract

Level of Service (LS) is one of the most widely used general risk and need assessment tools in criminal justice agencies across North America. However, there is significant interstudy variability in the magnitude of the validity estimates. This study was conducted to examine possible sources of this variability. The predictive validity of LS risk and need increased with length of follow-up period and with investigator allegiance to LS. The combination of these two variables reveals consistent increases in mean predictive validity estimates from modest (in the .20s) through large (in the mid .30s) to very large (in the .40s) in samples of both male and female offenders. We hypothesized that the “allegiance effect” reflects the integrity of LS implementation and support provided by the agency for risk assessment. This is akin to the difference between “demonstration projects” and “practical” rehabilitation programming in the offender treatment research. Moreover, controls for Canadian versus non-Canadian evaluations reduced the effect of allegiance and length of follow-up to nonsignificant levels. Possible explanations for these findings include the degree of integrity in conducting risk and need assessments, the accuracy of recidivism as the criterion measure, and generalizability across international boundaries.