From noise to knowledge: Improving evidentiary standards for program efficacy to better inform public policy and management decisions

Abstract

AbstractCurrent approaches employed by U.S.‐based clearinghouses to rate the efficacy of interventions to address social problems typically do not result in sufficient information to help practitioners. Current standards of evidence employed across the United States apply a positivist notion of validity with quantitative research criteria that discourage answering important how and why questions, explicitly privilege quantitative/RCT evidence, offer few contextual insights, and rarely discuss disparities in outcomes across participants differing by race, gender, and ethnicity. We offer a set of standards of evidence to assess qualitative and mixed methods studies, as well as RCTs and quasi‐experimental designs, and probe the extent to which the studies address context and equity. We applied our proposed new standards of evidence to all intervention studies rated as the highest quality by the What Works Clearinghouse (WWC) sponsored by the U.S. Department of Education from 2017 to 2021 to demonstrate the usefulness of our standards.