The Importance of Using Multiple Data Sources in Policy Assessments: Lessons From Two Conditional Cash Transfer Programs in New York City

Abstract

Background: The high costs of implementing surveys are increasingly leading research teams to either cut back on surveys or to rely on administrative records. Yet no policy should be based on a single set of estimates, and every approach has its weaknesses. A mixture of approaches, each with its own biases, should provide the analyst with a better understanding of the underlying phenomenon. This claim is illustrated with a comparison of employment effect estimates of two conditional cash transfer programs in New York City using survey and administrative unemployment insurance (UI) data. Objectives: This article explores whether using administrative data and survey data produce different impact estimates and investigates the source of differential effects between data sources. Research design: The results of a survey nonresponse bias analysis and an analysis of characteristics of non-UI-covered job characteristics using data collected on 6,000 families who enrolled in either the Family Rewards or Work Rewards evaluation are presented. Results: In both evaluations, survey data showed positive employment effects, while administrative data showed no statistically significant employment effects. Family Rewards increased employment mostly in non-UI-covered jobs, while the positive survey impact estimates in Work Rewards were partially due to survey nonresponse bias. Conclusions: Despite cost pressures leading researchers to collect and analyze only administrative records, the results suggest that survey and administrative records data both suffer from different kinds of sample attrition, and researchers may need to triangulate data sources to draw accurate conclusions about program effects. Developing more economical data collection practices is a major priority.