Impact Evaluation Using Analysis of Covariance With Error-Prone Covariates That Violate Surrogacy

Abstract

Background: Analysis of covariance (ANCOVA) is commonly used to adjust for potential confounders in observational studies of intervention effects. Measurement error in the covariates used in ANCOVA models can lead to inconsistent estimators of intervention effects. While errors-in-variables (EIV) regression can restore consistency, it requires surrogacy assumptions for the error-prone covariates that may be violated in practical settings. Objectives: The objectives of this article are (1) to derive asymptotic results for ANCOVA using EIV regression when measurement errors may not satisfy the standard surrogacy assumptions and (2) to demonstrate how these results can be used to explore the potential bias from ANCOVA models that either ignore measurement error by using ordinary least squares (OLS) regression or use EIV regression when its required assumptions do not hold. Results: The article derives asymptotic results for ANCOVA with error-prone covariates that cover a variety of cases relevant to applications. It then uses the results in a case study of choosing among ANCOVA model specifications for estimating teacher effects using longitudinal data from a large urban school system. It finds evidence that estimates of teacher effects computed using EIV regression may have smaller bias than estimates computed using OLS regression when the data available for adjusting for students’ prior achievement are limited.