An Experimental Study of the Effects of Monetary Incentives on Performance on the 12th-Grade NAEP Reading Assessment

Abstract

Background/context The National Assessment of Educational Progress (NAEP) is the only comparative assessment of academic competencies regularly administered to nationally representative samples of students enrolled in Grades 4, 8, and 12. Because NAEP is a low-stakes assessment, there are long-standing questions about the level of engagement and effort of the 12th graders who participate in the assessment and, consequently, about the validity of the reported results. Purpose/Focus This study investigated the effects of monetary incentives on the performance of 12th graders on a reading assessment closely modeled on the NAEP reading test in order to evaluate the likelihood that scores obtained at regular administrations underestimate student capabilities. Population The study assessed more than 2,600 students in a convenience sample of 59 schools in seven states. The schools are heterogeneous with respect to demographics and type of location. Intervention There were three conditions: a control and two incentive interventions. For the fixed incentive, students were offered $20 at the start of the session. For the contingent incentive, students were offered $5 in advance and $15 for correct responses to each of two randomly chosen questions, for a maximum payout of $35. All students were administered one of eight booklets comprising two reading blocks (a passage with associated questions) and a background questionnaire. All reading blocks were operational blocks released by NAEP. Research Design This was a randomized controlled field trial. Students agreed to participate without knowing that monetary incentives would be offered. Random allocation to condition was conducted independently in each school. Data Collection/Analysis Regular NAEP contractors administered the assessments and carried out preliminary data processing. Scaling of results and linking to the NAEP reporting scale were conducted using standard NAEP procedures. Findings Monetary incentives have a statistically significant and substantively important impact on both student engagement/effort and performance overall, and for most subgroups defined by gender, race, and background characteristics. For both males and females, the effect of the contingent incentive was more than 5 NAEP score points, corresponding to one quarter of the difference in the average scores between Grades 8 and 12. In general, the effect of the contingent incentive was larger than that of the fixed incentive, particularly for lower scoring subgroups. Conclusions/Recommendations There is now credible evidence that NAEP may both underestimate the reading abilities of students enrolled in 12th grade and yield biased estimates of certain achievement gaps. Responsible officials should take this into account as they plan changes to the NAEP reading framework and expand the scope of the 12th-grade assessment survey.