Using a Brier Score Analysis to Assess the Effectiveness of a Mathematics Placement Policy

Abstract

This article describes how a Brier score analysis can be used as an evaluative tool to estimate the predictive accuracy of a course placement policy that was established based on professional or subjective judgment. The policy being evaluated uses the score received on the mathematics portion of the SAT examination as the primary mechanism to place incoming freshmen into the appropriate college mathematics courses best matched to their ability. This placement policy was established based on the idea that the mathematics portion of the SAT examination and a commercially available computerized mathematics placement examination may be placing the majority of students at a similar level in the hierarchy of mathematics courses. The findings of this study indicate that the mathematics portion of the SAT examination can be an effective mechanism for initially placing students in their first mathematics course which is appropriate for their academic ability, and thus can increase their chance of succeeding in their first college mathematics course. Furthermore, a Brier score analysis allowed for comparing and contrasting different scenarios when the placement policy is effective versus when it is not. Implementing this policy allowed for a simpler placement process as well as a substantial financial savings for the institution.