Estimating Classification Accuracy for Complex Decision Rules Based on Multiple Scores

Abstract

Important decisions about students are made by combining multiple measures using complex decision rules. Although methods for characterizing the accuracy of decisions based on a single measure have been suggested by numerous researchers, such methods are not useful for estimating the accuracy of decisions based on multiple measures. This study presents a simulation method useful for estimating classification accuracy for any measurement model and complex decision rule. The utility of the method is demonstrated by application to actual data from General Educational Development (GED) test takers. Findings in this study illustrate how multiple scores that are combined can have a large impact on the accuracy of the resulting decision. Furthermore, choice of measure of agreement is of central importance in describing decision accuracy.