Solving Measurement Problems with an Answer-Until-Correct Scoring Procedure

Abstract

Answer-until-correct (AUC) tests have been in use for some time. Pressey (1950) pointed to their ad vantages in enhancing learning, and Brown (1965) proposed a scoring procedure for AUC tests that appears to increase reliability (Gilman & Ferry, 1972; Hanna, 1975). This paper describes a new scoring procedure for AUC tests that (1) makes it possible to determine whether guessing is at ran dom, (2) gives a measure of how "far away" guess ing is from being random, (3) corrects observed test scores for partial information, and (4) yields a mea sure of how well an item reveals whether an ex aminee knows or does not know the correct re sponse. In addition, the paper derives the optimal linear estimate (under squared-error loss) of true score that is corrected for partial information, as well as another formula score under the assumption that the Dirichlet-multinomial model holds. Once certain parameters are estimated, the latter formula score makes it possible to correct for partial infor mation using only the examinee's usual number- correct observed score. The importance of this for mula score is discussed. Finally, various statistical techniques are described that can be used to check the assumptions underlying the proposed scoring procedure.