A Comparative Study of Measures of Partial Knowledge in Multiple-Choice Tests

Abstract

A common belief among many test experts is that measurements obtained from multiple-choice (MC) tests can be improved by using evidence about partial knowledge. A large number of methods designed to extract such information from direct reports provided by examinees have been developed over the last 50 years. Most methods require modifications in test instructions, response modes, and scoring rules. These testing methods are reviewed and the results of a large-scale empirical study of the most promising among them are reported. Seven testing methods were applied to MC tests from four different content areas using a between-persons design. To identify the most efficient methods and the optimal conditions for their application, the results were analyzed with respect to six different criteria. The results showed a surprisingly large tendency on the part of the examinees to take advantage of the special features of the alternative methods and indicated that, on average, high ability examinees were better judges of their level of knowledge and, consequently, could benefit more from these methods. Systematic interactions were found between the testing method and the test content, indicating that no method was uniformly superior.