Item-Option Weighting of Achievement Tests: Comparative Study of Methods

Abstract

Previous research has studied the effects of dif ferent methods of item-option weighting on the re liability and the concurrent and predictive validity of achievement tests. Generally, increases in relia bility are found, but with mixed results for validity. This research attempted to interrelate several methods of producing option weights (i.e., Gutt man internal and external weights and judges' weights) and examined their effects on reliability and on concurrent, predictive, and face validity. Option weights to maximize reliability produced cross-validated ( N = 974) increases in Hoyt reli ability over rights-only scoring (.82 versus .58, re spectively) ; decreases in correlations with other achievement tests; few changes in predictive valid ity ; and a loss in face validity (i.e., some correct op tions had lower weights than incorrect options). Weights to maximize validity did not cross-validate and led to a reduction in reliability and to mixed validity results. Judges' weights produced increases in reliability and mixed results with validity. The size of Guttman weights were shown to interact with item-option and test characteristics. It was concluded that option weighting offered limited, if any, improvement over unit weighting.