The Effects of Guessing and Item Dependence on the Reliability and Validity of Recognition Based Cloze Tests

Abstract

Matching doze and multiple-choice doze tests of elementary level reading comprehension have demonstrated promising construct and concurrent validity. However, their formats include guessing and, in the case of matching cloze, item dependence effects. This study used a Monte Carlo design to examine how these effects influence test characteristics and student scores. First, a true score test with six levels of difficulty was constructed. Then, two versions of matching doze and three of multiple-choice doze results were generated for each level of difficulty of the true score data by applying the doze assumptions of random guessing and item dependence. Although validity for all five doze variants was high, multiple-choice doze had significantly lower reliabilities than did the true score equivalents. Item dependence found in matching doze had little or no effect on the characteristics of these tests.