The effect of the multiple-choice item format on the measurement of knowledge of language structure

Abstract

Noting the widespread use of multiple-choice items in tests in English language education in Thailand, this study compared their effect against that of constructed-response items. One hundred and fifty-two university undergraduates took a test of English structure first in constructed-response format, and later in three, stem-equivalent multiple-choice formats, with the distractors based on incorrect answers from the constructed-response test. A significant and substantial increase in mean, and generally in individual scores between the two tests was found although the scores in the tests were quite closely correlated, often taken to indicate that a similar construct was measured by the two test formats. However, direct comparison of the responses to the items in the two tests showed that only 26% of the responses were the same, suggesting that most of what the multiple-choice items measured was directly dependent on the item format. The study found remarkable consistency in the response patterns between the tests among three experimental groups of participants, who sat different option number formats of the multiple choice test, pointing to the possibility of a general effect of multiple-choice items in testing the learning of structure in second and foreign languages.