Multiple dichotomous-scored items in second language testing: investigating the multiple true-false item type under norm-referenced conditions

Abstract

This study examined the multiple true-false (MTF) test format in second language testing by comparing multiple-choice (MCQ) and multiple true-false (MTF) test formats in two language areas of general English: vocabulary and reading. Two counter-balanced experimental designs - one for each language area - were examined in terms of the number of MCQ and MTF differentially responded to, MTF item dependency, reliability, and concurrent validity. The data were analysed by classical test theory (CTT) and Rasch analysis. The results showed a two- and three-fold increase in vocabulary and reading items answered, respectively. Participants responded to significantly more MTF items than MCQ, and further analysis revealed no item dependency for both language domains. Reliability increases were found in the reading tests. Item conversions did not alter the basic functioning of the MTF items, and common person equating plots demonstrated a steady relationship between MCQ and MTF person ability estimates.