Position Bias in Multiple-Choice Questions

Abstract

The author argues that rotation of answer alternatives in multiple-choice questions does not eliminate position bias—i.e., bias caused by respondents’ tendency to opt for alternatives in certain positions (e.g., the first few mentioned) regardless of their content. Researchers seem to have overlooked this conclusion, possibly because they have approached the problem as a purely manifest phenomenon without reference to a statistical model of the data-generating process. The study of such models and their application to empirical data also reveals that position bias (1) is not necessarily disclosed by rotation, (2) is often much more important than sampling error, and (3) precludes usual estimation of the parameters of the data-generating process. These findings also have implications for nonverbal stimuli and raise questions about efforts to handle position-biased experimental data when the response variable is only nominally scaled.