The Inadequacy of ANOVA for Detecting Test Bias

Abstract

The inadequacy of ANOVA for detecting bias in test items should already be well understood, yet it persists as a popular method. Here, previous arguments are extended to explain why ANOVA may obscure test bias when it exists, as well as create false impressions of bias. In fact, it is demonstrated in this paper that ANOVA will fail to detect even absurdly large amounts of bias. More specifically, it is shown that bias contributes relatively more to the group main effect than to the group-by-item interaction.