Investigating response order effects in web surveys using eye tracking

Abstract

Response order effects are a well-known phenomenon that can occur when answering survey questions with multiple response categories. Although various theoretical explanations exist, the empirical evidence is contradictory. Moreover, different scale types produce different effect sizes. In the current study, we investigate the occurrence and causes of response order effects in horizontal and vertical rating scales by means of eye tracking. We conducted an experiment (n = 84) with two groups and varied the scale direction so that the response scales either ran from agree to disagree or vice versa. The results indicate that response order effects in rating scales are relatively small and are more likely to occur in vertical than in horizontal rating scales. Moreover, our eye-tracking data reveal that respondents do not read all categories, nor do they pay equal attention to all categories; these data support the survey satisficing theory of response order effects (Krosnick, 1991).