Different Order, Different Results? The Effects of Dimension Order in Factorial Survey Experiments

Abstract

Factorial surveys are widely used in the social sciences to measure respondents’ attitudes, beliefs, or behavioral intentions. In such surveys, respondents evaluate short descriptions of hypothetical situations, persons, or objects that vary across several dimensions. An important prerequisite of the method’s validity is that respondents are able to deal with the highly complex task created by the need to consider several variable dimensions within one coherent judgment. We analyze the effects of the order in which dimensions are presented in running text vignettes. An experimental setup with four order treatments was randomly allocated to 787 respondents (based on a random sample of register data), yielding 3,119 vignette evaluations. The analyses compare respondent groups across age, education, and response speed. Overall, there is no strong evidence for order effects. However, we find a slight tendency for fast responders to be more prone to recency effects.