Agree or Disagree: Does It Matter Which Comes First? An Examination of Scale Direction Effects in a Multi-device Online Survey

Abstract

Previous research shows that the direction of rating scales can influence participants’ response behavior. Studies also suggest that the device used to complete online surveys might affect the susceptibility to these effects due to the different question layouts (e.g., horizontal grids vs. vertical individual questions). This article contributes to previous research by examining scale direction effects in an online multi-device survey conducted with panelists in Spain. In this experiment, respondents were randomly assigned to two groups where the scale direction was manipulated (incremental vs. decremental). Respondents completed the questionnaire using the device of their choosing (57.8% used PCs; 36.5% used smartphones; and 5.7% used tablets). The results show that scale direction influenced response distributions but did not significantly affect data quality. In addition, our findings indicate that scale direction effects were comparable across devices. Findings are discussed and implications are highlighted.