Drop Downs and Scroll Mice: The Effect of Response Option Format and Input Mechanism Employed on Data Quality in Web Surveys

Abstract

Online researchers face whether to use radio buttons or drop downs when presenting respondents with “select one answer from many” questions. However, empirical evidence regarding response effects does not provide direction for favoring one. Using data collected in a New Zealand general population Web survey of 2,400 people, this study contributes to the decision process by investigating format response effects at multiple levels and exploring the potential for input mechanisms to interfere with drop-down answer selection. Format choice did not significantly affect survey completions, number of nonsubstantial answers, or time to completion. However, drop downs led to higher item nonresponse and longer response times. Furthermore, the 76% of respondents using scroll mice to complete the survey were prone to accidentally changing an answer if presented with drop-down questions. This increased average response times and skewed distribution of responses to the drop-down treatment questions toward the bottom of the response list. Implications of these findings for Web-based data collection are discussed.