Comparing Ranking Techniques in Web Surveys

Abstract

In an experimental split ballot design, I test four different ranking techniques (drag and drop, numbering, arrows, and most–least) to explore potential effects on substantive answers, dropouts, and item nonresponse and response time between the groups. As an example, I use six items from Inglehart’s materialism–postmaterialism index. Data come from 1,225 members of an access panel who entered the set of items to be rank ordered. With respect to sex, education, and age, there are no significant differences between the four experimental groups. However, the groups differ extensively in response time, item nonresponse, and estimation of the percentage of materialists and postmaterialists. Drag and drop is shown to be the best-suited method for collecting rank data in web surveys.