Detecting and Deterring Information Search in Online Surveys

Abstract

AbstractThis article introduces a framework for evaluating methods of combatting information search in online surveys. Three empirical studies based on the framework suggest that search is a serious but manageable problem. Search frequency varies substantially according to question content, ranging from 2% to 30% in batteries of general political knowledge questions. Deterrence works: a pledge not to cheat reduces search by half. Detection also works: web browser paradata identify 70% to 85% of search, while 60% to 85% of search on knowledge questions is undertaken by respondents who correctly answer “catch” questions about obscure Supreme Court cases. Detection and deterrence are complementary: deterrence reduces search ex ante, while detection quantifies success and provides ex post options for dealing with undeterred search. In combination, the three methods tested (pledge, paradata, and catch) deter or detect more than 90% of search, leaving search to affect about 0.5% of the remaining observations.