Innovating Web Probing: Comparing Written and Oral Answers to Open-Ended Probing Questions in a Smartphone Survey

Abstract

Abstract Cognitive interviewing in the form of probing is key for developing methodologically sound survey questions. For a long time, probing was tied to the laboratory setting, making it difficult to achieve large sample sizes and creating a time-intensive undertaking for both researchers and participants. Web surveys paved the way for administering probing questions over the Internet in a time- and cost-efficient manner. In so-called web probing studies, respondents first answer a question and then they receive one or more open-ended questions about their response process, with requests for written answers. However, participants frequently provide very short or no answers at all to open-ended questions, in part because answering questions in writing is tedious. This is especially the case when the web survey is completed via a smartphone with a virtual on-screen keypad that shrinks the viewing space. In this study, we examine whether the problem of short and uninterpretable answers in web probing studies can be mitigated by asking respondents to complete the web survey on a smartphone and to record their answers via the built-in microphone. We conducted an experiment in a smartphone survey (N = 1,001), randomizing respondents to different communication modes (written or oral) for answering two comprehension probes about two questions on national identity and citizenship. The results indicate that probes with requests for oral answers produce four to five times more nonresponse than their written counterparts. However, oral answers contain about three times as many words, include about 0.3 more themes (first probing question only), and the proportion of clearly interpretable answers is about 6 percentage points higher (for the first probing question only). Nonetheless, both communication modes result in similar themes mentioned by respondents.