Toloka platform as a source of online survey participants: an experience of assessing data quality

Abstract

The article presents the experience of using Yandex Toloka crowdsourcing platform to recruit respondents for an online survey. Analyzing methodological publications on a similar foreign platform Amazon Mechanical Turk we put forward hypotheses about the data quality obtained via Toloka in comparison with the results collected using other convenience sample types –online panels and recruitment of respondents through social networks. Additionally, only based on the Toloka data, we assessed the indicator of respondent’s attentiveness. The main conclusion is that Toloka allows to recruit respondents quickly and at low cost, and the results are comparable in terms of quality to those obtained by other methods. In particular, respondents from Toloka almost always complete the survey, fill out questionnaires faster than other types of respondents, but less often than participants of the online panel have a tendency to “straightline” (i.e., give the same answers in a tabular question), just as often as social media respondents give answers to the open-ended question (but less frequently than online panel participants), although their responses are shorter. Only 36% of the respondents passed the attention check question, attentive participants had a longer questionnaire complete time and were less likely to be straightliners. The increase of reward did not increase the proportion of attentive respondents, but decreased the questionnaire filling out speed, increased the number of answers to the open question, and reduced the proportion of straightliners.