Vignette Method in Online Research: Measuring Perceptions of Fairness

Abstract

The article covers the use of the vignette method in online research. The possibilities of the method based on the measurement of the perception of justice are demonstrated and the tools developed for these purposes are evaluated. The analysis of the quality of the proposed scales is carried out using quantitative assessments of the main psychometric indicators: constructive and criterion validity. The substantiation and calculation of these indicators are described in detail, as well as the interpretation of the obtained values. A moderate indicator of construct validity and a satisfactory indicator of criterion validity indicate the need for further optimisation of the scales. Special attention is paid to the peculiarities of using this method online with the help of participants recruited from a commercial panel of test subjects. The paradata are analysed and the dependence of the meaningful results of the study on the non-semantic aspects of the tools is traced. In particular, the influence of the interaction between the order of presentation of stimuli and the type of devices on judgments about fairness is demonstrated. It is shown that participants who completed tasks using mobile devices tend to consider the conditions that they were shown first to be more fair. In contrast to participants who performed tasks on desktop devices, where this effect is practically not observed. Recruiting participants from an online panel for a study using the vignette method demonstrated a number of limitations that should be considered when designing a study. Given the demonstrated inattention of participants when completing tasks from mobile devices, it is recommended to limit the completion of questionnaires to the use of desktop devices - computers and laptops. Also, unlike previous studies in this area, when using the vignette method, it is recommended to limit the number of levels of factors in the experimental plan to reduce the cognitive complexity of tasks in the conditions of the inability to control the attentiveness of their performance. The results obtained are analysed in the context of methodological research. Methodological solutions are proposed for using the online vignette method, current and possible sources of data distortion are indicated, and recommendations are given for the further use of the developed tools.