Master Turkers: An assessment of data quality

Abstract

Amazon’s Mechanical Turk has greatly increased in popularity in recent years considering recent world events as well as due to the increased acceptance of technology in the field of research. Because of this, it is essential that the research methods associated with conducting research online be evaluated. The present study evaluated if Amazon’s upper echelon of workers, Master Turkers, provide a higher quality of data relative to workers without that designation. This was evaluated using two scales that are validated and have been extensively used in research. The results showed that Master Turkers were found to have worse performance on scales (lower reliability) compared to non-Master Turkers. This data highlights an issue that potential researchers should be aware of when using the Mechanical Turk, as well as problem that should be addressed by Amazon.