Abstract
AbstractTools like ChatGPT, which allow people to unlock the potential of large language models (LLMs), have taken the world by storm. ChatGPT’s ability to produce written output of remarkable quality has inspired, or forced, academics to consider its consequences for both research and education. In particular, the question of what constitutes authorship, and how to evaluate (scientific) contributions has received a lot of attention. However, its impact on (online) human data collection has mostly flown under the radar. The current paper examines how ChatGPT can be (mis)used in the context of generating norming data. We found that ChatGPT is able to produce sensible output, resembling that of human participants, for a typicality rating task. Moreover, the test–retest reliability of ChatGPT’s ratings was similar to that of human participants tested 1 day apart. We discuss the relevance of these findings in the context of (online) human data collection, focusing both on opportunities (e.g., (risk-)free pilot data) and challenges (e.g., data fabrication).
Publisher
Springer Science and Business Media LLC
Subject
General Psychology,Psychology (miscellaneous),Arts and Humanities (miscellaneous),Developmental and Educational Psychology,Experimental and Cognitive Psychology
Reference35 articles.
1. Allaire, J., Cheng, J., Xie, Y., McPherson, J., Chang, W., Allen, J., ... Hyndman, R. (2016). rmarkdown: Dynamic Documents for R. https://CRAN.R-project.org/package=rmarkdown
2. Aust, F., & Barth, M. (2017). papaja: Create APA Manuscripts with R Markdown. https://github.com/crsh/papaja
3. Banks, B., & Connell, L. (2022). Category production norms for 117 concrete and abstract categories. Behavior Research Methods. https://doi.org/10.3758/s13428-021-01787-z
4. Baroni, M., Dinu, G., & Kruszewski, G. (2014). Don’t count, predict! A systematic comparison of context-counting vs. Context-predicting semantic vectors. Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, 238–247. https://doi.org/10.3115/v1/P14-1023
5. Barsalou, L. W. (1987). Concepts and conceptual development: Ecological and intellectual factors in categorization, Neisser, U. (Ed.); pp. 101–140. Cambridge: Cambridge University Press.