Affiliation:
1. Institute of Communication and Media Studies, University of Bern, Switzerland
2. Research Group “Platform Algorithms and Digital Propaganda,” Weizenbaum Institute, Germany
Abstract
Research on digital misinformation has turned its attention to large language models (LLMs) and their handling of sensitive political topics. Through an AI audit, we analyze how three LLM-powered chatbots (Perplexity, Google Bard, and Bing Chat) generate content in response to the prompts linked to common Russian disinformation narratives about the war in Ukraine. We find major differences between chatbots in the accuracy of outputs and the integration of statements debunking Russian disinformation claims related to prompts’ topics. Moreover, we show that chatbot outputs are subject to substantive variation, which can result in random user exposure to false information.
Funder
Bundesministerium für Bildung und Forschung
Publisher
Shorenstein Center for Media, Politics, and Public Policy
Reference34 articles.
1. Atkins, C., Zhao, B. Z. H., Asghar, H. J., Wood, I., & Kaafar, M. A. (2023). Those aren’t your memories, they’re somebody else’s: Seeding misinformation in chat bot memories. In M. Tibouchi & X. Wang (Eds.), Applied Cryptography and Network Security (pp. 284–308). Springer. https://doi.org/10.1007/978-3-031-33488-7_11
2. Au, C. H., Ho, K. K. W., & Chiu, D. K. W. (2022). The role of online misinformation and fake news in ideological polarization: Barriers, catalysts, and implications. Information Systems Frontiers, 24(4), 1331–1354. https://doi.org/10.1007/s10796-021-10133-9
3. Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? 🦜. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency (pp. 610–623). Association for Computing Machinery. https://doi.org/10.1145/3442188.3445922
4. Bradshaw, S. (2019). Disinformation optimised: Gaming search engine algorithms to amplify junk news. Internet Policy Review, 8(4). https://doi.org/10.14763/2019.4.1442
5. Caramancion, K. M. (2024). Large language models vs. search engines: Evaluating user preferences across varied information retrieval scenarios. arXiv. https://doi.org/10.48550/arXiv.2401.05761