Abstract
This study explores the potential of using large language models to assist content analysis by conducting a case study to identify adverse events (AEs) in social media posts. The case study compares ChatGPT’s performance with human annotators’ in detecting AEs associated with delta-8-tetrahydrocannabinol, a cannabis-derived product. Using the identical instructions given to human annotators, ChatGPT closely approximated human results, with a high degree of agreement noted: 94.4% (9436/10,000) for any AE detection (Fleiss κ=0.95) and 99.3% (9931/10,000) for serious AEs (κ=0.96). These findings suggest that ChatGPT has the potential to replicate human annotation accurately and efficiently. The study recognizes possible limitations, including concerns about the generalizability due to ChatGPT’s training data, and prompts further research with different models, data sources, and content analysis tasks. The study highlights the promise of large language models for enhancing the efficiency of biomedical research.
Reference10 articles.
1. ChatGPTOpenAI2024-04-25https://chat.openai.com/
2. Self-reported adverse events associated with ∆8-Tetrahydrocannabinol (Delta-8-THC) Use
3. Utilizing social media data for pharmacovigilance: A review
4. MedWatch: The FDA Safety InformationAdverse Event Reporting ProgramUS Food and Drug Administration20229152023-01-03https://www.fda.gov/safety/medwatch-fda-safety-information-and-adverse-event-reporting-program