Use of large language models to assess likelihood of epidemics from content of Tweets: Infodemiology Study (Preprint)

Abstract

Cost-effective automated surveillance systems could leverage social media content analysis, with the potential to serve as early indicators of conjunctivitis and other systemic infectious diseases.

We investigated whether large language models, specifically GPT-3.5 and GPT-4, can provide probabilistic assessments of whether or not social media posts about conjunctivitis could indicate an outbreak.

12,194 conjunctivitis-related Tweets were obtained using a targeted Boolean search in multiple languages for 9 countries. These Tweets were provided in prompts to GPT-3.5 and GPT-4, obtaining probabilistic assessments which were validated by two human raters. We then calculated Pearson correlations of these time series with post volume and the occurrence of known outbreaks in nine selected countries, with time series bootstrap used to compute confidence intervals.

Probabilistic assessments derived from GPT-3.5 showed correlations of 0.60 (95% CI: 0.47–0.70) and 0.53 (95% CI: 0.40–0.65) with the two human raters, with higher results for GPT-4. Weekly averages of GPT-3.5 probabilities showed substantial correlations with weekly Tweet volume for some countries, with correlations ranging from 0.10 (95% CI: 0.0–0.29) to 0.53 (95% CI: 0.39–0.89), with larger correlations for GPT-4. More modest correlations were found for correlation with known epidemics, with substantial correlation only in American Samoa (0.40 (95% CI: 0.16–0.81)).

These findings suggest that GPT prompting can efficiently assess content of social media post and possible outbreaks to a degree comparable to that of humans. Further, we found that automated content analysis of Twitter content is related to Twitter volume for conjunctivitis-related posts in some locations, and to the occurrence of actual epidemics. Future work may improve the sensitivity and specificity of these methods for outbreak detection.