Restoring Sentiments: Understanding Citizens’ Response to Social Activities on Twitter in U.S. Metropolises During the COVID-19 Pandemic Using Fine-tuned Large Language Model (Preprint)

Abstract

The COVID-19 pandemic remains significant in the collective memory in 2024. As of March 2024, over 676 million cases, 6 million deaths, and 13 billion vaccine doses have been reported. It’s crucial to evaluate sociopsychological impacts as well as public health indicators such as these to understand the pandemic’s effects.

This paper explores the sentiments of residents of major U.S. cities toward restrictions on social activities in 2022 during the transitional phase of the COVID-19 pandemic, from the peak of the pandemic to its gradual decline. By illuminating people’s susceptibility to COVID-19, we provide insights into the general sentiment trends during the recovery phase of the pandemic.

To analyze these trends, we collected posts (N = 119,437) on the social media platform Twitter (now X) made by people living in New York City, Los Angeles, and Chicago, which were impacted by the COVID-19 pandemic in similar ways. Additionally, we constructed a sentiment estimation model by fine-tuning a large language model on the collected data and used it to analyze citizens’ sentiments as the pandemic changed.

In the evaluation of models, GPT-3.5 Turbo with fine-tuning outperformed GPT-3.5 Turbo without fine-tuning and RoBERTa-Large with fine-tuning, demonstrating significant accuracy (0.80) and F1 score (0.79). The findings using GPT-3.5 Turbo with fine-tuning reveal a significant relationship between sentiment levels and actual cases in all three cities. Specifically, the correlation coefficient for New York City is 0.89, for Los Angeles is 0.39, and for Chicago is 0.65. Furthermore, feature words analysis shows that coronavirus-related keywords were replaced with non-coronavirus-related keywords in New York City and Los Angeles from January 2022 onwards and Chicago from March 2022 onwards.

The results show a gradual decline in sentiment and interest in restrictions across all three cities as the pan-demic approached its conclusion. These results are also ensured by a sentiment estimation model fine-tuned on actual Twitter posts. This study represents the first attempt from a macro perspective to depict sentiment using a classification model created with actual data from the period when COVID-19 was prevalent. This approach can be applied to the spread of other infectious diseases by adjusting search keywords for observational data.