Modeling and Measuring Expressed (Dis)belief in (Mis)information

Abstract

The proliferation of online misinformation has been raising increasing societal concerns about its potential consequences, e.g., polarizing the public and eroding trust in institutions. These consequences are framed under the public's susceptibility to such misinformation — a narrative that needs further investigation and quantification. To this end, our paper proposes an observational approach to model and measure expressed (dis)beliefs in (mis)information by leveraging social media comments as a proxy. We collect a sample of tweets in response to (mis)information and annotate them with (dis)belief labels, explore the dataset using lexicon-based methods, and finally build classifiers based on the state-of-the-art neural transfer-learning models (BERT, XLNet, and RoBERTa). Under a domain-specific thresholding strategy for unbiasedness, the best-performing classifier archives macro-F1 scores around 0.86 for disbelief and 0.80 for belief. Applying the classifier, we conduct a large-scale measurement study and show that, for true/mixed/false claims on social media, 12%/14%/15% of comments express disbelief and 26%/21%/20% of comments express belief. In addition, our results suggest an extremely slight time effect of falsehood awareness, a positive effect of fact-checks to false claims, and differences in (dis)belief across social media platforms.