Identifying and Responding to Health Misinformation on Reddit Dermatology Forums With Artificially Intelligent Bots Using Natural Language Processing: Design and Evaluation Study (Preprint)

Abstract

Reddit, the fifth most popular website in the United States, boasts a large and engaged user base on its dermatology forums where users crowdsource free medical opinions<i>.</i> Unfortunately, much of the advice provided is unvalidated and could lead to the provision of inappropriate care. Initial testing has revealed that artificially intelligent bots can detect misinformation regarding tanning and essential oils on Reddit dermatology forums and may be able to produce responses to posts containing misinformation.

To analyze the ability of bots to find and respond to tanning and essential oil–related health misinformation on Reddit’s dermatology forums in a controlled test environment.

Using natural language processing techniques, we trained bots to target misinformation, using relevant keywords and to post prefabricated responses. By evaluating different model architectures across a held-out test set, we compared performances.

Our models yielded data test accuracies ranging 95%-100%, with a Bidirectional Encoder Representations from Transformers (BERT) fine-tuned model resulting in the highest level of test accuracy. Bots were then able to post corrective prefabricated responses to misinformation in a test environment.

Using a limited data set, bots accurately detected examples of health misinformation within Reddit dermatology forums. Given that these bots can then post prefabricated responses, this technique may allow for interception of misinformation. Providing correct information does not mean that users will be receptive or find such interventions persuasive. Further studies should investigate this strategy’s effectiveness to inform future deployment of bots as a technique in combating health misinformation.