Visualizing the Interpretation of a Criterion-Driven System that Automatically Evaluates the Quality of Health News: an Exploratory Study of Two Approaches (Preprint)

Abstract

Machine learning techniques have been shown efficient at identifying health misinformation. However, interpreting a classification model remains challenging due to the model's intricacy. The absence of a justification for the classification result and disclosure of the model's domain knowledge may erode end-users’ trust in such models. This diminished trust may also undermine the effectiveness of artificial intelligence-based initiatives to counteract health misinformation.

The study objective is to address the public's need for help evaluating the quality of health news and the typical opaqueness of an AI approach. This work aims to create an interpretable, criteria-driven system. We also aim to find the best approach by comparing the accuracy of two methods for selecting and visualizing the system's interpretation. Both methods highlight the sentences that contributed to the analysis performed by the system for a given criterion.

We employ an interpretable, criteria-based approach for automatically assessing the quality of health news on the Internet. One of ten well-established criteria was chosen for our experimentation. To automate the evaluation of the criterion, we tested Logistic Regression, Naive Bayes, Support Vector Machine, and Random Forest algorithms. We then experimented with two approaches for developing interpretable representations of the results. For the first approach, (1) we calculate word feature weights, which explains how classification models distill keywords that are relevant to the positive prediction;(2) then using the Local Interpretable Model Explanations (LIME) framework, we automatically select keywords for visualization that show how classification models identify positive news articles, based on related keywords;(3)and finally, the system highlights target sentences containing keywords to justify the criterion evaluation result. For the second approach, (1) we extract sentences that provide evidence to support the evaluation result from 100 health news articles, (2) based on these results, we train a typology classification model at a sentence level; (3) then, the system highlights positive sentence instances for the result justification. We assess the accuracy of both methods using a small held-out test set.

The Random Forest model achieved the highest average AUC score of 0.74 in evaluating the criterion. Additionally, LIME provided a visualization of how keywords impact the system's evaluation results. Both approaches could successfully visualize the interpretation of the system. When tested with four cases, the hybrid approach performs slightly better in highlighting correct sentences with an accuracy of 55.00%, compared to 50.00% by the typology approach.

We provide an interpretable, criteria-based method to evaluate health news. The method incorporates rule-based and statistical machine learning approaches. Results suggest one might automate criterion-based health news quality evaluation successfully using either approach, but larger differences may arise when multiple quality-related criteria are considered. This work can increase public trust in computerized health information evaluation.