Influenza Surveillance using Search Engine, SNS, On-line Shopping, Q&A Service and Past Flu Patients (Preprint)

Abstract

Influenza, an infectious disease, causes many deaths worldwide. Predicting influenza victims during epidemics is an important task for clinical, hospital, and community outbreak preparation.

On-line user-generated contents (UGC), primarily in the form of social media posts or search query logs, are generally used for prediction for reaction to sudden and unusual outbreaks. However, most studies rely only on the UGC as their resource and do not use various UGCs. Our study aims to solve these questions about Influenza prediction: Which model is the best? What combination of multiple UGCs works well? What is the nature of each UGC?

We adapt some models, LASSO Regression, Huber Regression, Support Vector Machine regression with Linear kernel (SVR) and Random Forest, to test the influenza volume prediction in Japan during 2015 – 2018. For that, we use on-line five data resources: (1) past flu patients, (2) SNS (Twitter), (3) search engines (Yahoo! Japan), (4) shopping services (Yahoo! Shopping), and (5) Q&A services (Yahoo! Chiebukuro) as resources of each model. We then validate respective resources contributions using the best model, Huber Regression, with all resources except one resource. Finally, we use Bayesian change point method for ascertaining whether the trend of time series on any resources is reflected in the trend of flu patient count or not.

Our experiments show Huber Regression model based on various data resources produces the most accurate results. The coefficient of determination R2 are 0.907 from 2015 – 2016, 0.889 from 2016 – 2017 and 0.917 from 2017 – 2018 in predicting influenza 2 week ahead of the current date. Then, from the change point analysis, we get the result that search query logs and social media posts for three years represents these resources as a good predictor. The results of Sensitivity, is one metric, are 80% in search query and 75% in social media posts from 2017-2018.

We show that Huber Regression based on various data resources is strong for outliers and is suitable for the flu prediction. Additionally, we indicate the characteristics of each resource for the flu prediction.