Abstract
Null models provide a critical baseline for the evaluation of predictive disease models. Many studies consider only the grand mean null model (i.e. R2) when evaluating the predictive ability of a model, which is insufficient to convey the predictive power of a model. We evaluated ten null models for human cases of West Nile virus (WNV), a zoonotic mosquito-borne disease introduced to the United States in 1999. The Negative Binomial, Historical (i.e. using previous cases to predict future cases) and Always Absent null models were the strongest overall, and the majority of null models significantly outperformed the grand mean. The length of the training timeseries increased the performance of most null models in US counties where WNV cases were frequent, but improvements were similar for most null models, so relative scores remained unchanged. We argue that a combination of null models is needed to evaluate the forecasting performance of predictive models for infectious diseases and the grand mean is the lowest bar.
Funder
Centers for Disease Control and Prevention
Division of Intramural Research, National Institute of Allergy and Infectious Diseases
National Science Foundation
Publisher
Public Library of Science (PLoS)
Reference32 articles.
1. An open challenge to advance probabilistic forecasting for dengue epidemics;MA Johansson;Proceedings of the National Academy of Sciences,2019
2. Drivers, dynamics, and control of emerging vector-borne zoonotic diseases;AM Kilpatrick;LANCET,2012
3. Predictive Models of Fish Species Distributions: A Note on Proper Validation and Chance Predictions.;JD Olden,2002