When are predictions useful? a new method for evaluating epidemic forecasts

Abstract

AbstractWe introduce the Weighted Contextual Interval Score (WCIS), a new method for evaluating the performance of short-term interval-form forecasts. The WCIS provides a pragmatic utility-based characterization of probabilistic predictions, developed in response to the challenge of evaluating forecast performances in the turbulent context of the COVID-19 pandemic. Current widely-used scoring techniques generally fall into two groups: those that generate an individually interpretable metric, and those that generate a comparable and aggregable metric. The WCIS harmonizes these attributes, resulting in a normalized score that is nevertheless intuitively representative of the in-situ quality of individual forecasts. This method is expressly intended to enable practitioners and policy-makers who may not have expertise in forecasting but are nevertheless essential partners in epidemic response to use and provide insightful analysis of predictions. In this paper, we detail the methodology of the WCIS and demonstrate its utility in the context of US state-level COVID-19 predictions.