Retrospective evaluation of short-term forecast performance of ensemble sub-epidemic frameworks and other time-series models: The 2022-2023 mpox outbreak across multiple geographical scales, July 14th, 2022, through February 26th, 2023

Abstract

AbstractIn May 2022, public health officials noted an unprecedented surge in mpox cases in non-endemic countries worldwide. As the epidemic accelerated, multi-model forecasts of the epidemic’s trajectory were critical in guiding the implementation of public health interventions and determining policy. As the case levels have significantly decreased as of early September 2022, evaluating model performance is essential to advance the growing field of epidemic forecasting.Using laboratory-confirmed mpox case data from the Centers for Disease Control and Prevention (CDC) and Our World in Data (OWID) teams through the week of January 26th, 2023, we generated retrospective sequential weekly forecasts (e.g., 1-4-weeks) for Brazil, Canada, France, Germany, Spain, the United Kingdom, the USA, and at the global scale using models that require minimal input data including the auto-regressive integrated moving average (ARIMA), general additive model (GAM), simple linear regression (SLR), Facebook’s Prophet model, as well as the sub-epidemic wave (spatial-wave) andn-sub-epidemic modeling frameworks. We assess forecast performance using average mean squared error (MSE), mean absolute error (MAE), weighted interval score (WIS), 95% prediction interval coverage (95% PI coverage), and skill scores. Average Winkler scores were used to calculate skill scores for 95% PI coverage.Overall, then-sub-epidemic modeling framework outcompeted other models across most locations and forecasting horizons, with the unweighted ensemble model performing best across all forecasting horizons for most locations regarding average MSE, MAE, WIS, and 95% PI coverage. However, many locations had multiple models performing equally well for the average 95% PI coverage. Then-sub-epidemic and spatial-wave frameworks improved considerably in average MSE, MAE, and WIS, and Winkler scores (95% PI coverage) relative to the ARIMA model. Findings lend further support to sub-epidemic frameworks for short-term forecasting epidemics of emerging and re-emerging infectious diseases.SummaryIn the face of many unknowns (i.e., transmission, symptomology) posed by the unprecedented 2022-2023 mpox epidemic, near real-time short-term forecasts of the epidemic’s trajectory were essential in intervention implementation and guiding policy. As case levels continue to dissipate, evaluating the modeling strategies used in producing real-time forecasts is critical to refine and grow the field of epidemiological forecasting. Here, we systematically evaluate the performance of an ensemblen-sub-epidemic and related sub-epidemic wave (spatial-wave) modeling frameworks against ARIMA, GAM, Prophet, and SLR models in producing sequential retrospective weekly (1-4 week) forecasts of mpox cases for the highest burdened countries (i.e., Brazil, Canada, France, Germany, Spain, the United Kingdom, and the United States) and on a global scale. Overall, then-sub-epidemic framework outperformed all other models most frequently, followed closely in success by the spatial-wave framework, GAM, and ARIMA models regarding average MSE, MAE, and WIS metrics. Then-sub-epidemic unweighted model and spatial-wave framework performed best overall based on average 95% PI coverage, and we noted widespread success for both frameworks in average Winkler scores. The considerable success seen with both frameworks highlights the continued utility of sub-epidemic methodologies in producing short-term forecasts and their potential application to other epidemiologically different diseases.