Evaluation of individual and ensemble probabilistic forecasts of COVID-19
mortality in the US
Author:
Cramer Estee YORCID, Ray Evan L, Lopez Velma KORCID, Bracher JohannesORCID, Brennen Andrea, Castro Rivadeneira Alvaro J, Gerding Aaron, Gneiting Tilmann, House Katie H, Huang Yuxin, Jayawardena Dasuni, Kanji Abdul H, Khandelwal Ayush, Le Khoa, Mühlemann Anja, Niemi JaradORCID, Shah Apurv, Stark Ariane, Wang Yijin, Wattanachit Nutcha, Zorn Martha W, Gu Youyang, Jain Sansiddh, Bannur Nayana, Deva Ayush, Kulkarni Mihir, Merugu SrujanaORCID, Raval Alpan, Shingi Siddhant, Tiwari Avtansh, White JeromeORCID, Abernethy Neil FORCID, Woody SpencerORCID, Dahan Maytal, Fox Spencer, Gaither Kelly, Lachmann Michael, Meyers Lauren Ancel, Scott James G, Tec Mauricio, Srivastava AjiteshORCID, George Glover EORCID, Cegan Jeffrey CORCID, Dettwiller Ian D, England William P, Farthing Matthew WORCID, Hunter Robert HORCID, Lafferty Brandon, Linkov Igor, Mayo Michael LORCID, Parno Matthew DORCID, Rowland Michael AORCID, Trump Benjamin DORCID, Zhang-James YanliORCID, Chen Samuel, Faraone Stephen VORCID, Hess Jonathan, Morley Christopher PORCID, Salekin Asif, Wang Dongliang, Corsetti Sabrina MORCID, Baer Thomas MORCID, Eisenberg Marisa CORCID, Falb Karl, Huang Yitao, Martin Emily TORCID, McCauley Ella, Myers Robert L, Schwarz TomORCID, Sheldon DanielORCID, Gibson Graham CaseyORCID, Yu RoseORCID, Gao LiyaoORCID, Ma YianORCID, Wu DongxiaORCID, Yan Xifeng, Jin Xiaoyong, Wang Yu-Xiang, Chen YangQuanORCID, Guo LihongORCID, Zhao YantingORCID, Gu QuanquanORCID, Chen JinghuiORCID, Wang LingxiaoORCID, Xu PanORCID, Zhang WeitongORCID, Zou DifanORCID, Biegel Hannah, Lega JocelineORCID, McConnell Steve, Nagraj VPORCID, Guertin Stephanie LORCID, Hulme-Lowe Christopher, Turner Stephen DORCID, Shi Yunfeng, Ban Xuegang, Walraven Robert, Hong Qi-JunORCID, Kong Stanley, van de Walle AxelORCID, Turtle James AORCID, Ben-Nun MichalORCID, Riley StevenORCID, Riley PeteORCID, Koyluoglu UgurORCID, DesRoches David, Forli Pedro, Hamory Bruce, Kyriakides Christina, Leis Helen, Milliken John, Moloney Michael, Morgan James, Nirgudkar Ninad, Ozcan Gokce, Piwonka Noah, Ravi Matt, Schrader Chris, Shakhnovich Elizabeth, Siegel Daniel, Spatz Ryan, Stiefeling Chris, Wilkinson Barrie, Wong Alexander, Cavany SeanORCID, España GuidoORCID, Moore SeanORCID, Oidtman RachelORCID, Perkins AlexORCID, Kraus DavidORCID, Kraus AndreaORCID, Gao Zhifeng, Bian Jiang, Cao Wei, Ferres Juan Lavista, Li Chaozhuo, Liu Tie-Yan, Xie Xing, Zhang Shun, Zheng Shun, Vespignani AlessandroORCID, Chinazzi MatteoORCID, Davis Jessica TORCID, Mu KunpengORCID, y Piontti Ana PastoreORCID, Xiong XinyueORCID, Zheng AndrewORCID, Baek JackieORCID, Farias VivekORCID, Georgescu AndreeaORCID, Levi RetsefORCID, Sinha DeekshaORCID, Wilde JoshuaORCID, Perakis Georgia, Bennouna Mohammed Amine, Nze-Ndong David, Singhvi Divya, Spantidakis Ioannis, Thayaparan Leann, Tsiourvas Asterios, Sarker ArnabORCID, Jadbabaie AliORCID, Shah DevavratORCID, Penna Nicolas DellaORCID, Celi Leo AORCID, Sundar Saketh, Wolfinger RussORCID, Osthus DaveORCID, Castro LaurenORCID, Fairchild GeoffreyORCID, Michaud Isaac, Karlen DeanORCID, Kinsey Matt, Mullany Luke C., Rainwater-Lovett Kaitlin, Shin Lauren, Tallaksen Katharine, Wilson Shelby, Lee Elizabeth CORCID, Dent Juan, Grantz Kyra HORCID, Hill Alison LORCID, Kaminsky Joshua, Kaminsky Kathryn, Keegan Lindsay TORCID, Lauer Stephen AORCID, Lemaitre Joseph CORCID, Lessler JustinORCID, Meredith Hannah RORCID, Perez-Saez JavierORCID, Shah Sam, Smith Claire P, Truelove Shaun AORCID, Wills Josh, Marshall Maximilian, Gardner LaurenORCID, Nixon Kristen, Burant John C., Wang LilyORCID, Gao LeiORCID, Gu ZhilingORCID, Kim MyungjinORCID, Li XinyiORCID, Wang GuannanORCID, Wang YueyingORCID, Yu ShanORCID, Reiner Robert C, Barber Ryan, Gakidou Emmanuela, Hay Simon I., Lim Steve, Murray Chris J.L., Pigott David, Gurung Heidi L, Baccam Prasith, Stage Steven A, Suchoski Bradley T, Prakash B. AdityaORCID, Adhikari BijayaORCID, Cui Jiaming, Rodríguez AlexanderORCID, Tabassum AnikaORCID, Xie Jiajia, Keskinocak PinarORCID, Asplund John, Baxter ArdenORCID, Oruc Buse EylulORCID, Serban Nicoleta, Arik Sercan O, Dusenberry Mike, Epshteyn Arkady, Kanal Elli, Le Long T, Li Chun-Liang, Pfister Tomas, Sava Dario, Sinha Rajarishi, Tsai Thomas, Yoder Nate, Yoon Jinsung, Zhang Leyou, Abbott SamORCID, Bosse Nikos IORCID, Funk SebastianORCID, Hellewell Joel, Meakin Sophie RORCID, Sherratt Katharine, Zhou Mingyuan, Kalantari Rahi, Yamana Teresa KORCID, Pei SenORCID, Shaman Jeffrey, Li Michael LORCID, Bertsimas Dimitris, Lami Omar SkaliORCID, Soni Saksham, Bouardi Hamza TaziORCID, Ayer Turgay, Adee Madeline, Chhatwal Jagpreet, Dalgic Ozden O, Ladd Mary A, Linas Benjamin P, Mueller Peter, Xiao Jade, Wang Yuanjia, Wang Qinxia, Xie Shanghong, Zeng Donglin, Green Alden, Bien Jacob, Brooks Logan, Hu Addison J, Jahja Maria, McDonald Daniel, Narasimhan Balasubramanian, Politsch Collin, Rajanala Samyak, Rumack Aaron, Simon Noah, Tibshirani Ryan J, Tibshirani Rob, Ventura Valerie, Wasserman Larry, O’Dea Eamon BORCID, Drake John MORCID, Pagano Robert, Tran Quoc TORCID, Tung Ho Lam Si, Huynh Huong, Walker Jo WORCID, Slayton Rachel BORCID, Johansson Michael AORCID, Biggerstaff MatthewORCID, Reich Nicholas GORCID
Abstract
Abstract
Short-term probabilistic forecasts of the trajectory of the COVID-19
pandemic in the United States have served as a visible and important
communication channel between the scientific modeling community and both the
general public and decision-makers. Forecasting models provide specific,
quantitative, and evaluable predictions that inform short-term decisions such as
healthcare staffing needs, school closures, and allocation of medical supplies.
Starting in April 2020, the US COVID-19 Forecast Hub (https://covid19forecasthub.org/) collected, disseminated,
and synthesized tens of millions of specific predictions from more than 90
different academic, industry, and independent research groups. A multi-model
ensemble forecast that combined predictions from dozens of different research
groups every week provided the most consistently accurate probabilistic
forecasts of incident deaths due to COVID-19 at the state and national level
from April 2020 through October 2021. The performance of 27 individual models
that submitted complete forecasts of COVID-19 deaths consistently throughout
this year showed high variability in forecast skill across time, geospatial
units, and forecast horizons. Two-thirds of the models evaluated showed better
accuracy than a naïve baseline model. Forecast accuracy degraded as models made
predictions further into the future, with probabilistic error at a 20-week
horizon 3-5 times larger than when predicting at a 1-week horizon. This project
underscores the role that collaboration and active coordination between
governmental public health agencies, academic modeling teams, and industry
partners can play in developing modern modeling capabilities to support local,
state, and federal response to outbreaks.
Significance Statement
This paper compares the probabilistic accuracy of short-term forecasts
of reported deaths due to COVID-19 during the first year and a half of the
pandemic in the US. Results show high variation in accuracy between and
within stand-alone models, and more consistent accuracy from an ensemble
model that combined forecasts from all eligible models. This demonstrates
that an ensemble model provided a reliable and comparatively accurate means
of forecasting deaths during the COVID-19 pandemic that exceeded the
performance of all of the models that contributed to it. This work
strengthens the evidence base for synthesizing multiple models to support
public health action.
Publisher
Cold Spring Harbor Laboratory
Cited by
31 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献
|
|