Abstract
AbstractA long noted difficulty when assessing calibration (or reliability) of forecasting systems is that calibration, in general, is a hypothesis not about a finite dimensional parameter but about an entire functional relationship. A calibrated probability forecast for binary events for instance should equal the conditional probability of the event given the forecast, whatever the value of the forecast. A new class of tests is presented that are based on estimating thecumulativedeviations from calibration. The supremum of those deviations is taken as a test statistic, and the asymptotic distribution of the test statistic is established rigorously. It turns out to be universal, provided the forecasts “look one step ahead” only, or in other words, verify at the next time step in the future. The new tests apply to various different forecasting problems and are compared with established approaches which work in a regression based framework. In comparison to those approaches, the new tests develop power against a wider class of alternatives. Numerical experiments for both artificial data as well as operational weather forecasting systems are presented, and possible extensions to longer lead times are discussed.
Publisher
Springer Science and Business Media LLC
Subject
Computational Theory and Mathematics,Statistics, Probability and Uncertainty,Statistics and Probability,Theoretical Computer Science
Reference25 articles.
1. Atger, F.: Estimation of the reliability of ensemble based probabilistic forecasts. Quater. J. Royal Meteorol. Soc. 130, 627–646 (2004)
2. Bierens, H.J.: A consistent conditional moment test of functional form. Econ. J. Econ. Soc. 58, 1443–1458 (1990)
3. Bröcker, J.: Probability forecasts. In: Jolliffe, I.T., Stephenson, D.B. (eds.) Forecast Verification; A practicioner’s Guide in Athmospheric Science, 2nd edn., pp. 119–139. John Wiley & Sons Ltd, Chichester (2012)
4. Bröcker J.: franz, a python library for statistical assessment of forecasts (release 1.0). GitHub, 2020. URL https://github.com/eirikbloodaxe/franz/releases/tag/v1.0
5. Bröcker, J.: Testing the reliability of forecasting systems. J. Appl. Stat. (2021). https://doi.org/10.1080/02664763.2021.1981833
Cited by
1 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献