Information-Based Probabilistic Verification Scores for Two-Dimensional Ensemble Forecast Data: A Madden–Julian Oscillation Index Example

Abstract

Abstract Probabilistic forecasting is a common activity in many fields of the Earth sciences. Assessing the quality of probabilistic forecasts—probabilistic forecast verification—is therefore an essential task in these activities. Numerous methods and metrics have been proposed for this purpose; however, the probabilistic verification of vector variables of ensemble forecasts has received less attention than others. Here we introduce a new approach that is applicable for verifying ensemble forecasts of continuous, scalar, and two-dimensional vector data. The proposed method uses a fixed-radius near-neighbors search to compute two information-based scores, the ignorance score (the logarithmic score) and the information gain, which quantifies the skill gain from the reference forecast. Basic characteristics of the proposed scores were examined using idealized Monte Carlo simulations. The results indicated that both the continuous ranked probability score (CRPS) and the proposed score with a relatively small ensemble size (<25) are not proper in terms of the forecast dispersion. The proposed verification method was successfully used to verify the Madden–Julian oscillation index, which is a two-dimensional quantity. The proposed method is expected to advance probabilistic ensemble forecasts in various fields. Significance Statement In the Earth sciences, stochastic future states are estimated by solving a large number of forecasts (called ensemble forecasts) based on physical equations with slightly different initial conditions and stochastic parameters. The verification of probabilistic forecasts is an essential part of forecasting and modeling activity in the Earth sciences. However, there is no information-based probabilistic verification score applicable for vector variables of ensemble forecasts. The purpose of this study is to introduce a novel method for verifying scalar and two-dimensional vector variables of ensemble forecasts. The proposed method offers a new approach to probabilistic verification and is expected to advance probabilistic ensemble forecasts in various fields.