Diagnosing the Relative Impact of “Sneaks,” “Phantoms,” and Volatility in Sequences of Lagged Ensemble Probability Forecasts with a Simple Dynamic Decision Model

Abstract

Abstract Monte Carlo simulation of sequences of lagged ensemble probability forecasts is undertaken using Markov transition law estimated from a reforecast ensemble. A simple three-state, three-action dynamic decision model is then applied to the Monte Carlo sequence realizations using a basket of cost functions, and the resulting expense incurred by the decision model is conditioned upon the structure of the sequence realizations. Findings show that the greatest average expense is incurred by “sneak” and “volatile” sequence structures, which are structures characterized by large and rapid increases in event probability at short lag times. These findings are simple quantitative illustration of the adage that large run-to-run variability of forecasts can be troublesome to a decision maker. The experiments also demonstrate how even small improvements in the amount of advance warning of an event can translate into a substantial reduction in decision expense. In general, the conditioned decision expense is found to be sensitive to sequence structure for a given cost function, to the parameters of a given cost function, and to the choice of cost function itself.