Statistical Evaluation of Different Surface Precipitation-Type Algorithms and Its Implications for NWP Prediction and Operational Decision-Making

Abstract

Abstract Several new precipitation-type algorithms have been developed to improve NWP predictions of surface precipitation type during winter storms. In this study, we evaluate whether it is possible to objectively declare one algorithm as superior to another through comparison of three precipitation-type algorithms when validated using different techniques. The apparent skill of the algorithms is dependent on the choice of performance metric—algorithms can have high scores for some metrics and poor scores for others. It is also possible for an algorithm to have high skill at diagnosing some precipitation types and poor skill with others. Algorithm skill is also highly dependent on the choice of verification data/methodology. Just by changing what data are considered “truth,” we were able to substantially change the apparent skill of all algorithms evaluated herein. These findings suggest an objective declaration of algorithm “goodness” is not possible. Moreover, they indicate that the unambiguous declaration of superiority is difficult, if not impossible. A contributing factor to algorithm performance is uncertainty of the microphysical processes that lead to phase changes of falling hydrometeors, which are treated differently by each algorithm, thus resulting in different biases in near −0°C environments. These biases are evident even when algorithms are applied to ensemble forecasts. Hence, a multi-algorithm approach is advocated to account for this source of uncertainty. Although the apparent performance of this approach is still dependent on the choice of performance metric and precipitation type, a case-study analysis shows it has the potential to provide better decision support than the single-algorithm approach. Significance Statement Many investigators are developing new-and-improved algorithms to diagnose the surface precipitation type in winter storms. Whether these algorithms can be declared as objectively superior to existing strategies is unknown. Herein, we evaluate different methods to measure algorithm performance to assess whether it is possible to state one algorithm is superior to another. The results of this study suggest such claims are difficult, if not impossible, to make, at least not for the algorithms considered herein. Because algorithms can have certain biases, we advocate a multi-algorithm approach wherein multiple algorithms are applied to forecasts and a probabilistic prediction of precipitation type is generated. The potential value of this is demonstrated through a case-study analysis that shows promise for enhanced decision support.