A “Meta” Analysis of the Fractions Skill Score: The Limiting Case and Implications for Aggregation

Abstract

AbstractThe fractions skill score (FSS) is arguably one of the most popular spatial verification metrics in use today. The fraction of grid points exceeding a threshold within a forecast and observed field neighborhood are examined to compute a score. By definition a perfect forecast has an FSS of 1, and a “no skill” forecast has a score of 0. It is shown that the denominator defines the score’s characteristics. The FSS is undefined for instances in which both the forecast and the observed field do not exceed a threshold. In the limiting case, the FSS for a perfect null (zero) forecast is also undefined, unless a threshold of 0 is used, in which case it would be 1 (i.e., perfect). Furthermore the FSS is 0 if either the forecast or the observed field does not exceed a threshold. This symmetry means it cannot differentiate between what are traditionally referred to as false alarms or misses. Additional supplementary information is required. The FSS is greater than 0 if and only if there are values exceeding a given threshold in both the forecast and the observed field. The magnitude of an overall score computed over many forecasts is sensitive to the pooling method. Zero scores are nontrivial. Excluding them implies excluding all situations associated with false alarms or misses. Omitting near-zero scores is a more credible decision, but only if it can be proven that these are related to spurious artifacts in the observed field. To avoid ambiguity the components of the FSS should be aggregated separately for computing an overall score for most applications and purposes.