Elicitability of Instance and Object Ranking

Abstract

Assessing the quality of a forecasting model crucially depends on a proper scoring rule or suitable loss function. As for point forecasts, the existence of a strictly consistent loss function that allows for a fair comparison of competing forecast models has to be guaranteed, which means that the corresponding statistical functional has to be elicitable. We consider instance and object ranking problems that intend to correctly predict the ordering of instances in a data set. A ranking prediction is naturally identified with a point forecast in the respective symmetric group, that is, the forecaster predicts one single permutation of the row indices. We show that, in the presence of ties, this strategy does not allow for strictly consistent scoring functions because of multiple true permutations. Those multiple optima cannot be entirely covered by a single point forecast, which causes all corresponding optima to be minimizers of standard scoring functions that operate on symmetric groups, so these scoring functions are not strictly consistent. As a remedy, we consider accurately accounting for ties. This is done by treating each configuration of clear orderings and ties as an additional category, which induces extended decision spaces with a clearly defined single optimum. Because these decision spaces are still finite, each type of instance ranking problem that we consider in this work and corresponding ranking functional, mapping into a symmetric group, can be identified with a certain classification problem and corresponding classification functional, mapping into one of our extended decision spaces, which is elicitable.