Pooling Rankings to Obtain a Set of Scores for a Composite Indicator of Erasmus + Mobility Effects

Abstract

AbstractIn this paper, we study how to assign weights to a set of evaluations obtained at the end of an international mobility experience in order to aggregate them into a composite indicator. The mobility experience was evaluated by three categories of actor: the participant; the school or company sending the participant; and the school or company hosting the participant. We estimated the weights starting from the assessors’ mutual evaluations of the beneficiaries of the mobility experiences. In particular, the aim of the paper was to compare two strategies for estimating the weights: (1) a weighted function of the univariate rank distribution of frequencies; and (2) the normalised elements of the first eigenvector of the dominance matrix computed by mediating the actors’ dominance matrices derived from the rankings of mobility beneficiaries. Variants of the two strategies were also introduced. Even though each strategy had different assumptions, the analyses produced several important findings. First, the optimum weighting model depends on the loss function used to evaluate the quality of the results. In particular a between-ranking variability function favours both univariate and unweighted multivariate models, while a bias-based function favours weighted multivariate models. Second, in both univariate and multivariate analyses, the application of rank-order-centroid and rank-reciprocal rules give more accurate results than both linear and exponential rules.