A Versatile Framework for Evaluating Ranked Lists in Terms of Group Fairness and Relevance

Abstract

We present a simple and versatile framework for evaluating ranked lists in terms of Group Fairness and Relevance, in which the groups (i.e., possible attribute values) can be either nominal or ordinal in nature. First, we demonstrate that when our framework is applied to a binary hard group membership setting, our Group Fairness and Relevance (GFR) measures can easily quantify the overall polarity of each ranked list. Second, by utilising an existing diversified search test collection and treating each intent as an attribute value, we demonstrate that our framework can also handle soft group membership and that the GFR measures are highly correlated with a diversified information retrieval (IR) measure in this context as well. Third, using real data from a Japanese local search service, we demonstrate how our framework enables researchers to study intersectional group fairness based on multiple attribute sets. We also show that the similarity function for comparing the achieved and target distributions over the attribute values should be chosen carefully when the attribute values are ordinal. For such situations, our recommendation is to use multiple similarity functions with our framework: for example, one based on Jensen-Shannon Divergence (which disregards the ordinal nature of the groups) and another based on Root Normalised Order-aware Divergence (which has been designed specifically for handling ordinal groups). In addition, we highlight the fundamental differences between our framework and Attention-Weighted Rank Fairness (AWRF), a group fairness measure used at the TREC Fair Ranking Track.