Distribution fairness in Internet-scale networks

Abstract

We address the issue of measuring distribution fairness in Internet-scale networks. This problem has several interesting instances encountered in different applications, ranging from assessing the distribution of load between network nodes for load balancing purposes, to measuring node utilization for optimal resource exploitation, and to guiding autonomous decisions of nodes in networks built with market-based economic principles. Although some metrics have been proposed, particularly for assessing load balancing algorithms, they fall short. We first study the appropriateness of various known and previously proposed statistical metrics for measuring distribution fairness. We put forward a number of required characteristics for appropriate metrics. We propose and comparatively study the appropriateness of the Gini coefficient ( G ) for this task. Our study reveals as most appropriate the metrics of G , the fairness index ( FI ), and the coefficient of variation ( C V ) in this order. Second, we develop six distributed sampling algorithms to estimate metrics online efficiently, accurately, and scalably. One of these algorithms ( 2-PRWS ) is based on two effective optimizations of a basic algorithm, and the other two (the sequential sampling algorithm, LBS-HL , and the clustered sampling one, EBSS ) are novel, developed especially to estimate G . Third, we show how these metrics, and especially G , can be readily utilized online by higher-level algorithms, which can now know when to best intervene to correct unfair distributions (in particular, load imbalances). We conclude with a comprehensive experimentation which comparatively evaluates both the various proposed estimation algorithms and the three most appropriate metrics ( G , C V , and FI ). Specifically, the evaluation quantifies the efficiency (in terms of number of the messages and a latency indicator), precision, and accuracy achieved by the proposed algorithms when estimating the competing fairness metrics. The central conclusion is that the proposed metric, G , can be estimated with a small number of messages and latency, regardless of the skew of the underlying distribution.