Clustering validation by distribution hypothesis learning

Abstract

Abstract We present a new clustering validation technique named: "Hypothesis Learning". We build our method on three concepts: 1) clustering cohesion, 2) clustering dispersion and, 3) quality of hypothesis. The first two notions focus on individual clusters quality. We measure them using a classifier estimating the tightness and separation as a likelihood. The third notion evaluates the complexity of learning the clustering partition. Similar to cohesion and dispersion, we get a likelihood value. Next, we aggregate these three measures to find a single index reporting clustering quality. Our work's core is the use of learning algorithms as means to estimate these three indexes. In our experiments, we tested "Hypothesis Learning" with a fast classifier, K Nearest Neighbour (KNN). However, in the discussion of the method, we explore other classifiers like CART and Random Forest. Furthermore, we provide a novel approach from previous validation methods mixing supervised, unsupervised algorithms and stability concepts. For instance, our method is based on using clusters probabilities to calculate likelihoods. Also, we show how to regularize a classifier to handle overfit, thus making the use of stability optional. Finally, we present experimental results comparing our approach with a similar method and many other well-known clustering indexes.