An Automated and Goal-Oriented Clustering Procedure

Abstract

AbstractClustering techniques are convenient tools for preparing and organizing unstructured and unclassified data. Depending on the data, they can be used to prepare for an analysis or to gain insight. However, choosing a clustering technique can be challenging when dealing with high-dimensional datasets. Most often, application requirements and data distribution need to be considered. Since clustering is defined as a complex problem to calculate, different algorithms may produce different results that meet the application's needs. This study presents an automated threshold-based and goal-oriented clustering procedure. It is based on the AutoML mechanism to estimate the most suitable hyperparameters according to predefined needs and can learn four clustering performance metrics thresholds for a given dataset. The significant advantages of this method are the automatic selection of clustering technique (i.e., partitional, hierarchical, density-based, or graph-based) and the ability to determine the output dynamically, according to predefined goals. We tested our method over four datasets and analyzed the results according to different goals. The results show that our method improved the silhouette score by 549.5% (from 0.105 to 0.682) compared to popular and commonly used K-means. Furthermore, clustering based on multiple metrics yielded more information than clustering by a single metric.