The Parallel Fuzzy C-Median Clustering Algorithm Using the Spark for the Big Data

Abstract

Abstract Big data for sustainable development is a global issue due to the explosive growth of data and according to the forecasting of International Data Corporation(IDC), the amount of data in the world will double every 18 months, and the Global Data-sphere is expected to more than double in size from 2022 to 2026. The analysis, processing, and storing of big data is a challenging research concern due to data imperfection, massive data size, computational difficulty, and lengthy evaluation time. Clustering is a fundamental technique in data analysis and data mining, and it becomes particularly challenging when dealing with big data due to the sheer volume, velocity, and variety of the data. When the size of the data is exceedingly enormous, clustering has a scalability issue that causes it to utilize more memory and take longer to process data. Big Data frameworks like Hadoop MapReduce and Spark are potent tools that provide an effective way to analyze huge datasets that are being processed by the Hadoop cluster. But Hadoop reads and writes data from the Hadoop Distributed File System (HDFS) for each iteration, which consumes considerable time. Apache Spark is one of the most widely used large-scale data processing engines due to its speed, low latency in-memory computing, and powerful analytics. Therefore, we develop a Parallel Fuzzy C-Median Clustering Algorithm Using the Spark for the Big Data that can handle large datasets while maintaining high accuracy and scalability. The algorithm employs a distance-based clustering approach to determine the similarity between data points and group them in combination with sampling and partitioning techniques. In the sampling phase, a representative subset of the dataset is selected, and in the partitioning phase, the data is partitioned into smaller subsets that can be clustered in parallel across multiple nodes. The suggested method, implemented in the Databricks cloud platform provides high clustering accuracy, as measured by clustering evaluation metrics such as the silhouette coefficient, cost function, partition index, and clustering entropy. The experimental results show that c = 5, which is consistent for cost function with the ideal silhouette coefficient of 1, is the optimal number of clusters for this dataset. For the validation of the proposed algorithm, a comparative study is done by implementing the other contemporary algorithms for the same dataset. The comparison analysis exhibits that our suggested approach outperforms the others, especially for computational time. The developed approach is the benchmarked with the existing methods such as MiniBatchKmeans, AffinityPropagation, SpectralClustering, Ward, OPTICS, and BRICH in terms of silhouette index and cost function.