Optimization of K-means Algorithm Base on MapReduce

Abstract

Abstract In the context of big data, the K-means clustering algorithm is sensitive to the initial clustering center, the detection and removal of outliers are difficult, and the data is tilted. The KMEANS-BRMS (K-means algorithm based on range mean and sampling) algorithm based on range mean and sampling is proposed. First, for the initial data set, the “range mean value method”(Mean Range method, MRM) is proposed to obtain the initial cluster centers. In order to eliminate the influence of outliers, a range threshold is set above and below the range mean, and all points under this threshold constitute an initial center. The set of points to be selected, in which the maximum and minimum distance criteria are adopted to select the initial clustering center, which effectively avoids the problem of the sensitivity of the initial center point and the influence of outliers caused by the random selection of the initial clustering center. Next, the BSA(Based on pond sampling and first adaptation algorithm) strategy of adapting algorithm for the first time to deal with the data skew problem in the MapReduce stage, and improve the clustering efficiency. Finally, combined with the MapReduce framework model, the data cluster center is mining in parallel to generate the final clustering result. Experiments show that the clustering results of the KMEANS-BRMS algorithm are more stable and effective, and at the same time it can more effectively improve the efficiency of parallel computing in the big data environment.