Research on Security Anomaly Detection for Big Data Platforms Based on Quantum Optimization Clustering

Abstract

Due to the explosive growth of data in the Internet, more and more applications are being deployed on Big Data platforms. However, as the scale of data continues to increase, the probability of anomalies in the platform is also increasing. However, traditional anomaly detection techniques cannot effectively handle the massive amount of historical data and can hardly meet the security requirements of big data platforms. In order to solve the above problems, this paper proposes a security anomaly detection method for big data platforms based on quantum optimization clustering. Firstly, a framework of big data platform anomaly detection system is designed based on distributed software architecture through Hadoop and Spark big data open source technology. The system achieves effective detection of network anomalies by collecting and analyzing big data platform server log data. Secondly, an offline anomaly detection algorithm based on quantum ant colony optimized affinity propagation clustering is designed for various anomalies mined from historical data. The bias parameters of the affinity propagation clustering are treated as individual ants to construct an ant colony, and the clustering accuracy is set as fitness. Finally, in order to improve the accuracy of the optimal path search of the ant colony, quantum bit encoding is applied to the ant colony position to refine the granularity of the individual ant colony position update. The experimental results show that the proposed method can effectively complete the anomaly clustering detection of massive data. With a reasonable threshold, the quantum ant colony–based affinity propagation clustering has high detection accuracy.