Distributed system anomaly detection using deep learning‐based log analysis

Abstract

AbstractAnomaly detection is a key step in ensuring the security and reliability of large‐scale distributed systems. Analyzing system logs through artificial intelligence methods can quickly detect anomalies and thus help maintenance personnel to maintain system security. Most of the current works only focus on the temporal or spatial features of distributed system logs, and they cannot sufficiently extract the global features of distributed system logs to achieve a good correct rate of anomaly detection. To further address the shortcomings of existing methods, this paper proposes a deep learning model with global spatiotemporal features to detect the presence of anomalies in distributed system logs. First, we extract semi‐structured log events from log templates and model them as natural language. In addition, we focus on the temporal characteristics of logs using the bidirectional long short‐term memory network and the spatial invocation characteristics of logs using the Transformer. Extensive experimental evaluations show the advantages of our proposed model for distributed system log anomaly detection tasks. The optimal F1‐Score on three open‐source datasets and our own collected distributed system datasets reach 98.04%, 94.34%, 88.16%, and 97.40%, respectively.