LogEDL: Log Anomaly Detection via Evidential Deep Learning
-
Published:2024-08-12
Issue:16
Volume:14
Page:7055
-
ISSN:2076-3417
-
Container-title:Applied Sciences
-
language:en
-
Short-container-title:Applied Sciences
Author:
Duan Yunfeng1, Xue Kaiwen2, Sun Hao1, Bao Haotong2, Wei Yadong2, You Zhangzheng2, Zhang Yuantian2, Jiang Xiwei2, Yang Sangning2, Chen Jiaxing1, Duan Boya1, Ou Zhonghong2
Affiliation:
1. China Mobile Communications Group Co., Ltd., Beijing 102206, China 2. State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing 100876, China
Abstract
With advancements in digital technologies such as 5G communications, big data, and cloud computing, the components of network operation systems have become increasingly complex, significantly complicating system monitoring and maintenance. Correspondingly, automated log anomaly detection has become a crucial means to ensure stable network operation and protect networks from malicious attacks or failures. Conventional machine learning and deep learning methods assume consistent distributions between the training and testing data, adhering to a closed-set recognition paradigm. Nevertheless, in realistic scenarios, systems may encounter new anomalies that were not present in the training data, especially in log anomaly detection. Inspired by evidential learning, we propose a novel anomaly detector called LogEDL, which supervises the training of the model through an evidential loss function. Unlike traditional loss functions, the evidential loss function not only focuses on correct classification but also quantifies the uncertainty of predictions. This enhances the robustness and accuracy of the model in handling anomaly detection tasks while achieving functionality similar to open-set recognition. To evaluate the proposed LogEDL method, we conduct extensive experiments on three datasets, i.e., HDFS, BGL, and Thunderbird, to detect anomalous log sequences. The experimental results demonstrate that our proposed LogEDL achieves state-of-the-art performance in anomaly detection.
Funder
National Natural Science Foundation of China
Reference36 articles.
1. A survey on automated log analysis for reliability engineering;He;ACM Comput. Surv. (CSUR),2021 2. Liu, Y., Zhang, X., He, S., Zhang, H., Li, L., Kang, Y., Xu, Y., Ma, M., Lin, Q., and Dang, Y. (2022, January 25–29). Uniparser: A unified log parser for heterogeneous log data. Proceedings of the ACM Web Conference 2022, Lyon, France. 3. Ma, L., Yang, W., Xu, B., Jiang, S., Fei, B., Liang, J., Zhou, M., and Xiao, Y. (2024, January 14–20). KnowLog: Knowledge Enhanced Pre-trained Language Model for Log Understanding. Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, Lisbon, Portugal. 4. Lin, Q., Zhang, H., Lou, J.G., Zhang, Y., and Chen, X. (2016, January 14–22). Log clustering based problem identification for online service systems. Proceedings of the 38th International Conference on Software Engineering Companion, Austin, TX, USA. 5. Xie, Y., Zhang, H., Zhang, B., Babar, M.A., and Lu, S. (2021, January 22–25). Logdp: Combining dependency and proximity for log-based anomaly detection. Proceedings of the Service-Oriented Computing: 19th International Conference, ICSOC 2021, Virtual Event. Proceedings 19.
|
|