AS-Parser: Log Parsing Based on Adaptive Segmentation

Abstract

System logs have long been recognized as valuable data for analyzing and diagnosing system failures. One fundamental task of log processing is to convert unstructured logs into structured logs through log parsing. All previous log parsing approaches follow a general framework that first segments each log into a token sequence and then computes similarity between two sequences. However, all existing approaches share the common drawback: the flat segmentation with fixed delimiters fails to understand the structural information of logs, which causes low parsing accuracy. To address this problem, we propose a novel log parsing approach, AS-Parser. Our approach introduces a hierarchical log segmentation mechanism that can adaptively segment logs into a tree structure. It can automatically recognize the appropriate delimiters and capture the common structural information. Moreover, we propose three improvements that enhance both the effectiveness and efficiency of our approach. On the public benchmark, AS-Parser performs best on 14 out of 16 datasets, with an average parsing accuracy of 0.943, far exceeding existing approaches.