A novel data streaming method for detecting abnormal flows in distributed monitoring systems

Abstract

AbstractThis paper concentrates on the issue of detecting abnormal flows in distributed monitoring systems, which has many network management applications such as anomaly detection and traffic engineering. Collecting massive network traffic in real‐time remains a large challenge due to the limited system resource. Most existing approaches perform abnormal flow detection at one measurement point, while they cause large computation and memory overhead for recovering abnormal flows. In this paper, we propose a novel data streaming method that supports accurate abnormal flow detection with a low memory requirement. The key idea of our method is that each monitor compresses flow information to summary data structure, sends the generated data structure to the controller; then the controller aggregates the received data structures, recovers candidates of abnormal flows and estimates their size and change to find abnormal flows on the basis of the aggregated data structure. The experimental results based on real network traffic show that the proposed approach can detect up to 97% of abnormal flows with low memory and update requirements in comparison with related approaches.