A novel subspace outlier detection method by entropy-based clustering algorithm

Abstract

Abstract Subspace outlier detection has emerged as a practical approach for outlier detection. Classical full space outlier detection methods become ineffective in high dimensional data due to the “curse of dimensionality”. Subspace outlier detection methods have great potential to overcome the problem. However, the challenge becomes how to determine which subspaces to be used for outlier detection among a huge number of all subspaces. In this paper, firstly, we propose an intuitive definition of outliers in subspaces. We study the desirable properties of subspaces for outlier detection and investigate the metrics for those properties.Then, a novel subspace outlier detection algorithm with a statistical foundation is proposed. Our method adopts an entropy-based clustering algorithm to yield interesting subspace for outlier detection.Outliers are then detected by comparing outlierness of its neighbors, and the method only makes use of a small number of most interesting subspaces for outlier detection. We show by experiments that outliers discovered in a very small number of most interesting subspaces achieves remarkable higher accuracy than in the full space, and that the proposed method outperforms competing subspace outlier detection approaches on real world data sets.