An enhanced Hidden Semi-Markov model for outlier detection in multivariate datasets

Abstract

Outlier detection in multivariate data is one of the critical challenges in preprocessing phase. Many outlier detection methods have been emerged for the past few years to perform outlier detection efficiently in multivariate datasets. The prediction accuracy cannot be improved without proper outlier analysis and the prediction model might not confirm the expected behavior. The generation of huge data in real time applications makes the outlier detection process more crucial and challenging. Most of the currently available detection methods are based on mean and covariance that are not suitable for handling large volume of datasets, they are suitable for handlind static data and simple data to detect outliers. They cannot cope up with large scale data. So, there is a need for an efficient outlier detection model to detect the outliers in multivariate datasets. The primary objective of this research work is to develop a robust model for outlier detection in multivariate data. To achieve this, the work proposed an enhanced Hidden Semi-Markov Model (HSMM) which allows arbitrary time distribution in its states to detect outliers. The proposed work utilized six benchmark datasets and the performance is compared with several outlier detection algorithms such as HMM, iForest, FastABOD, and Expose. The work achieves 98.2 % of accuracy which is significantly better for detecting outliers in multivariate dataset. The proposed work improvised the percentage of acheivements between 2% to 25% than the currently available models.. The experimental analysis shows that the proposed model performs well than the currently available models in terms of accuracy, and receiver operation curve (ROC).