UIFSS: An aid for univariate data with large missing gap in IoT applications

Abstract

In the realm of Internet of Things (IoT) sensor data, missing patterns often occur due to sensor glitches and communication problems. Conventional missing data imputation methods struggle to handle multiple missing patterns, as they fail to fully leverage the available data as well as partially imputed data. To address this challenge, we propose a novel approach called Univariate data Imputation using Fast Similarity Search (UIFSS). The proposed method solved the missing data problem of IoT data using fast similarity search that can suits different patterns of missingness. Exploring similarities between data elements, a problem known as all-pairs-similarity-search, has been extensively studied in fields like text analysis. Surprisingly, applying this concept to time series subsequences hasn’t seen much progress, likely due to the complexity of the task. Even for moderately sized datasets, the traditional approach can take a long time, and common techniques to speed it up only help a bit. Notably, for very large datasets, our algorithm can be easily adapted to produce high-quality approximate results quickly. UIFSS consists of two core components:Sensor sorting with Similar Node Clustering (SSNC) and Imputation Estimator using Fast Similarity Search(IEFSS). The SSNC, encompassing missing sensor sorting depending on their entropy to guide the imputation process. Subsequently, IEFSS uses global similar sensors and captures local region volatility, prioritizing data preservation while improving accuracy through z-normalized query based similarity search. Through experiments on simulated and bench mark datasets, UIFSS outperforms existing methods across various missing patterns. This approach offers a promising solution for handling missing IoT sensor data and with improved imputation accuracy.