Covert fault detection with imbalanced data using an improved autoencoder for optical networks

Abstract

Covert faults are characterized by the performance parameters falling within the normal range, without any observable abnormalities. These types of faults pose a significant risk as they present no apparent warning signs of potential danger. Therefore, it is crucial to establish an efficient covert fault detection method to ensure the reliable and stable operation of optical networks. Data-driven technology, which reveals the internal relations and data patterns between the historical data by mining and analyzing the historical data, offers a new perspective for covert fault detection. However, equipment failures are extremely rare in real optical network systems, and the data imbalance of covert fault samples poses a challenge for standard machine learning classifiers in learning precise decision boundaries. To address this challenge, we propose a fault detection scheme based on an improved autoencoder for covert fault detection under data imbalance. The designed covert fault detection model exclusively utilizes normal samples during training and remains unaffected by data imbalance. Specifically, the model is specifically designed according to a number of encoder and decoder components to learn the normal sample data patterns in the latent space and detect covert faults based on the reconstruction errors in that space. To validate the proposed scheme, we conducted experiments using actual backbone data. According to the results, the detection accuracy and F1 score of the designed model on the test set were 0.9811 and 0.9527, and the false negative and false positive rates were 0.0026 and 0.0227, respectively. Furthermore, the visualization of the latent space reconstruction error principle for detecting covert faults was implemented using the principal component analysis dimension reduction and scatter plots.