Deep Q-Network with Reinforcement Learning for Fault Detection in Cyber-Physical Systems

Abstract

Cyber-Physical System (CPS) is an integration of physical components like actuators, sensors and various types of equipment with the Internet possessing computational ability for efficient communication. A Heterogeneous Independent Network (HINT) is a realistic model that is used for the analysis of inter-dependability between the power grid and communications network. In the traditional Deep [Formula: see text]-Learning method, action needs to be stored in the [Formula: see text] table for the prediction. In real case studies, many state and action values affect the performance of the model. Existing Deep [Formula: see text]-Network (DQN) model generates all possible actions for the [Formula: see text]-values and this involves the generation of excessive information that causes the model to overfit. In this research, the Neural Network is applied to estimate the state–action in the DQN and to store the particular state–action value instead of storing all the state–action values as followed in the traditional method. The HINT model provides realistic failure propagation in the network and its state–action value overfits the existing DQN method due to the presence of more information. The proposed DQN with reinforcement learning stores selected state–action values in the [Formula: see text] tables and eliminates irrelevant information that helps to increase the accuracy and reduce the computational time. The DQN with reinforcement learning is applied to adaptively learn the system to select the optimal action in a continuous interaction with a stochastic environment. The proposed DQN model involves the application of reward function to store state–action value with higher probability based on prediction and eliminates other state–action values. Features such as intra-degree, inter-betweenness, substation-betweenness, relay-betweenness and feature vector are extracted and given as input to the DQN to characterize the critical nodes. The proposed DQN method is evaluated on the HINT network and synthetic network to analyze its efficiency in fault detection. The result shows that the HINT network has a lower prediction error compared to the existing Deep Neural Network (DNN) method. The proposed DQN and LSTM models have accuracies of 98% and 93% in fault prediction, respectively.