Attention and masking embedded ensemble reinforcement learning for smart energy optimization and risk evaluation under uncertainties

Abstract

Integrating residential-level photovoltaic energy generation and energy storage for the on-grid system is essential to reduce electricity use for residential consumption from the grid. However, reaching a reliable and optimal control policy is highly challenging due to the intrinsic uncertainties in the renewable energy sources and fluctuating demand profile. In this work, we proposed and designed an ensemble deep reinforcement learning (DRL) algorithm combined with risk evaluation to solve the energy optimization problem under uncertainties. Meanwhile, the attention and masking layer, the state-of-the-art natural language processing techniques, were incorporated into the algorithm to handle the issue of hard constraints, which are frequently encountered in the renewable energy optimization problem. To the best of our knowledge, this work is the first attempt to tackle the energy optimization problem under uncertainty using a scenario-based ensemble DRL approach with a risk evaluation. Through a well-designed single household microgrid energy management system, we found that the attention and masking layer played a crucial role in fulfilling the hard constraint. The ensemble DRL with the increased number of agents showed a significantly improved energy management policy leading to ∼75% of the cost reduction compared with those obtained by using conventional DRL with a single agent. The risk evaluation revealed that the current ensemble DRL approach possessed a high-risk/high-profit feature, which could be significantly improved by designing a risk-aware reward function in future investigations.