Model-based deep reinforcement learning for active control of flow around a circular cylinder using action-informed episode-based neural ordinary differential equations

Abstract

To date, applications of deep reinforcement learning (DRL) to active flow control (AFC) have been largely achieved via model-free DRL wherein the agent's policy is optimized through direct interactions with the actual physical system represented by computational fluid dynamics solvers. However, high computational demands and tendency of numerical divergence can significantly compromise the effectiveness of model-free DRL as the Reynolds number increases. A model-based DRL paradigm, which utilizes neural ordinary differential equations (NODE) to develop an environment model through integration with dimensionality reduction, offers a promising way forward to overcome this problem. This study presents an inaugural application of NODE model-based DRL to control the vortex shedding process from a two-dimensional circular cylinder using two synthetic jet actuators at a freestream Reynolds number of 100. An action-informed episode-based NODE (AENODE) method is developed to overcome the error cascading effect caused by recursive predictions in the existing studies, which typically adopt a single-step prediction NODE (denoted as the time step-based NODE (TNODE) in this paper). Both the AENODE and TNODE methods are employed in this study, and they are amalgamated with three distinct feature extraction approaches, expert-placed velocity sensors, proper orthogonal decomposition, and autoencoders, to construct six low-dimensional dynamical models (LDMs) of the DRL environment. It is found that AENODE resulted in over 90% fewer prediction errors at the end of an episode than TNODE with all LDMs via effectively mitigating the accumulation of long-term prediction errors associated with the recursive use of TNODE, leading to a more robust convergence in training the agents throughout repeated runs. Furthermore, the model-based DRL with either AENODE or TNODE is capable of identifying very similar control strategies to that obtained by the model-free DRL. The AENODE agents achieved 66.2%–72.4% of the rewards obtained by the model-free DRL, whereas the TNODE agents attained merely 43.4%–54.7%, indicating that AENODE provides a more accurate modeling of environment dynamics in DRL. It is also shown that completing a model-based DRL task using either TNODE or AENODE utilized only 10% of the data size requiring either 14% or 33% of the total wall-clock time required by the model-free DRL, and the actual time required for training the agents within the environment model was less than 1% of that required by the model-free DRL. Therefore, the AENODE method developed in this work not only enables a significant saving in computational costs but also outperforms the TNODE method in training convergence and reward. It represents a novel low-dimensional dynamical modeling method tailored for model-based DRL, which would enable the DRL-aided AFC to be applied to more complex flow scenarios occurring at high Reynolds numbers.