Policy transfer of reinforcement learning-based flow control: From two- to three-dimensional environment

Abstract

In the current paper, the zero-mass synthetic jet flow control combined with a proximal policy optimization (PPO) algorithm in deep reinforcement learning is constructed, and a policy transfer strategy which is trained in two-dimensional (2D) environment and migrated to three-dimensional (3D) environment is proposed and analyzed. By policy, we mean the flow control strategy of the agent learned by interacting with environment through deep reinforcement learning (DRL) algorithm. Through comprehensive evaluations of vortex separation in the cylindrical boundary layer and wake region at different Reynolds (Re) numbers, the PPO model trained in the 2D environment can reduce the drag coefficient by approximately 6.3%, 18.6%, and 23.7% at Re = 100, 200, and 300, respectively, when the spanwise length of the 3D environment is equal to the cylinder's diameter. Moreover, when the spanwise length is three times the diameter, the drag reduction capability is about 5.8%, 15.4%, and 13.1% at the three Re numbers, respectively. Additionally, the PPO model trained in the 2D environment also demonstrated outstanding migration learning capability in a new 3D flow field environment with varying Re numbers, successfully suppressing vortex shedding and reducing drag coefficient. Furthermore, the results illustrate that the model trained at high Re numbers could still reduce the drag coefficient in the 3D environment with low Re numbers, while the model trained at low Re numbers was not as effective at achieving drag reduction in the environments under high Re numbers. Overall, the proposed policy transfer strategy has been proven to be an effective method applying DRL agent trained in 2D flow to a new 3D environment.