P-DRL: A Framework for Multi-UAVs Dynamic Formation Control under Operational Uncertainty and Unknown Environment

Abstract

Unmanned aerial vehicle (UAV) formation flying is an efficient and economical operation mode for air transportation systems. To improve the effectiveness of synergetic formation control for UAVs, this paper proposes a pairwise conflict resolution approach for UAV formation through mathematical analysis and designs a dynamic pairing and deep reinforcement learning framework (P-DRL formation control framework). Firstly, a new pairwise UAV formation control theorem is proposed, which breaks down the multi-UAVs formation control problem into multiple sequential control problems involving UAV pairs through a dynamic pairing algorithm. The training difficulty of Agents that only control each pair (two UAVs) is lower compared to controlling all UAVs directly, resulting in better and more stable formation control performance. Then, a deep reinforcement learning model for a UAV pair based on the Environment–Agent interaction is built, where segmented reward functions are designed to reduce the collision possibility of UAVs. Finally, P-DRL completes the formation control task of the UAV fleet through continuous pairing and Agent-based pairwise formation control. The simulations used the dynamic pairing algorithm combined with the DRL architectures of asynchronous advantage actor–critic (P-A3C), actor–critic (P-AC), and double deep q-value network (P-DDQN) to achieve synergetic formation control. This approach yielded effective control results with a strong generalization ability. The success rate of controlling dense, fast, and multi-UAV (10–20) formations reached 96.3%, with good real-time performance (17.14 Hz).