Retrospective-Based Deep Q-Learning Method for Autonomous Pathfinding in Three-Dimensional Curved Surface Terrain

Abstract

Path planning in complex environments remains a challenging task for unmanned vehicles. In this paper, we propose a decoupled path-planning algorithm with the help of a deep reinforcement learning algorithm that separates the evaluation of paths from the planning algorithm to facilitate unmanned vehicles in real-time consideration of environmental factors. We use a 3D surface map to represent the path cost, where the elevation information represents the integrated cost. The peaks function simulates the path cost, which is processed and used as the algorithm’s input. Furthermore, we improved the double deep Q-learning algorithm (DDQL), called retrospective-double DDQL (R-DDQL), to improve the algorithm’s performance. R-DDQL utilizes global information and incorporates a retrospective mechanism that employs fuzzy logic to evaluate the quality of selected actions and identify better states for inclusion in the memory. Our simulation studies show that the proposed R-DDQL algorithm has better training speed and stability compared to the deep Q-learning algorithm and double deep Q-learning algorithm. We demonstrate the effectiveness of the R-DDQL algorithm under both static and dynamic tasks.