Reinforcement Learning Based Relay Selection for Underwater Acoustic Cooperative Networks

Abstract

In the complex and dynamically varying underwater acoustic (UWA) channel, cooperative communication can improve throughput for UWA sensor networks. In this paper, we design a reasonable relay selection strategy for efficient cooperation with reinforcement learning (RL), considering the characteristics of UWA channel variation and long transmission delay. The proposed scheme establishes effective state and reward expression to better reveal the relationship between RL and UWA environment. Meanwhile, simulated annealing (SA) algorithm is integrated with RL to improve the performance of relay selection, where exploration rate of RL is dynamically adapted by SA optimization through the temperature decline rate. Furthermore, the fast reinforcement learning (FRL) strategy with pre-training process is proposed for practical UWA network implementation. The whole proposed SA-FRL scheme has been evaluated by both simulation and experimental data. The simulation and experimental results show that the proposed relay selection scheme can converge more quickly than classical RL and random selection with the increase of the number of iterations. The reward, access delay and data rate of SA-FRL can converge at the highest value and are close to the ideal optimum value. All in all, the proposed SA-FRL relay selection scheme can improve the communication efficiency through the selection of the relay nodes with high link quality and low access delay.