Addressing deadlock in large‐scale, complex rail networks via multi‐agent deep reinforcement learning

Abstract

AbstractRail freight planning problems pose specific challenges that have attracted the attention of academics and industry professionals for many decades. They involve multiple types of assets (trains, stations, terminals, etc.) and are subjected to structural, operational and safety constraints. Even though various approaches have been proposed, few can address the complexity and size of real‐world scenarios, and decentralized techniques, like multi‐agent systems (MAS), have become more prevalent. The current state of the art in disciplines such as agent technology, reinforcement learning and discrete‐event simulation allows the implementation of complex architectures, with multiple actors interacting and learning simultaneously. Therefore, this study takes advantage of these current advances and proposes an innovative approach to real‐time traffic management problems in freight railway networks through multi‐agent deep reinforcement learning (MADRL). This study was motivated by the decision‐making scheduling problems arising in the Hunter Valley Coal Chain (HVCC), located in New South Wales, Australia. The MADRL algorithm uses as the training environment the simulation model currently utilized for capacity planning of the HVCC, allowing experiments with actual data. Thus, we enhanced the simulation model to accommodate a MAS with intelligent agents representing system elements, such as trains, dump stations, and load points. Furthermore, these agents act in a decentralized fashion based on local observations, constituting a partially‐observed Markov decision process (dec‐POMDP). Three variations of the MADRL approach are presented: a baseline model, an extended model, and one that directly addresses deadlocks. Finally, we present a transfer learning method that improves deadlock resolution and leverages performance. In the experiments, we explore specific, complex scenarios arising in the HVCC, where trains frequently face deadlock conditions. The baseline model outperforms a first‐come‐first‐serve (FCFS) based heuristic used by HVCC's simulation model and a genetic algorithm in instances with up to 60 trains – but fails in more complex scenarios. On the other hand, the most advanced model, which addresses deadlocks via transfer learning, always finds feasible solutions and produces policies that outperform the FCFS‐based heuristic in 94% of the instances.