A novel tree-based method for interpretable reinforcement learning

Abstract

Deep reinforcement learning (DRL) has garnered remarkable success across various domains, propelled by advancements in deep learning (DL) technologies. However, the opacity of DL presents significant challenges, limiting the application of DRL in critical systems. In response, decision tree (DT)-based methods, known for their transparent decision-making mechanisms, have shown promise in making interpretable policies for decision-making problems. Existing methods often employ differential DTs to model RL policies and discretize them to conventional DTs for higher interpretability. Yet, this method leads to discrepancies between the trained differential DTs and the discretized DTs. To address this issue, we introduce Generative Consistent Trees (GCTs), a novel solution that circumvents the information loss typically associated with the argmax operation in prior research. By implementing a reparameterization technique to approximate the categorical distribution, GCTs ensure the consistencies between trained GCTs and discretized counterparts. Moreover, we have developed an imitation-learning-based framework for interpretable reinforcement learning. This framework is designed to train GCTs by efficiently mimicking expert policies. Our extensive experiments across multiple environments have validated the effectiveness of this approach, highlighting the potential of GCTs in enhancing the interpretability and applicability of DRL.