Global Convergence of Localized Policy Iteration in Networked Multi-Agent Reinforcement Learning-Reference-Cited by-同舟云学术

Global Convergence of Localized Policy Iteration in Networked Multi-Agent Reinforcement Learning

Published:2023-06-26 Issue:1 Volume:51 Page:83-84
ISSN:0163-5999
Container-title:ACM SIGMETRICS Performance Evaluation Review
language:en
Short-container-title:SIGMETRICS Perform. Eval. Rev.

Author:

Zhang Yizhou¹^ORCID,Qu Guannan²^ORCID,Xu Pan³^ORCID,Lin Yiheng⁴^ORCID,Chen Zaiwei⁴^ORCID,Wierman Adam⁴^ORCID

Affiliation:

1. Tsinghua University, Beijing, China

2. Carnegie Mellon University, Pittsburgh, PA, USA

3. Duke University, Durham, NC, USA

4. California Institute of Technology, Pasadena, CA, USA

Abstract

We study a multi-agent reinforcement learning (MARL) problem where the agents interact over a given network. The goal of the agents is to cooperatively maximize the average of their entropy-regularized long-term rewards. To overcome the curse of dimensionality and to reduce communication, we propose a Localized Policy Iteration (LPI) algorithm that provably learns a near-globally-optimal policy using only local information. In particular, we show that, despite restricting each agent's attention to only its κ-hop neighborhood, the agents are able to learn a policy with an optimality gap that decays polynomially in κ. In addition, we show the finite-sample convergence of LPI to the global optimal policy, which explicitly captures the trade-off between optimality and computational complexity in choosing κ. Numerical simulations demonstrate the effectiveness of LPI. This extended abstract is an abridged version of [12].

Funder

Amazon AWS

PIMCO Postdoc Fellowship

National Science Foundation

C3 AI Institute

Simoudis Discovery Prize

PIMCO Graduate Fellowship

Publisher

Association for Computing Machinery (ACM)

Subject

Computer Networks and Communications,Hardware and Architecture,Software

Link

https://dl.acm.org/doi/pdf/10.1145/3606376.3593545

Reference12 articles.

1. Xin Chen , Guannan Qu , Yujie Tang , Steven Low , and Na Li. 2022. Reinforcement learning for selective key applications in power systems: Recent advances and future challenges . IEEE Transactions on Smart Grid ( 2022 ). Xin Chen, Guannan Qu, Yujie Tang, Steven Low, and Na Li. 2022. Reinforcement learning for selective key applications in power systems: Recent advances and future challenges. IEEE Transactions on Smart Grid (2022).

2. The dynamics of reinforcement learning in cooperative multiagent systems;Claus Caroline;AAAI/IAAI,1998

3. Multi-agent reinforcement learning in stochastic networked systems;Lin Yiheng;Advances in Neural Information Processing Systems,2021

4. Volodymyr Mnih , Koray Kavukcuoglu , David Silver , Andrei A Rusu , Joel Veness , Marc G Bellemare , Alex Graves , Martin Riedmiller , Andreas K Fidjeland , Georg Ostrovski , 2015 . Human-level control through deep reinforcement learning . Nature , Vol. 518 , 7540 (2015), 529. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. 2015. Human-level control through deep reinforcement learning. Nature, Vol. 518, 7540 (2015), 529.