Exploiting environmental signals to enable policy correlation in large-scale decentralized systems

Abstract

AbstractCan artificial agents benefit from human conventions? Human societies manage to successfully self-organize and resolve the tragedy of the commons in common-pool resources, in spite of the bleak prediction of non-cooperative game theory. On top of that, real-world problems are inherently large-scale and of low observability. One key concept that facilitates human coordination in such settings is the use of conventions. Inspired by human behavior, we investigate the learning dynamics and emergence of temporal conventions, focusing on common-pool resources. Extra emphasis was given in designing arealistic evaluation setting: (a) environment dynamics are modeled on real-world fisheries, (b) we assume decentralized learning, where agents can observe only their own history, and (c) we run large-scale simulations (up to 64 agents). Uncoupled policies and low observability make cooperation hard to achieve; as the number of agents grow, the probability of taking a correct gradient direction decreases exponentially. By introducing anarbitrary common signal(e.g., date, time, or any periodic set of numbers) as a means to couple the learning process, we show that temporal conventions can emerge and agents reachsustainableharvesting strategies. The introduction of the signal consistently improves the social welfare (by$$258\%$$258%on average, up to$$3306\%$$3306%), the range of environmental parameters where sustainability can be achieved (by$$46\%$$46%on average, up to$$300\%$$300%), and the convergence speed in low abundance settings (by$$13\%$$13%on average, up to$$53\%$$53%).