Generative Adversarial Implicit Successor Representation

Abstract

ABSTRACTWe propose a novel method to implicitly encode Successor Representations (SRs) using a Generative Adversarial Network (GAN). SRs are a method to encode states of the environment in terms of their predictive relationships with other states, which can be used to predict long-term future rewards. In standard explicit methods, the value of SR is found from an explicit map between future states after an action or to find an approximate function. Instead, our method encodes SR implicitly using a GAN. The distribution of samples generated by the GAN system approximates the successor representation. We also suggest an action decision procedure for the implicit encoding of SR. The system makes the decision using an analysis-by-synthesis procedure that it attempts to synthesize a sample that can explain the action decision constraints of the current and target states. Our system is different from the classical SR in several points. It can sample actual samples reflecting SR distribution, which is not easy for explicit models. It can also get around the issue of explicitly representing probabilities or successor representation values and doing math over them. We tested our system in a toy environment, where the agent could learn the implicit successor representation successfully and use it for action decisions.