Structural knowledge transfer by spatial abstraction for reinforcement learning agents

Abstract

In this article we investigate the role of abstraction principles for knowledge transfer in agent control learning tasks. We analyze abstraction from a formal point of view and characterize three distinct facets: aspectualization, coarsening, and conceptual classification. The taxonomy we develop allows us to interrelate existing approaches to abstraction, leading to a code of practice for designing knowledge representations that support knowledge transfer. We detail how aspectualization can be utilized to achieve knowledge transfer in reinforcement learning. We propose the use of so-called structure space aspectualizable knowledge representations that explicate structural properties of the state space and present a posteriori structure space aspectualization (APSST) as a method to extract generally sensible behavior from a learned policy. This new policy can be used for knowledge transfer to support learning new tasks in different environments. Finally, we present a case study that demonstrates transfer of generally sensible navigation skills from simple simulation to a real-world robotic platform.