HDRLPIM: A Simulator for H yper D imensional R einforcement L earning based on P rocessing I n M emory

Abstract

Processing In-Memory (PIM) is a data-centric computation paradigm that performs computations inside the memory, hence eliminating the memory wall problem in traditional computational paradigms used in Von-Neumann architectures. The associative processor, a type of PIM architecture, allows performing parallel and energy-efficient operations on vectors. This architecture is found useful in vector-based applications such as Hyper-Dimensional (HDC) Reinforcement Learning (RL). HDC is rising as a new powerful and lightweight alternative to costly traditional RL models such as Deep Q-Learning. The HDC implementation of Q-Learning relies on encoding the states in a high-dimensional representation where calculating Q-values and finding the maximum one can be done entirely in parallel. In this paper, we propose to implement the main operations of a hyper-dimensional reinforcement learning framework on the associative processor. This acceleration achieves up to \(152.3\times\) and \(6.4\times\) energy and time savings compared to an FPGA implementation. Moreover, HDRLPIM shows that an SRAM-based AP implementation promises up to \(968.2\times\) energy-delay product gains compared to the FPGA implementation.