Reinforcement Learning Methods for Computation Offloading: A Systematic Review

Abstract

Today, cloud computation offloading may not be an appropriate solution for delay-sensitive applications due to the long distance between end-devices and remote datacenters. In addition, offloading to a remote cloud can consume bandwidth and dramatically increase costs. However, end-devices such as sensors, cameras, and smartphones have limited computing and storage capacity. Processing tasks on such battery-powered and energy-constrained devices becomes even more complex. To address these challenges, a new paradigm called Edge Computing (EC) emerged nearly a decade ago to bring computing resources closer to end-devices. Here, edge servers located between the end-device and the remote cloud perform user tasks. Recently, several new computing paradigms such as Mobile Edge Computing (MEC) and Fog Computing (FC) have emerged to complement Cloud Computing (CC) and EC. Although these paradigms are heterogeneous, they can further reduce energy consumption and task response time, especially for delay-sensitive applications. Computation offloading is a multi-objective, NP-hard optimization problem. A significant part of previous research in this field is devoted to Machine Learning (ML) methods. One of the essential types of ML is Reinforcement Learning (RL), in which an agent learns how to make the best decision using the experiences gained from the environment. This article provides a systematic review of the widely used RL approaches in computation offloading. It covers research in complementary paradigms such as mobile cloud computing, edge computing, fog computing, and the Internet of Things. We explain the reasons for using various RL methods in computation offloading from a technical point of view. This analysis includes both binary offloading and partial offloading techniques. For each method, the essential elements of RL and the characteristics of the environment are discussed regarding the most important criteria. Research challenges and Future trends are also mentioned.