A Novel Paradigm for Deep Reinforcement Learning of Biomimetic Systems

Abstract

Mechanisms behind neural control of movement have been an active area of research. Goal-directed movement is a common experimental setup used to understand these mechanisms and relevant neural pathways. On the one hand, optimal feedback control theory is used to model and make quantitative predictions of the coordinated activations of the effectors, such as muscles, joints or limbs. While on the other hand, evidence shows that higher centres such as Basal Ganglia and Cerebellum are involved in activities such as reinforcement learning and error correction. In this paper, we provide a framework to build a digital twin of relevant sections of the human spinal cord using our NEUROiD platform. The digital twin is anatomically and physiologically realistic model of the spinal cord at cellular, spinal networks and system level. We then build a framework to learn the supraspinal activations necessary to perform a simple goal directed movement of the upper limb. The NEUROiD model is interfaced to an Opensim model for all the musculoskeletal simulations. We use Deep Reinforcement Learning to obtain the supraspinal activations necessary to perform the goal directed movement. As per our knowledge, this is the first time an attempt is made to learn the stimulation pattern at the spinal cord level, especially by limiting the observation space to only the afferent feedback received on the Ia, II and Ib fibers. Such a setup results in a biologically realistic constrained environment for learning. Our results show that (1) Reinforcement Learning algorithm converges naturally to the triphasic response observed during goal directed movement (2) Increasing the complexity of the goal gradually helped to accelerate learning (3) Modulation of the afferent inputs were sufficient to execute tasks which were not explicitly learned, but were closely related to the learnt task.