Investigations on collaborative remote control of virtual robotic manipulators by using a Kinect v2 sensor

Abstract

The present work deals with design aspects and testing results of a novel solution regarding the remote control of two virtual robotic manipulators by using hands motion tracking and palms gesture recognition. The spatial position and orientation of the user's left and right hands' wrists are interpreted by a custom built software application and then forwarded as positioning data and gripper status to a V-Rep server running the simulation scene. The goal of the presented setup is for the user to be able to manipulate objects in the scene in a collaborative way, i.e. pick one object with the first robot and give it to the second one for elsewhere depositing. The real-life applications for this approach are to provide a framework for simplified operator training and programming of robots used in pick-and-place or assembly operations, and also in healthcare for assisting patients in hand -coordination and mobility recovery.