Hand Gesture Recognition System for Controlling VLC Media Player Based on Two Stream Transfer Learning

Abstract

Abstract The hand gesture recognition (HGR) is one of the most prominent cogent in machine learning and computer vision applications. This paper implements an application of HGR which is designed for human computer interaction using deep learning based two stream transfer learning for controlling the VLC media player. The aim and objectives of this application are to use a natural device free interface, which recognizes hand gestures as commands. The application uses a webcam which is used for image acquisition. To control the VLC media player using defined gesture, the application focuses on some function of VLC which are used more frequently such as play, pause, and stop. In order to incorporate this, a decision fusion based system using the transfer learning architectures is proposed to achieve the said task. Two pretrained models namely ‘MobileNet’ and ‘Inception V3’ are used in the research work. To find the region of interest (ROI) in the image, YOLO (You Only Look Once) architecture is used which also decides the type of model. Edge map images and the spatial images are trained using two separate versions of the MobileNet based transfer learning architecture and then the final joint probabilities are combined to decide upon the hand sign of the image. The experimentation carried out demonstrates that the system provides high accuracy in real time with 100% accuracy.