Novel Video Benchmark Dataset Generation and Real-Time Recognition of Symbolic Hand Gestures in Indian Dance Applying Deep Learning Techniques

Abstract

A computational approach towards promoting, preservation, and dissemination of knowledge in the domain of cultural heritage, is one of the research areas that has a wide scope. There has been a seismic shift in the way many sectors in society have adapted themselves to the pandemic situation, be it healthcare, food, education, diplomacy, and performing arts. Virtual learning and performing have become the need of the hour in the field of performing arts as well. The objective of this work is threefold; first, this creates benchmark datasets to be shared to make a beneficial impact and for a meaningful engagement by capturing, recognising, and classifying the multimedia content for hastamudras (hand poses) in Bharatanatyam, an Indian classical dance form, which plays a significant role in the conservation of intangible cultural heritage, second, as tutoring system to aspiring learners and third, to build video recommendation systems to promote art as a tool for building an international relationship and further elevate the significance of soft-power through performing arts. This article proposes applying the deep-learning techniques of CNNs as a critical technology to recognise the correct mudra. Experimental results on our challenging mudra dataset through the MobileNet architecture show 85%–95% accuracy in real-time, which outperforms the Sebastien-Marcel dataset. The time taken to process an image is 0.172 seconds, and the result is significant considering that the images are dynamic. This work proves the accuracy of the proposed method, which significantly outperforms another CNN-based Inception v3 model.