Applying LSTM and GRU Methods to Recognize and Interpret Hand Gestures, Poses, and Face-Based Sign Language in Real Time-Reference-Cited by-同舟云学术

Applying LSTM and GRU Methods to Recognize and Interpret Hand Gestures, Poses, and Face-Based Sign Language in Real Time

Published:2024-03-20 Issue:2 Volume:28 Page:265-272
ISSN:1883-8014
Container-title:Journal of Advanced Computational Intelligence and Intelligent Informatics
language:en
Short-container-title:JACIII

Author:

Ilham Amil Ahmad¹^ORCID,Nurtanio Ingrid¹^ORCID,Ridwang ²^ORCID,Syafaruddin ³^ORCID

Affiliation:

1. Department of Informatics, Universitas Hasanuddin, Jl. Poros Malino Km. 6, Bontomarannu, Gowa, Sulawesi Selatan 92171, Indonesia

2. Department of Electrical Engineering, Universitas Muhammadiyah Makassar, Jl. Sultan Alauddin No.259, Makassar, Sulawesi Selatan 90221, Indonesia

3. Department of Electrical Engineering, Universitas Hasanuddin, Jl. Poros Malino Km. 6, Bontomarannu, Gowa, Sulawesi Selatan 92171, Indonesia

Abstract

This research uses a real-time, human-computer interaction application to examine sign language recognition. This work develops a rule-based hand gesture approach for Indonesian sign language in order to interpret some words using a combination of hand movements, mimics, and poses. The main objective in this study is the recognition of sign language that is based on hand movements made in front of the body with one or two hands, movements which may involve switching between the left and right hand or may be combined with mimics and poses. To overcome this problem, a research framework is developed by coordinating hand gestures with poses and mimics to create features by using holistic MediaPipe. To train and test data in real time, the long short time memory (LSTM) and gated recurrent unit (GRU) approaches are used. The research findings presented in this paper show that hand gestures in real-time interactions are reliably recognized, and some words are interpreted with the high accuracy rates of 94% and 96% for the LSTM and GRU methods, respectively.

Publisher

Fuji Technology Press Ltd.

Reference21 articles.

1. E. Ayodele et al., “A review of deep learning approaches in glove-based gesture classification,” P. Kumar, Y. Kumar, and M. A. Tawhid (Eds.), “Machine Learning, Big Data, and IoT for Medical Informatics,” pp. 143-164, Academic Press, 2021. https://doi.org/10.1016/B978-0-12-821777-1.00012-4

2. J. P. Sahoo, S. Ari, and S. K. Patra, “A user independent hand gesture recognition system using deep CNN feature fusion and machine learning technique,” S. Chakraverty (Ed.), “New Paradigms in Computational Modeling and its Applications,” pp. 189-207, Academic Press, 2021. https://doi.org/10.1016/B978-0-12-822133-4.00011-6

3. I. A. Adeyanju, O. O. Bello, and M. A. Adegboye, “Machine learning methods for sign language recognition: A critical review and analysis,” Intelligent Systems with Applications, Vol.12, Article No.200056, 2021. https://doi.org/10.1016/j.iswa.2021.200056

4. P. K. Athira, C. J. Sruthi, and A. Lijiya, “A signer independent sign language recognition with co-articulation elimination from live videos: An Indian scenario,” J. of King Saud University – Computer and Information Sciences, Vol.34, No.3, pp. 771-781, 2022. https://doi.org/10.1016/j.jksuci.2019.05.002

5. S. Sharma and S. Singh, “Vision-based hand gesture recognition using deep learning for the interpretation of sign language,” Expert Systems with Applications, Vol.182, Article No.115657, 2021. https://doi.org/10.1016/j.eswa.2021.115657