A Novel Machine Learning Based Two-Way Communication System for Deaf and Mute
-
Published:2022-12-29
Issue:1
Volume:13
Page:453
-
ISSN:2076-3417
-
Container-title:Applied Sciences
-
language:en
-
Short-container-title:Applied Sciences
Author:
Saleem Muhammad Imran, Siddiqui AtifORCID, Noor Shaheena, Luque-Nieto Miguel-AngelORCID, Otero PabloORCID
Abstract
Deaf and mute people are an integral part of society, and it is particularly important to provide them with a platform to be able to communicate without the need for any training or learning. These people rely on sign language, but for effective communication, it is expected that others can understand sign language. Learning sign language is a challenge for those with no impairment. Another challenge is to have a system in which hand gestures of different languages are supported. In this manuscript, a system is presented that provides communication between deaf and mute (DnM) and non-deaf and mute (NDnM). The hand gestures of DnM people are acquired and processed using deep learning, and multiple language support is achieved using supervised machine learning. The NDnM people are provided with an audio interface where the hand gestures are converted into speech and generated through the sound card interface of the computer. Speech from NDnM people is acquired using microphone input and converted into text. The system is easy to use and low cost. The system is modular and can be enhanced by adding data to support more languages in the future. A supervised machine learning dataset is defined and created that provides automated multi-language communication between the DnM and NDnM people. It is expected that this system will support DnM people in communicating effectively with others and restoring a feeling of normalcy in their daily lives. The hand gesture detection accuracy of the system is more than 90% for most, while for certain scenarios, this is between 80% and 90% due to variations in hand gestures between DnM people. The system is validated and evaluated using a series of experiments.
Subject
Fluid Flow and Transfer Processes,Computer Science Applications,Process Chemistry and Technology,General Engineering,Instrumentation,General Materials Science
Reference48 articles.
1. Vaidya, O., Gandhe, S., Sharma, A., Bhate, A., Bhosale, V., and Mahale, R. (2020, January 4–6). Design and development of hand gesture based communication device for deaf and mute people. Proceedings of the IEEE Bombay Section Signature Conference (IBSSC), Mumbai, India. 2. Marin, G., Dominio, F., and Zanuttigh, P. (2014, January 27–30). Hand gesture recognition with Leap Motion and Kinect devices. Proceedings of the IEEE International Conference on Image Processing (ICIP), Paris, France. 3. Saleem, M.I., Otero, P., Noor, S., and Aftab, R. (2020, January 6–8). Full duplex smart system for Deaf & Dumb and normal people. Proceedings of the Global Conference on Wireless and Optical Technologies (GCWOT), Mlaga, Spain. 4. Augmented Sign Language Modeling (ASLM) with interaction design on smartphone—An assistive learning and communication tool for inclusive classroom;Deb;Procedia Comput. Sci.,2018 5. Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. (2017). MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv.
Cited by
2 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献
|
|