A Machine Translation System from Indian Sign Language to English Text

Abstract

Sign language recognition and translation is a crucial step towards improving communication between the deaf and the rest of the society. According to the Indian Sign Language Research and Training Centre (ISLRTC), India has around 300 certified human interpreters. With such a shortage of human interpreters, an alternative service is desired that helps people to achieve smooth communicate with deaf. In this study, an approach is presented that translates ISL sentences in English text using MobileNetV2 model and neural machine translation (NMT). The system features ISL corpus created from Brown corpus using ISL grammar rules. The approach converts the ISL videos into ISL gloss sequence using MobileNetV2 model and recognised ISL gloss sequence is then fed to machine translation module. MobileNetV2 was proven best-suited model for recognition of ISL sentences and NMT gives better result than statistical machine translation (SMT) to convert ISL gloss sequence into English text. The automatic and human evaluation of the proposed approach gives 83.3% and 86.1% accuracy, respectively.