Greek sign language recognition for an education platform

Abstract

AbstractRecent advances in sign language (SL) technologies, along with demand for SL education, have led to increased interest in developing tools that enable automatic assessment of learners’ SL video productions, helping both students and their instructors. At the very least, such tools should perform automatic SL recognition (SLR) of non-studio quality videos in a signer-independent (SI) fashion, thus providing simple binary feedback on learners’ signing under realistic usage scenarios. Motivated by the above and the lack of any such tools for the Greek SL (GSL), we have been developing the “SL-ReDu” education platform for both receptive and productive GSL learning and student assessment. In this paper, we present our SLR module for GSL, developed for and integrated to the “SL-ReDu” system. The module incorporates state-of-the-art deep-learning based visual detection, feature extraction, and classification, operates in an SI mode on web-cam videos, and accommodates a small-size vocabulary of isolated signs and continuously fingerspelled letter sequences. We train the module on collected GSL data and demonstrate its superiority over a number of alternative SLR algorithms. We then conduct its objective evaluation within the “SL-ReDu” system and carry out a subjective evaluation of the overall platform, obtaining very satisfactory results in both.