Affiliation:
1. School of Information Engineering, Capital Normal University, Beijing 100048, China
2. School of Aerospace Engineering, Beijing Institute of Technology, Beijing 100081, China
Abstract
Sign language recognition has been utilized in human–machine interactions, improving the lives of people with speech impairments or who rely on nonverbal instructions. Thanks to its higher temporal resolution, less visual redundancy information and lower energy consumption, the use of an event camera with a new dynamic vision sensor (DVS) shows promise with regard to sign language recognition with robot perception and intelligent control. Although previous work has focused on event camera-based, simple gesture datasets, such as DVS128Gesture, event camera gesture datasets inspired by sign language are critical, which poses a great impediment to the development of event camera-based sign language recognition. An effective method to extract spatio-temporal features from event data is significantly desired. Firstly, the event-based sign language gesture datasets are proposed and the data have two sources: traditional sign language videos to event stream (DVS_Sign_v2e) and DAVIS346 (DVS_Sign). In the present dataset, data are divided into five classification, verbs, quantifiers, position, things and people, adapting to actual scenarios where robots provide instruction or assistance. Sign language classification is demonstrated in spike neuron networks with a spatio-temporal back-propagation training method, leading to the best recognition accuracy of 77%. This work paves the way for the combination of event camera-based sign language gesture recognition and robotic perception for the future intelligent systems.
Funder
R&D Program of Beijing Municipal Education Commission
the National Natural Science Foundation of China
Subject
Electrical and Electronic Engineering,Computer Networks and Communications,Hardware and Architecture,Signal Processing,Control and Systems Engineering
Reference45 articles.
1. Design and development of a humanoid robot for sign language interpretation;Nihal;SN Comput. Sci.,2021
2. Fellinger, J. (2015). Public Health of Deaf People, Gallaudet University Press.
3. A survey of robot perception and control based on event camera;Su;Acta Autom. Sin.,2022
4. Amir, A., Taba, B., Berg, D., Melano, T., McKinstry, J., Di Nolfo, C., and Modha, D. (2017, January 21–26). A low power, fully event-based gesture recognition system. Proceedings of the IEEE Conference on Computer Vision and Recognition, Honolulu, HI, USA.
5. Liu, Z., Wu, F., and Das, S.K. (2021). Wireless Algorithms, Systems, and Applications. WASA 2021. Lecture Notes in Computer Science, Springer.
Cited by
6 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献