Affiliation:
1. Zhejiang University, Hangzhou, China
2. University of Macau, Taipa, China
Abstract
The goal of sign language recognition (SLR) is to help those who are hard of hearing or deaf overcome the communication barrier. Most existing approaches can be typically divided into two lines, i.e., Skeleton-based, and RGB-based methods, but both lines of methods have their limitations. Skeleton-based methods do not consider facial expressions, while RGB-based approaches usually ignore the fine-grained hand structure. To overcome both limitations, we propose a new framework called the Spatial-temporal Part-aware network (StepNet), based on RGB parts. As its name suggests, it is made up of two modules: Part-level Spatial Modeling and Part-level Temporal Modeling. Part-level Spatial Modeling, in particular, automatically captures the appearance-based properties, such as hands and faces, in the feature space without the use of any keypoint-level annotations. On the other hand, Part-level Temporal Modeling implicitly mines the long short-term context to capture the relevant attributes over time. Extensive experiments demonstrate that our StepNet, thanks to spatial-temporal modules, achieves competitive Top-1 Per-instance accuracy on three commonly used SLR benchmarks, i.e., 56.89% on WLASL, 77.2% on NMFs-CSL, and 77.1% on BOBSL. Additionally, the proposed method is compatible with the optical flow input and can produce superior performance if fused. For those who are hard of hearing, we hope that our work can act as a preliminary step.
Funder
National Natural Science Foundation of China
Fundamental Research Funds for the Central Universities
Publisher
Association for Computing Machinery (ACM)
Reference83 articles.
1. Samuel Albanie, Gül Varol, Liliane Momeni, Triantafyllos Afouras, Joon Son Chung, Neil Fox, and Andrew Zisserman. 2020. BSL-1K: Scaling up co-articulated sign language recognition using mouthing cues. In European Conference on Computer Vision (ECCV’20). 35–53.
2. Samuel Albanie Gül Varol Liliane Momeni Hannah Bull Triantafyllos Afouras Himel Chowdhury Neil Fox Bencie Woll Rob Cooper Andrew McParland and Andrew Zisserman. 2021. BBC-Oxford British sign language dataset. arXiv:2111.03635 (2021).
3. Matyáš Boháček and Marek Hrúz. 2022. Sign pose-based transformer for word-level sign language recognition. In IEEE/CVF Winter Conference on Applications of Computer Vision (WACV’22). 182–191.
4. Patrick Buehler, Andrew Zisserman, and Mark Everingham. 2009. Learning sign language by watching TV (using weakly aligned subtitles). In IEEE/CVF Computer Vision and Pattern Recognition Conference (CVPR’09). 2961–2968.
5. Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, and Richard Bowden. 2020. Sign language transformers: Joint end-to-end sign language recognition and translation. In IEEE/CVF Computer Vision and Pattern Recognition Conference (CVPR’20). 10023–10033.
Cited by
1 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献