Variable Structure and Modeling Units for Chinese Lipreading

Abstract

Lipreading is a type of Human–Computer Interaction (HCI) based on visual information. From a linguistic point of view, Chinese is a monosyllabic language with a much higher proportion of homophones than English. Identifying homophones in Chinese Mandarin lipreading is very challenging. Since the lip shape in the context can distinguish homophones, and smaller recognition units can reduce the types of recognition and alleviate data sparsity, we propose to improve the accuracy of lipreading by simultaneously exploiting the correlation of lip features at different distances and smaller modeling units. We implement a long short-term multi-feature space to represent lip features, and CTC–Attention to learn temporal correlations. We also introduce Weight Finite State Transducer (WFST) to enhance the semantic analysis capability of the model. Our model aims to distinguish homophones and improve the accuracy of lipreading. To reduce data sparsity, we use Tonal Initials and Finals (TIF) as the modeling units. We record a sentence-level Chinese lipreading dataset, ICSLR, and label Mandarin characters, syllables, and TIF. We demonstrate the effectiveness of the proposed approach compared to its counterparts through extensive experiments on Grid, ICSLR, and CMLR datasets.