Scaling Up Sign Spotting Through Sign Language Dictionaries

Abstract

AbstractThe focus of this work issign spotting–given a video of an isolated sign, our task is to identifywhetherandwhereit has been signed in a continuous, co-articulated sign language video. To achieve this sign spotting task, we train a model using multiple types of available supervision by: (1)watchingexisting footage which is sparsely labelled using mouthing cues; (2)readingassociated subtitles (readily available translations of the signed content) which provide additionalweak-supervision; (3)looking upwords (for which no co-articulated labelled examples are available) in visual sign language dictionaries to enable novel sign spotting. These three tasks are integrated into a unified learning framework using the principles of Noise Contrastive Estimation and Multiple Instance Learning. We validate the effectiveness of our approach on low-shot sign spotting benchmarks. In addition, we contribute a machine-readable British Sign Language (BSL) dictionary dataset of isolated signs,BslDict, to facilitate study of this task. The dataset, models and code are available at our project page.