Zero-shot Video-based Visual Question Answering for Visually Impaired People

Abstract

83% of the world's population owned a smartphone today. The use of smartphones as personal assistants is also emerging. This article proposes a new video dataset suitable for few-shot or zero-shot learning. The dataset contains handheld product videos captured using a handheld smartphone by visually impaired (VI) people. With the ultimate goal of improving assistive technology for the VI, the dataset is designed to facilitate question-answering based on both textual and visual features. One of the objectives of such video analytics is to develop assistive technology for visually impaired people for day-to-day activity management and also provide an independent shopping experience. This article highlights the limitations of existing deep learning-based approaches when applied to the dataset, suggesting that they pose novel challenges for computer vision researchers. We propose a zero-shot VQA for the problem. Despite the current approaches' poor performance, they foster a training-free zero-shot approach, providing a baseline for visual question-answering towards the foundation for future research. We believe the dataset provides new challenges and attracts many computer vision researchers. This dataset will be available.