Object-Assisted Question Featurization and Multi-CNN Image Feature Fusion for Visual Question Answering

Abstract

Visual question answering (VQA) demands a meticulous and concurrent proficiency in image interpretation and natural language understanding to correctly answer the question about an image. The existing VQA solutions either focus only on improving the joint multi-modal embedding or on the fine-tuning of visual understanding through attention. This research, in contrast to the current trend, investigates the feasibility of an object-assisted language understanding strategy titled semantic object ranking (SOR) framework for VQA. The proposed system refines the natural language question representation with the help of detected visual objects. For multi-CNN image representation, the system employs canonical correlation analysis (CCA). The suggested model is assessed using accuracy and WUPS measures on the DAQUAR dataset. On the DAQUAR dataset, the analytical outcomes reveal that the presented system outperforms the prior state-of-the-art by a significant factor. In addition to the quantitative analysis, proper illustrations are supplied to observe the reasons for performance improvement.