Design of knowledge incorporated VQA based on spatial GCNN with structured sentence embedding and linking algorithm

Abstract

Visual question Answering (VQA) is a computer vision task that requires a system to infer an answer to a text-based question about an image. Prior approaches did not take into account an image’s positional information or the questions’ grammatical and semantic relationships during image and question processing. Featurization, which leads to the false answering of the question. Hence to overcome this issue CNN –Graph based LSTM with optimized BP Featurization technique is introduced for feature extraction of image as well as question. The position of the subjects in the image has been determined using CNN with a dropout layer and the optimized momentum backpropagation during the extraction of image features without losing any image data. Then, using a graph-based LSTM with loopy backpropagation, the questions’ syntactic and semantic dependencies are retrieved. However, due to their lack of external knowledge about the input image, the existing approaches are unable to respond to common sense knowledge-based questions (open domain). As a result, the proposed Spatial GCNN knowledge retrieval with PDB Model and Spatial Graph Convolutional Neural Network, which recovers external data from Wikidata, have been used to address the open domain problems. Then the Probabilistic Discriminative Bayesian model, based Attention mechanism predicts the answer by referring to all concepts in question. Thus, the proposed method answers the open domain question with high accuracy of 88.30%.