An adaptive and late multifusion framework in contextual representation based on evidential deep learning and Dempster–Shafer theory

Abstract

AbstractThere is a growing interest in multidisciplinary research in multimodal synthesis technology to stimulate diversity of modal interpretation in different application contexts. The real requirement for modality diversity across multiple contextual representation fields is due to the conflicting nature of data in multitarget sensors, which introduces other obstacles including ambiguity, uncertainty, imbalance, and redundancy in multiobject classification. This paper proposes a new adaptive and late multimodal fusion framework using evidence-enhanced deep learning guided by Dempster–Shafer theory and concatenation strategy to interpret multiple modalities and contextual representations that achieves a bigger number of features for interpreting unstructured multimodality types based on late fusion. Furthermore, it is designed based on a multifusion learning solution to solve the modality and context-based fusion that leads to improving decisions. It creates a fully automated selective deep neural network and constructs an adaptive fusion model for all modalities based on the input type. The proposed framework is implemented based on five layers which are a software-defined fusion layer, a preprocessing layer, a dynamic classification layer, an adaptive fusion layer, and an evaluation layer. The framework is formalizing the modality/context-based problem into an adaptive multifusion framework based on a late fusion level. The particle swarm optimization was used in multiple smart context systems to improve the final classification layer with the best optimal parameters that tracing 30 changes in hyperparameters of deep learning training models. This paper applies multiple experimental with multimodalities inputs in multicontext to show the behaviors the proposed multifusion framework. Experimental results on four challenging datasets including military, agricultural, COIVD-19, and food health data provide impressive results compared to other state-of-the-art multiple fusion models. The main strengths of proposed adaptive fusion framework can classify multiobjects with reduced features automatically and solves the fused data ambiguity and inconsistent data. In addition, it can increase the certainty and reduce the redundancy data with improving the unbalancing data. The experimental results of multimodalities experiment in multicontext using the proposed multimodal fusion framework achieve 98.45% of accuracy.