Prediction of broken areas in murals based on MLP-fused long-range semantics

Abstract

AbstractPredicting the broken areas in murals plays a key role in mural virtual restoration. Mural damage may arise for various reasons and the broken areas also vary greatly in terms of type. The existing methods, however, are limited to predicting a single type of damage and often struggle to identify the dispersed damage with accuracy. Moreover, these methods make it difficult to capture the global information in the broken areas for their insufficient understanding of contexts. To fully use the features at different scales, we propose a novel hierarchical multi-scale encoder-decoder framework termed as Mixer of Dual Attention and Convolution (DACMixer). With the introduction of an attention-convolution dual-branch module in the encoder, DACMixer can not only improve its ability to extract intricate features of small broken areas but also capture long-range dependencies of independent broken areas. Within DACMixer, the MFF (Multi-layer perceptron-based feature fusion) module integrates both local and global information in the broken areas, facilitating efficient and explicit modeling image hierarchies in the global and local range. Contrary to the encoder, DACMixer uses only lightweight multi-level decoder to decode the features of the broken masks, thus reducing the computational cost. Additionally, DACMixer preserves skip-connection to effectively integrate features from different levels of the MFF module. Furthermore, we provide a diversified mural dataset with elaborated broken annotation, which is named YMDA [YMDA denotes our dataset Yunnan_Murals_Dataset_Aug.], to further improve DACMixer’s generalization ability to predict the broken areas. The experimental results demonstrate that DACMixer is capable of predicting the texture, edges, and details of the broken areas in murals with complex backgrounds. DACMixer outperforms the conventional methods with superb results: it achieves 78.3% broken areas IoU (Intersection over Union), 87.5% MIoU (Mean Intersection over Union), and 85.7% Dice coefficient.