A Multi-label Sewer Defects Classification Methodology Using Attention-based Dual Branch network

Abstract

Abstract The sewage system, essential for human welfare and ecological health, requires regular inspections to prevent defects such as cracks, deformation, joint displacement, etc. Traditionally, inspections have relied on Closed-Circuit TeleVision (CCTV), involving two stages: on-site video collection and time-consuming off-site video analysis. In this study, we propose a novel attention-based dual branch model for efficiently classifying multiple sewer pipe defects. It employs spatial and semantic relation graphs that have complementary relationship. Specifically, we first generate word embeddings from embedding layer using our defects corpus, then, we use the word embeddings as input data for the dual branch. For the first branch, we capture relationship between word embeddings and image feature maps, while for the second branch, we exploit co-occurrence dependencies of defect classifiers from correlation matrix of defects. The model was validated on Sewer-ML dataset which consists of 1.3 million multi-label sewer images and 17 specific defects classes. Compared with the state-of-the-art methods in related field, our model achieved a defect weighted F2 score of 83.71% and a normal pipe F1 score of 86.59%, showing its superior capability on the latest benchmark methods. The code was available at http://www.github.com/iamstarlee/Attention-based-Dual-Branch-Network.