Multiple Defects Inspection of Dam Spillway Surface Using Deep Learning and 3D Reconstruction Techniques

Abstract

After a lengthy period of scouring, the reinforced concrete surface of the dam spillway (i.e., drift spillways and flood discharge spillways) will suffer from deterioration and damage. Regular manual inspection is time-consuming and dangerous. This paper presents a robotic solution to detect automatically, count defect instance numbers, and reconstruct the surface of dam spillways by incorporating the deep learning method with a visual 3D reconstruction method. The lack of a real dam defect dataset and incomplete registration of minor defects on the 3D mesh model in fusion step are two challenges addressed in the paper. We created a multi-class semantic segmentation dataset of 1711 images (with resolutions of 848 × 480 and 1280 × 720 pixels) acquired by a wall-climbing robot, including cracks, erosion, spots, patched areas, and power safety cable. Then, the architecture of the U-net is modified with pixel-adaptive convolution (PAC) and conditional random field (CRF) to segment different scales of defects, trained, validated, and tested using this dataset. The reconstruction and recovery of minor defect instances in the flow surface and sidewall are facilitated using a keyframe back-projection method. By generating an instance adjacency matrix within the class, the intersection over union (IoU) of 3D voxels is calculated to fuse multiple instances. Our segmentation model achieves an average IoU of 60% for five defect class. For the surface model’s semantic recovery and instance statistics, our method achieves accurate statistics of patched area and erosion instances in an environment of 200 m2, and the average absolute error of the number of spots and cracks has reduced from the original 13.5 to 3.5.