Dike health inspection is crucial in river channel regulating. The conventional manual collapse inspection is inefficient and costly so that the unmanned aerial vehicle (UAV)-based inspection has been widely applied. However, the existing vision-based defect detection methods face challenges, such as lack of defect sample data and closed specified data sets. To address them, a defect detection method based on improved YOLOv5 recognition combined with image morphology processing is proposed for dike health inspection with zero defect samples. Specifically, the coordinate attention mechanism is introduced in YOLOv5 model to improve recognition capability for dikes. Also, a rotating bounding box target detection is designed for arbitrary orientation of dikes under UAV view, due to ineffective horizontal bounding box detection. Furthermore, for suspected defect locating efficiency promotion, the specific recognized area of the dike is isolated in the image morphology process. The results show that the proposed method outperforms the traditional Yolov5 algorithm on recall rate, F1, and mAP.