Semantic segmentation algorithm for video from UAV based on adaptive keyframe scheduling via similarity measurement

Abstract

Abstract Unmanned aerial vehicle (UAV) videos exhibit complex object motion features and significant differences between frame features. To solve the problems of feature information loss and drastic accuracy decline in applying the video semantic segmentation method based on the fixed-period update strategy of keyframes, a keyframe recognition method based on similarity measurement is proposed, forming a video semantic segmentation algorithm based on adaptive keyframe scheduling. One keyframe recognition method based on pixel similarity measurement is established by modeling the similarity relationship between low-level pixels in adjacent frames. Meanwhile, the other keyframe recognition method based on feature similarity measurement is established by constructing a shallow Siamese network to measure the similarity relationship between features of frames. Then a discriminative network is constructed based on the obtained similarity of inter frames, and the segmentation process is accelerated by reusing features of keyframes through the optical flow network. Thereby a video semantic segmentation method for UAV based on adaptive keyframe strategy is established. The effectiveness of the proposed video semantic segmentation algorithm is verified on UAVid dataset. The results demonstrated that the speed of the proposed algorithm reaches 53.2 frames per second (FPS) and 54.5 FPS on the premise that the mean intersection over union is higher than 40% (this value is compared with the segmentation accuracy in the balanced mode when the similarity threshold is 0.76 and 0.88, respectively), which is 18.5 FPS and 19.8 FPS higher than the PSPNet image semantic segmentation algorithm. In addition, analysis of experiment results shows that the pixel-similarity-based keyframe recognition is suitable for high-precision video semantic segmentation scenes that need to improve segmentation efficiency, and the feature-similarity-based keyframe recognition is more suitable for high real-time video semantic segmentation tasks that require a small decrease in overall algorithm accuracy. In a word, the proposed video semantic segmentation algorithm based on adaptive keyframe scheduling via similarity measurement can improve segmentation speed while ensuring segmentation accuracy and stability.