Aiming at the problem of traditional portrait background similarity retrieval methods being low accuracy and time-consuming, a similarity retrieval method based on image background analysis is presented. The proposed method uses a combination of portrait segmentation and retrieval models. Firstly, the portrait segmentation model is used to remove the portraits in the images to eliminate the interference of portraits on background features; secondly, the image retrieval model is used to retrieve images with similar background features; LSH is added to improve the retrieval efficiency; finally, the retrieval results are used to further determine whether the background is similar. The experiment is implemented based on real data from a company. The results showed that the average precision, average map, and recall of this method reached 85%, 90%, and 50%, respectively. The average accuracy and recall are 10% better than the overall image retrieval model.