Automatic Segmentation of Cervical Cells Based on Star-Convex Polygons in Pap Smear Images

Abstract

Cervical cancer is one of the most common cancers that threaten women’s lives, and its early screening is of great significance for the prevention and treatment of cervical diseases. Pathologically, the accurate segmentation of cervical cells plays a crucial role in the diagnosis of cervical cancer. However, the frequent presence of adherent or overlapping cervical cells in Pap smear images makes separating them individually a difficult task. Currently, there are few studies on the segmentation of adherent cervical cells, and the existing methods commonly suffer from low segmentation accuracy and complex design processes. To address the above problems, we propose a novel star-convex polygon-based convolutional neural network with an encoder-decoder structure, called SPCNet. The model accomplishes the segmentation of adherent cells relying on three steps: automatic feature extraction, star-convex polygon detection, and non-maximal suppression (NMS). Concretely, a new residual-based attentional embedding (RAE) block is suggested for image feature extraction. It fuses the deep features from the attention-based convolutional layers with the shallow features from the original image through the residual connection, enhancing the network’s ability to extract the abundant image features. And then, a polygon-based adaptive NMS (PA-NMS) algorithm is adopted to screen the generated polygon proposals and further achieve the accurate detection of adherent cells, thus allowing the network to completely segment the cell instances in Pap smear images. Finally, the effectiveness of our method is evaluated on three independent datasets. Extensive experimental results demonstrate that the method obtains superior segmentation performance compared to other well-established algorithms.