TIM‐Net: A multi‐label classification network for TCM tongue images fusing global‐local features

Abstract

AbstractCombining the extracted tongue features with other medical indicators can effectively judge the diseases of patients. The previous work usually only analyzes a certain feature of the tongue body and is unable to extract multiple features simultaneously. In this study, a multi‐label classification network named TIM‐Net is proposed, which integrates global and local features to achieve multi‐label intelligent diagnosis of Chinese medicine tongue images. First, a feature extraction network based on ResNet is proposed to capture the features of tongue images more sufficiently. Then, a multi‐label classification algorithm fusing global and local features is proposed, and targeted screening operations are carried out on the class‐related feature maps based on global confidence. In addition, a logical masking algorithm is proposed to ensure that the local features can only correct the feature labels they represent, and do not interfere with other feature labels. The classification accuracy is further improved by using local feature confidence and correcting the global classification results. Finally, the experimental results indicate that the classification accuracy of the tongue images is gradually improved through optimizing the feature extraction network and fusing local features, and it exceeds other state‐of‐the‐art multi‐label classification networks.