TDT-MIL: a framework with a dual-channel spatial positional encoder for weakly-supervised whole slide image classification

Abstract

The classic multiple instance learning (MIL) paradigm is harnessed for weakly-supervised whole slide image (WSI) classification. The spatial position relationship located between positive tissues is crucial for this task due to the small percentage of these tissues in billions of pixels, which has been overlooked by most studies. Therefore, we propose a framework called TDT-MIL. We first serially connect a convolutional neural network and transformer for basic feature extraction. Then, a novel dual-channel spatial positional encoder (DCSPE) module is designed to simultaneously capture the complementary local and global positional information between instances. To further supplement the spatial position relationship, we construct a convolutional triple-attention (CTA) module to attend to the inter-channel information. Thus, the spatial positional and inter-channel information is fully mined by our model to characterize the key pathological semantics in WSI. We evaluated TDT-MIL on two publicly available datasets, including CAMELYON16 and TCGA-NSCLC, with the corresponding classification accuracy and AUC up to 91.54%, 94.96%, and 90.21%, 94.36%, respectively, outperforming state-of-the-art baselines. More importantly, our model possesses a satisfactory capability in solving the imbalanced WSI classification task using an ingenious but interpretable structure.