Spatial-Spectral-Associative Contrastive Learning for Satellite Hyperspectral Image Classification with Transformers

Abstract

Albeit hyperspectral image (HSI) classification methods based on deep learning have presented high accuracy in supervised classification, these traditional methods required quite a few labeled samples for parameter optimization. When processing HSIs, however, artificially labeled samples are always insufficient, and class imbalance in limited samples is inevitable. This study proposed a Transformer-based framework of spatial–spectral–associative contrastive learning classification methods to extract both spatial and spectral features of HSIs by the self-supervised method. Firstly, the label information required for contrastive learning is generated by a spatial–spectral augmentation transform and image entropy. Then, the spatial and spectral Transformer modules are used to learn the high-level semantic features of the spatial domain and the spectral domain, respectively, from which the cross-domain features are fused by associative optimization. Finally, we design a classifier based on the Transformer. The invariant features distinguished from spatial–spectral properties are used in the classification of satellite HSIs to further extract the discriminant features between different pixels, and the class intersection over union is imported into the loss function to avoid the classification collapse caused by class imbalance. Conducting experiments on two satellite HSI datasets, this study verified the classification performance of the model. The results showed that the self-supervised contrastive learning model can extract effective features for classification, and the classification generated from this model is more accurate compared with that of the supervised deep learning model, especially in the average accuracy of the various classifications.