Multi-scale hyperspectral recovery networks: RGB-hyperspectral imaging consistency empowered deep spectral super-resolution

Abstract

Hyperspectral imaging, which analyzes a spectrum of light instead of primary colors, can provide rich physical and chemical information not regularly available to traditional imaging modalities. To physically obtain hyperspectral images, various methods have been designed relying on dispersion elements, filters, and coded masks, inevitably making the system complicated and expensive. To address these challenges, emerging deep neural networks can achieve spectral super-resolution by tackling the ill-posed problem, thereby enhancing the spectral recovery capabilities of RGB cameras. However, most of them ignore the consistency in spatial structure between the RGB and hyperspectral images, which can provide additional physical prior knowledge for spectral super-resolution tasks. To increase the fidelity of hyperspectral recovery from RGB images, we propose multi-scale hyperspectral recovery networks (MHRNs), designed to take full consideration of the consistency. In addition, physical constraints are employed to train MHRNs for better performance and interpretability. The spectral super-resolution applicability of the MHRN on publicly available datasets as well as practical applications suggests that it has great potential for advancing spectral recovery across various practical applications such as remote sensing, medical imaging, agricultural testing, and industrial inspection.