TanrsColour: Transformer‐based medical image colourization with content and structure preservation

Abstract

AbstractMedical image colouring techniques enable to colourize grey‐scale medical images for assisting doctors in diagnosis. Benefiting from the non‐linear fitting ability of deep neural network, deep medical image colouring techniques have achieved remarkable results. However, existing methods are still facing content and structure feature leakage, unrealistic colouring and poor scale invariability. Thus, this paper, proposes a Transformer‐based medical image colouring algorithm with long‐term dependency to avoid feature leakage of coloured images. To be specific, this method employs two different Transformer encoders to generate and encode feature sequences for grey‐scale medical images and real human colour slice images, respectively. Then, a novel multi‐layer Transformer decoder is used to stylize grey‐scale map image features based on the real physical colour feature sequences. For colouring images at different scales, we implement content‐ aware positional encoding with scale invariance and propose style‐aware positional encoding strategy to take realistic and physical colour prior into account. Extensive experimental results indicate our method has achieved better colourization effects than recent state‐of‐the‐art medical image colourization methods.