A Siamese U-Transformer for change detection on MRI brain for multiple sclerosis, a model development and external validation study

Abstract

SummaryBackgroundMultiple Sclerosis (MS), is a chronic idiopathic demyelinating disorder of the CNS. Imaging plays a central role in diagnosis and monitoring. Monitoring for progression however, can be repetitive for neuroradiologists, and this has led to interest in automated lesion detection. Simultaneously, in the computer science field of Remote Sensing, Change Detection (CD), the identification of change between co-registered images at different times, has been disrupted by the emergence of Vision Transformers. CD offers an alternative to semantic segmentation leveraging the temporal information in the data.MethodsIn this retrospective study with external validation we reframe the clinical radiology task of new lesion identification as a CD problem. Consecutive patients who had MRI studies for MS at our institution between 2019 and 2022 were reviewed and those with new lesion(s) were included. External data was obtained from the MSSEG2 challenge and OpenMS. Multiple CD models, and a novel model (NeUFormer), were trained and tested. Results were analysed on both paired slices and at the patient level. Expected Cost (EC) and F2 were independently and prospectively chosen as our primary evaluation metrics. For external data we report DICE and F1 to allow for comparison with existing data. For each test set 1000 bootstrapping simulations were performed by sampling 10 patient samples with replacement giving a non parametric estimate of the confidence interval. Wilcoxon statistics were calculated to test for significance.Findings43,440 MR images were included for analysis (21,720 pairs). The internal set comprised of 170 patients (110 for training, 30 for tuning, 30 testing) with 120 females and 50 males, average age of 42 (range 21 – 74). 60 (40 + 20) patients were included for external validation.In the CD experiments (2D) our proposed NeuFormer model achieved the best (lowest) Expected Cost (EC) (p=0.0095), the best F2 and second best DICE (p<0.0001). At the patient level our NeUFormer model had the joint highest number of True Positive lesions, and lowest number of False negatives (p<0.002). For CD on external data, NeUFormer achieved the highest DICE on both datasets (p<0.0001). NeUFormer had the lowest or joint lowest number of False Positives on external data (p<0.0001 in all cases).InterpretationReformulating new lesion identification as a CD problem allows the use of new techniques and methods of evaluation. We introduce a novel Siamese U-Transformer, NeUFormer, which combines concepts from U-Net, Siamese Networks, and vision transformers to create a model with improved small lesion detection and the consistently best EC. Its ability to increase detection of small lesions, balanced with relatively few false positives, and superior generalisability has the potential to greatly impact the field of the identification of radiologic progression of MS with AI.Research in contextEvidence before this studyMultiple Sclerosis (MS), a chronic and idiopathic demyelinating disorder of the CNS, is diagnosed using the McDonald criteria based on MRI interpretation. Without a definitive MS biomarker, AI holds promise is for uncovering unique features indicative of MS, improving diagnostics and identifying progression. Research in the field typically centres on segmentation and classification, leaving a gap in evaluating temporal imaging changes. The MSSEG2 challenge has now enabled more research into new lesion identification in MS. Even so, most solutions are based on semantic segmentation architectures and rely on limited metrics for evaluation. The identification of small lesions also remains a challenge.Remote Sensing (RS) is the science of obtaining information about objects or areas from a distance, typically from aircraft or satellites. In the RS literature, Change Detection (CD) refers to the identification of significant alterations in co-registered images captured at different times. In this way CD offers an alternative to semantic segmentation leveraging the temporal information in the data. This field was dominated by convolutional neural networks but has recently been disrupted by transformer-based architectures. Transformers, fuelled by their success in NLP, are gaining popularity across all computer vision tasks due to their larger effective receptive field and enhanced context modelling between image pixels. Inspired by these developments, we incorporate some of these ideas into our NeUFormer model.Added value of this studyThis study redefines the task of identifying progression on MRI brain in MS as a CD problem, borrowing concepts from RS. This approach allows for both pixel- and patient-level evaluation and rethinks standard metrics to suit specific clinical needs. This acknowledges the distinction between trivial variation in segmentation and clinically significant change. State-of-the-art CD models are assessed at this task, and a novel model, NeuFormer, is introduced. NeuFormer synergistically combines concepts from the classical U-Net (which was originally intended for brain segmentation), Siamese architecture adaptations specifically for CD, Swin-UNETR (a U-Transformer developed by MONAI to integrate the shifting window structure of the Swin transformer into medical imaging) and ChangeFormer which also uses attention at scale specifically for CD, leveraging improved spaciotemporal reasoning to create a model which is better for small lesion identification and with the consistently lowest EC associated with its decisions.Implications of all the available evidenceReframing lesion identification as CD enables an alternative to semantic segmentation leveraging the temporal information in the data, enhancing the model’s relevance and customization for specific medical tasks. We also propose the flexible Expected Cost metric, as it facilitates varying action thresholds and helps to customise tools to stakeholder preferences.Siamese vision transformers show promise for CD on MRI in MS including for smaller lesions which are traditionally difficult for computer vision models to identify. This may be to the intrinsic spaciotemporal advantages of vision transformers, with positional embedding, over patch based convolutional methods.NeUFormer’s ability to increase detection of small lesions, balanced with relatively few false positives and excellent generalisability has the potential to greatly impact the field of the identification of radiologic progression of MS with AI.