Optimization of vision transformer-based detection of COVID-19 from chest X-ray images

Abstract

Abstract Background: For diagnosing coronavirus disease 2019 (COVID-19), chest X-rays have emerged as a preferred modality because of their accessibility, affordability, and capability to identify various pathologies. Recent advances in deep learning algorithms have shown promise in distinguishing COVID-19 from other lung diseases. However, the use of different optimization methods can affect the performance of the deep learning models. We aimed to compare the effects of the different optimization methods, identifying the best-performing algorithms for the detection of COVID-19 using chest X-rays. Methods: Chest X-ray images, including the seven classes of Normal, COVID-19, Viral Pneumonia, Bacterial Pneumonia, Middle East Respiratory Syndrome (MERS), Severe Acute Respiratory Syndrome (SARS), and Tuberculosis, were obtained. We trained the Vision Transformer (ViT) model using different optimizers such as Adaptive Moment Estimation (Adam), AdamW, Nesterov accelerated Adam (NAdam), Rectified Adam (RAdam), Stochastic Gradient Descent with weight decay (SGDW), and Momentum, and compared their performances. Results: We found that the RAdam optimizer at a learning rate of 10-5 achieved the highest accuracy, highest weighted average of F1-score, and lowest false negative rate of COVID-19 for both 4 Class and 7 Class Dataset. On the other hand, AdamW showed better performance for the samples with small sample sizes. The optimizers derived from Adam (i.e. Adam, AdamW, NAdam, and RAdam), showed robust results against different learning rates, while SGDW and Momentum showed less significant robustness. Conclusions: We suggest that Adam-derived optimizers, particularly RAdam, showed best performance in training the ViT model for detecting COVID-19 using chest X-ray images. Our results may help in the efforts to improve the performance of the model and to make it clinically useful.