Development of a scoring model for the Sharp/van der Heijde score using convolutional neural networks and its clinical application in predicting radiographic progression using a graph convolutional network

Abstract

ObjectiveTo construct a predictive model for the Sharp/van der Heijde score (SHS) and assess its applicability in clinical research settings.MethodsFirst, we built a rheumatoid arthritis (RA) image database linked to clinical information. We then constructed the prediction model in three steps: orientation, detection, and damage prediction. We assessed whether the SHS generated by our model replicated known findings on the association between radiographic progression and serological markers. Finally, after characterizing the SHSs of 4,264 patients using hierarchical clustering, we constructed a predictive model for joint destruction using a graph convolutional network (GCN).ResultsWe built a model with an accuracy of 100% in the correction of image orientation using EfficientNet and constructed a model with all predicted joint coordinates within 10 pixels of the correct coordinates using U-Net. In the damage prediction phase, the EfficientNet-based model combined with the modules achieved correlation coefficients of 0.879 for erosion and 0.868 for joint space narrowing between the model and expert, exceeding that of the previous best model. Our model replicated the known finding of erosion progression’s association with serologically positive patients. The areas under the curve for predicting finger and wrist erosion in the GCN model were 0.800 and 0.748, respectively. We observed that clusters generated by hierarchical clustering ranking in the top 10 were important features in the GCN for predicting erosion.ConclusionWe constructed a high-performance scoring model for SHSs applicable to clinical research. Our analysis revealed that clusters are important for predicting erosion using the GCN.Key messagesWhat is already known about this subject?►Several deep learning models that automatically predict Sharp/van der Heijde scores (SHSs) have been reported. However, the accuracy of their joint detection and erosion prediction was insufficient, and more importantly their clinical applicability was unclear.►Models for predicting joint destruction using clinical factors have been constructed with arbitrary factor selection by humans. No report has demonstrated the usefulness of deep learning models in predicting joint destruction using large SHS datasets.What does this study add?►Our deep learning model showed a high performance in both joint space narrowing and erosion, replicated previous findings on association between joint destruction and serological markers, thereby demonstrating, for the first time, that deep learning models could be clinically applicable in estimating SHSs.►We also demonstrated that a graph convolutional network (GCN) is a high performance model in predicting radiographic progression.How might this impact clinical practice or future developments?►We believe our model will be an essential tool for future studies, such as in genome-wide association studies (GWAS) for joint destruction on a scale of thousands to millions, which is difficult to achieve with human scoring. Ultimately, data from large-scale GWAS will be integrated into the GCN to build a powerful model for precision medicine.