Vision Impairment prediction for patients diagnosed with Multiple Sclerosis: Cosmos based machine learning model training and evaluation

Abstract

AbstractObjectivesMultiple sclerosis (MS) is a complex autoimmune neurological disorder that frequently impacts vision. One of the most frequent initial presentations of MS is acute vision loss due to optic neuritis, an acute disorder caused by MS involvement with the optic nerve. While vision impairment is often the first sign of MS, it can occur or recur at any time during the patient’s course. In this study, we aim to develop and evaluate machine learning models to predict vision impairment in patients with MS, both at the time of first MS diagnosis and throughout their course of care. Early awareness and intervention in patients likely to have vision loss can help preserve patient quality of life.Materials and MethodsUsing the Epic Cosmos de-identified electronic health record (EHR) dataset, we queried 213+ million patients to extract our MS cohort. Cases were defined as MS patients with vision impairment or optic neuritis (VI) following their first MS diagnosis, while controls were MS patients without VI. We trained logistic regression (LR), light gradient boosting machine (LGBM), and recurrent neural network (RNN) models to predict future VI in MS patients. The models were evaluated for two distinct clinical tasks: prediction of VI at the time of the first MS diagnosis and prediction of VI at the most recent visit. Similarly, we trained the models on different segments of the patient trajectory including up until the first MS diagnosis (MS-First Diagnosis), or until the most recent visit before developing the outcome (MS-Progress) as well as the combination of both (MS-General). Finally, we trained a survival model with the goal of predicting patient likelihood of vision loss over time. We compared the models’ performance using AUROC, AUPRC, and Brier scores.ResultsWe extracted a cohort of 377,097 patients with MS, including 42,281 VI cases. Our trained models achieved ∼80% AUROC, with RNN-based models outperforming LGBM and LR (79.6% vs 72.8% and 68.6%, respectively) when considering the full patient trajectory. The MS-General RNN model had the highest AUROC (64.4%) for predicting VI at the first MS diagnosis. The MS-Progress survival model achieved a 75% concordance index on the full trajectory, while the more clinically relevant MS-First Diagnosis model achieved 63.1% at initial diagnosis.Discussion and ConclusionThe MS-Progress and MS-General RNN models performed best in both prediction scenarios. While MS-General achieved the best performance at the time of first MS diagnosis with around 1% AUROC increase compared to the MS-First Diagnosis model, it showed around 1% AUROC decrease on the MS progress scenario. All RNN survival models performed the best when they were trained on data corresponding to the evaluation use-case scenarios. RNN based models showed promising performance that demonstrates that they can be useful clinical tools to predict risk of future VI events in patients with MS. Further development of these models will focus on expanding to predict other comorbidities associated with MS relapse or progression.