Prospective validation of a seizure diary forecasting falls short

Abstract

AbstractObjectiveRecently, a deep learning artificial intelligence (AI) model forecasted seizure risk using retrospective seizure diaries with higher accuracy than random forecasts. The present study sought to prospectively evaluate the same algorithm.MethodsWe recruited a prospective cohort of 46 people with epilepsy; 25 completed sufficient data entry for analysis (median = 5 months). We used the same AI method as in our prior study. Group‐level and individual‐level Brier Skill Scores (BSSs) compared random forecasts and simple moving average forecasts to the AI.ResultsThe AI had an area under the receiver operating characteristic curve of .82. At the group level, the AI outperformed random forecasting (BSS = .53). At the individual level, AI outperformed random in 28% of cases. At the group and individual level, the moving average outperformed the AI. If pre‐enrollment (nonverified) diaries (with presumed underreporting) were included, the AI significantly outperformed both comparators. Surveys showed most did not mind poor‐quality LOW‐RISK or HIGH‐RISK forecasts, yet 91% wanted access to these forecasts.SignificanceThe previously developed AI forecasting tool did not outperform a very simple moving average forecasting in this prospective cohort, suggesting that the AI model should be replaced.