Prospective validation of a seizure diary forecasting falls short

Abstract

AbstractOBJECTIVERecently, a deep learning AI model forecasted seizure risk using retrospective seizure diaries with higher accuracy than random forecasts. The present study sought to prospectively evaluate the same algorithm.METHODSWe recruited a prospective cohort of 46 people with epilepsy; 25 completed sufficient data entry for analysis (median 5 months). We used the same AI method as in our prior study. Group-level and individual-level Brier Skill Scores (BSS) compared random forecasts and simple moving average forecasts to the AI.RESULTSThe AI had an AUC of 0.82. At the group level, the AI outperformed random forecasting (BSS=0.53). At the individual level, AI outperformed random in 28% of cases. At the group and individual level, the moving average outperformed the AI. If pre-enrollment (non-verified) diaries (with presumed under-reporting) were included, the AI significantly outperformed both comparators. Surveys showed most did not mind poor quality LOW-RISK or HIGH-RISK forecasts, yet 91% wanted access to these forecasts.SIGNIFICANCEThe previously developed AI forecasting tool did not outperform a very simple moving average forecasting this prospective cohort, suggesting that the AI model should be replaced.Key pointsA previously developed e-diary based AI seizure forecasting tool was prospectively tested. Although by some metrics the tool was successful, the overall AI performance was unacceptably low.It was much easier to outperform a random forecast; it was much harder to outperform a moving average forecast.Using unverified diaries can skew forecasting metrics in favor of underperforming tools.