1. Prospective validation of a seizure diary forecasting falls short.
- Author
-
Goldenholz DM, Eccleston C, Moss R, and Westover MB
- Subjects
- Humans, Female, Male, Prospective Studies, Adult, Middle Aged, Epilepsy diagnosis, Artificial Intelligence trends, Young Adult, Deep Learning trends, Algorithms, Diaries as Topic, Cohort Studies, Aged, Seizures diagnosis, Forecasting methods
- Abstract
Objective: Recently, a deep learning artificial intelligence (AI) model forecasted seizure risk using retrospective seizure diaries with higher accuracy than random forecasts. The present study sought to prospectively evaluate the same algorithm., Methods: We recruited a prospective cohort of 46 people with epilepsy; 25 completed sufficient data entry for analysis (median = 5 months). We used the same AI method as in our prior study. Group-level and individual-level Brier Skill Scores (BSSs) compared random forecasts and simple moving average forecasts to the AI., Results: The AI had an area under the receiver operating characteristic curve of .82. At the group level, the AI outperformed random forecasting (BSS = .53). At the individual level, AI outperformed random in 28% of cases. At the group and individual level, the moving average outperformed the AI. If pre-enrollment (nonverified) diaries (with presumed underreporting) were included, the AI significantly outperformed both comparators. Surveys showed most did not mind poor-quality LOW-RISK or HIGH-RISK forecasts, yet 91% wanted access to these forecasts., Significance: The previously developed AI forecasting tool did not outperform a very simple moving average forecasting in this prospective cohort, suggesting that the AI model should be replaced., (© 2024 International League Against Epilepsy.)
- Published
- 2024
- Full Text
- View/download PDF