Automated scoring of the Rey-Osterrieth Complex Figure Test using a deep-learning algorithm

Objective To validate a fully automated scoring algorithm for the Rey-Osterrieth Complex Figure Test (ROCFT) by comparing the scoring results of the algorithm to the results of human raters. Method The algorithm consisted of a cascade of deep neural networks which were trained on human rater scores to extract the 18 segments of the figure, and to quantify the patient’s performance. Algorithm results were compared to six expert raters for 303 drawings. We tested whether the average correlation between algorithm scores and scores by all human raters was equivalent to the average inter-rater correlation (with equality bound Δr < .05). The immediate and delayed recall trial were used; the copy trial showed a strong ceiling effect. Results The mean Pearson correlation between raters was .94 (SD = 0.01). The correlation between to algorithm and the raters was .88 (SD = 0.02). A two-one-sided t-tests (TOST) equivalence test showed that these correlations were not strictly equivalent, t(5) = 4.02, p = .995, 95% CI [0.35, 0.52]. Conclusions Although not strictly equivalent to human ratings, the algorithm’s performance is high, approaching a level of reliability found among human raters. We expect that improved individual segment detection will bring the algorithm scoring accuracy on par with that of human raters. Algorithmic scoring of the ROCFT will likely save valuable time and lead to higher levels of standardization in clinical practice.