Towards EEG-to-Text: Handwritten Character Classification via Continuous Kinematic Decoding

Abstract

AbstractThe classification of handwritten letters from invasive neural signals has lately been subject of research to restore communication abilities in people with limited movement capacities. This study explores the classification of ten letters (a,d,e,f,j,n,o,s,t,v) from non-invasive neural signals of 20 participants using two methods: the direct classification from low-frequency and broadband electroencephalogram (EEG) and a two-step approach comprising the continuous decoding of hand kinematics and the application of those in subsequent classification. When using low-frequency EEG, results show moderate accuracies of 23.1 % for ten letters and 39.0 % for a subset of five letters with highest discriminability of the trajectories. The two-step approach yielded significantly higher performances of 26.2 % for ten letters and 46.7 % for the subset of five letters. Hand kinematics could be reconstructed with a correlation of 0.10 to 0.57 (average chance level: 0.04) between the decoded and original kinematic. The study shows the general feasibility of extracting handwritten letters from non-invasively recorded neural signals and indicates that the proposed two-step approach can improve performances. As an exploratory investigation of the neural mechanisms of handwriting in EEG, results suggest movement speed as the most informative kinematic for the decoding of short hand movements.