An Analysis on Multimodal Framework for Silent Speech Recognition

Abstract

A brain-computer interface (BCI) is a computer-based system that collects, analyses, and converts brain signals into commands that are sent to an output device to perform a desired action. BCI is used as an assistive and adaptive technology to track the brain activity. A silent speech interface (SSI) is a system that enables speech communication when an acoustic signal is unavailable. An SSI creates a digital representation of speech by collecting sensor data from the human articulatory, their neural pathways, or the brain. The data from a single stage is very minimal in order to capture for further processing. Therefore, multiple modalities could be used; a more complete representation of the speech production model could be developed. The goal is to detect speech tokens from speech imagery and create a language model. The proposal consists of multiple modalities by taking inputs from various biosignal sensors. The main objective of the proposal is to develop a BCI-based end-to-end continuous speech recognition system.