This section provides the general overview of imagined speech decoding system. The system consists of the following stages as shown in Figure 1. Data acquisition 2. Pre-processing 3. Feature extraction 4. Classification and 5. Performance analysis and evaluation.

Data acquisition in speech decoding system is a process of collection and recording of neural data from the subjects participated in an experiment. Imagine speech dataset can be recorded either through invasive or noninvasive. Invasive EEG are called Electrocorticography (ECoG) while noninvasive method is by using EEG. This method is commonly applied as it has low risk and safer to the subjects [23]. During the experiments, the subjects have to imagine a set of choosing phonemes, syllables, phrases or sentences while their EEG signals are recorded. The experimental protocol involves pre-trial period in which the subjects were prepared and acquainted with the experimental target. Following the pre-trial, the subjects were asked to imagine the set of stimuli in pre-defined time interval. Example of time-locked experiment conducted in [7] is shown in Figure 2.

To improve the efficiency of a classifier and also to reduce its computational complexity, the imagine speech data need to be preprocessed because not all the recorded data that are useful in classification stage as artifacts and noise are induced and contaminate the EEG data during the acquisition process. These artifacts include heart beat artifacts (ECG), eye winks (EOG), muscle movements (EMG), those caused by electrode faults, power line, and interference from equipment’s and devices [25-27]. The artifacts related to EEG signals can be divided into two types depending on their source. Those that their source is internally from a biological activities of a body are called as interior artifacts, while those that their source is external are called as exterior artifacts. The summary of types of artifacts are listed in Table 1. In addition, the EEG signal has a low signal to noise ratio, therefore it is highly imperative to preprocess the data which involves the process such as down sampling, windowing, and filtering.

Feature extraction involves extracting relevant and significant features that characterize the neural signals of imagined speech to be fed to the classifier. It is a very important step as it also determines the classifier’s accuracy by applying different techniques to ensure non-redundant and most discriminative information in the EEG signals are extracted. Domain approaches that are commonly employed in the feature extraction stage are time-domain, frequency-domain, time-frequency domain, and spatial domain. Other researchers combine two or more approaches in their imagine speech models termed as hybrid techniques. Details of the feature extraction techniques are discussed in section 3 and appendix II.

Early studies of speech recognition focused on identifying areas responsible for language processing in the human brain which uses patients that undergoing neurosurgery and those of neurological damage [28]. These studies began as early as 1861 by Paul Broca, a neurosurgeon who investigated the human brains using nine patients with lesions and find out that the language areas in humans are located at left hemisphere of the posterior frontal gyrus in the region called Broca’s area. Some years after, Carl Wernickle identified an area known as Wernicke’s area which he hypothesized that the posterior part of the left temporal lobe is also included in the language comprehension. Several researches have explored the speech and language processing neural regions and developed some functional models and their functional significance [29-31]. These models help in identifying areas that are involve in production, planning, and perception of speech. Figure 3 depicts these areas such as primary motor cortex, pre-motor cortex, Broca’s area, Wernicke’s area, primary auditory cortex etc.

Figure 4 shows how the EEG is recorded noninvasively using electrodes placed on the scalp with signals displayed on a computer to depicts the electrical activity of the brain when electrodes detected electrical charges. However, in some specific applications, invasive electrodes can be used termed as intracranial EEG. These electrical recordings from the surface of the brain or even from the outer surface of the head reveals that there is continuous electrical activity in the brain [33]. Based on the state of the brain, the frequencies and amplitudes of the brain signals changes such as during sleep, wakefulness, in a disease state like dementia, epilepsy, sleep disorder, etc. or in mental state. Figure 5 shows an example of normal EEG signals. EEG signal is measured as the potential difference over time, between the active electrode and the reference electrode. The international 10–20 system accessed from Brain Master Technologies Inc. [34] is shown in Figure 6. The multichannel EEG sets contain up to 128 or 256 active electrodes. These electrodes are made of silver chloride (AgCl). A gel is used to creates a conductive path between the skin and the electrode for the flow of current. Electrodes that do not use gels, called ‘dry’ electrodes are made of materials such as titanium and stainless-steel.

Sophisticated predictive models are required for targeting BCI application to decode cognitive functions in real time for researchers to use multivariable neural features in complex and rich behavioral conditions [35,36]. A regression framework is widely used modelling method to link neural processes, mental state, and stimulus features. For example, we can model the stimulus features at particular instance as a weighted sum of the neural processes as in equation (1).

\begin{equation} Y\left ( t \right )=\sum w\left ( p \right )\cdot X\left ( t,p \right ) \tag{1} \end{equation}

Where, \( Y\left ( t \right ) \) is the stimulus feature at time \( t \) , \( w\left ( p \right ) \) is the weight for a given feature \(p\), \( X\left ( t,p \right ) \) is the neural processes at time instance \( t \) and feature \( p \).

Classification can also be used as a decoding model in which from a finite set of options a neural activity can be recognized as a member to a discrete event type. Several machine learning algorithms can be use by both models such as support vector machine, neural networks, hidden markov models, and simple regression methods among others [37]. Some studies focused on summarizing the perception and imagination of speech and music into various models which relates the neuron’s responses with an auditory stimulus. These models include the one proposed by Kaneshiro et al. [38], Geirnaert et al. [39], music perception models were proposed and reported in [40-42].

For the successful implementation of BCI system using imagined speech based on EEG recordings, the future research direction must focus on key and challenging step which is to apply the different levels of speech processing and representation to imagined speech due to the absent of behavioral output and difficulty in monitoring the spectrotemporal structure. To time-lock a brain processes to a behavioral state or a quantifiable stimulus is complex task as an experimenter cannot directly monitor the imagination. Therefore, some standards techniques/models that matched input-output data has to be employed. Also, several factors such as age, emotion, gender, dialect, and pronunciation affects natural speech which results in temporal irregularities. Other challenges explored from our study with their remedy include those associated with how to design a good experimental task, proper training of participants, generating sufficient amount of data, using effective and good electrodes as well as improve their design, employing unsupervised machine learning techniques among others. Recently, most of the researchers focused their attention on applying deep learning algorithm in decoding and recognition of imagined speech, therefore huge amount of data must be generated to train the network effectively. Also, the requirement of higher computational resources in deep learning models must be addressed to enable other researches to experiment and evaluate their models to realize precise, practical, and reliable non-invasive models. Finally, homogeneous performance comparison among the developed techniques is difficult as there is lack of standardization for evaluating their performance due to different datasets with different sampling frequency, number of electrodes and other parameters.

In conclusion, this paper highlights the progress and explored the potential of using various decoding models and algorithms through collection of studies that used EEG signals recording to identified neural mechanisms related to complex speech production and functions. These models are also capable of recognizing and characterizing the important components of natural communication, that is speech and language perception and production directly from brain activity. Several studies reviewed in this paper reveals a promising result for classification of either phonemes, vowels and words from EEG brain signals but still shows that a lot of work need to be conducted to provide a realistic and efficient brain machine interface and neuroprosthetic devices. Recent machine learning techniques such as deep learning need to be further investigated, improvements in areas of experimental paradigm is paramount for better recognition, preprocessing, and feature extraction as they are the key aspects in development of new communication interfaces.