Search

Numerous individuals encounter challenges in verbal communication due to various factors, including physical disabilities, neurological disorders, and strokes. In response to this pressing need, technology has actively pursued solutions to bridge the communication gap, recognizing the inherent difficulties faced in verbal communication, particularly in contexts where traditional methods may be inadequate. Electroencephalogram (EEG) has emerged as a primary non-invasive method for measuring brain activity, offering valuable insights from a cognitive neurodevelopmental perspective. It forms the basis for Brain-Computer Interfaces (BCIs) that provide a communication channel for individuals with neurological impairments, thereby empowering them to express themselves effectively. EEG-based BCIs, especially those adapted to decode imagined speech from EEG signals, represent a significant advancement in enabling individuals with speech disabilities to communicate through text or synthesized speech. By utilizing cognitive neurodevelopmental insights, researchers have been able to develop innovative approaches for interpreting EEG signals and translating them into meaningful communication outputs. To aid researchers in effectively addressing this complex challenge, this review article synthesizes key findings from state-of-the-art significant studies. It investigates into the methodologies employed by various researchers, including preprocessing techniques, feature extraction methods, and classification algorithms utilizing Deep Learning and Machine Learning approaches and their integration. Furthermore, the review outlines the potential avenues for future research, with the goal of advancing the practical implementation of EEG-based BCI systems for decoding imagined speech from a cognitive neurodevelopmental perspective. [ABSTRACT FROM AUTHOR]

Speech imagery is emerging as a significant neuro‐paradigm for designing an electroencephalography (EEG)‐based brain–computer interface (BCI) system for the purpose of rehabilitation, medical neurology, and to aid people with disabilities in interacting with their surroundings. Neural correlates of speech imagery EEG signals are variable and weak as compared to the vocal state; hence, it is challenging to interpret them using machine learning (ML)–based classifiers. The applicability of modern deep learning methods such as convolutional neural networks (CNNs) and bidirectional long short‐term memory (Bi‐LSTM) networks has seen substantial advances in complex EEG signal analysis as compared to ML‐based methods. The objective of this article is to design a firefly‐optimized discrete wavelet transform (DWT) and CNN‐Bi‐LSTM–based imagined speech recognition (ISR) system to interpret imagined speech EEG signals. This study utilizes two publicly available datasets. EEG signal is enhanced using firefly optimization algorithm (FOA)–based optimized soft thresholding of high‐frequency detail components obtained by DWT decomposition. The enhanced EEG signal is augmented using sliding window data augmentation to increase the training data. Frequency‐domain features like power spectral density (PSD), frequency band power (FBP), band ratios, peak frequency, mean frequency, median frequency, spectral entropy, and relative power are extracted from augmented EEG segments. The extracted feature vector is fed to the designed CNN‐Bi‐LSTM classifier such that the EEG data are classified into two‐class, three‐class, and four‐class categories. To achieve optimal performance, the CNN‐Bi‐LSTM model was optimized using the Keras tuner library. The designed CNN consists of one‐dimensional (1‐D) convolutional layers and max pooling layers for familiarizing local associations along with mining hierarchical connections, and the Bi‐LSTM network acquires long‐term dependencies from the features learned by the former CNN. Bi‐LSTM network improves the performance and acquires potentially more affluent representations by looking at the sequence in both forward and reverse ways to capture representations that might be left unexploited by the sequential‐order kind alone. The performance of the designed FOA‐DWT‐CNN‐Bi‐LSTM–based ISR system is assessed using four evaluation measures: accuracy, F1 score, recall, and precision. It is found that the proposed system achieves the highest classification accuracy of 99.43 ± 2.5%, 94.41 ± 3.31%, and 89.57 ± 4.3% for two‐class, three‐class, and four‐class categories, respectively. [ABSTRACT FROM AUTHOR]

The classification performance of endogenous electroencephalogram (EEG) brain-computer interfaces (BCIs) can be improved by hybridizing the paradigm through the use of commands from multiple paradigms. Hybrid paradigms using motor imagery (MI) and speech imagery (SI) have shown promise, but there is a lack of research into: i) their effectiveness when compared to pure MI and SI for multiclass problems, and ii) automated command selection. This study investigates multiclass MI and SI hybrid paradigms and compares the results to those obtained using pure MI and SI. Performance was assessed using F1 score and accuracy. The performances of all possible hybrid paradigm designs were assessed. The analysis indicated that hybridization does not always guarantee improved performance when compared to the pure paradigms, and there is inter-subject variation in the best paradigm. This confirmed the need for automated subject-specific hybrid paradigm designs. An automated hybrid paradigm selection technique using successive halving (SH) for expedited computational times was developed and results were compared to those obtained using a standard grid search. The SH approach resulted in an improvement in F1 score of 21.09% and 36.86% compared to MI and SI and led to a reduction in computational times of 82.80% compared to grid search. [ABSTRACT FROM AUTHOR]

Brain-Computer Interfaces (BCIs) aim to establish a pathway between the brain and an external device without the involvement of the motor system, relying exclusively on neural signals. Such systems have the potential to provide a means of communication for patients who have lost the ability to speak due to a neurological disorder. Traditional methodologies for decoding imagined speech directly from brain signals often deploy static classifiers, that is, decoders that are computed once at the beginning of the experiment and remain unchanged throughout the BCI use. However, this approach might be inadequate to effectively handle the non-stationary nature of electroencephalography (EEG) signals and the learning that accompanies BCI use, as parameters are expected to change, and all the more in a real-time setting. To address this limitation, we developed an adaptive classifier that updates its parameters based on the incoming data in real time. We first identified optimal parameters (the update coefficient, UC) to be used in an adaptive Linear Discriminant Analysis (LDA) classifier, using a previously recorded EEG dataset, acquired while healthy participants controlled a binary BCI based on imagined syllable decoding. We subsequently tested the effectiveness of this optimization in a real-time BCI control setting. Twenty healthy participants performed two BCI control sessions based on the imagery of two syllables, using a static LDA and an adaptive LDA classifier, in randomized order. As hypothesized, the adaptive classifier led to better performances than the static one in this real-time BCI control task. Furthermore, the optimal parameters for the adaptive classifier were closely aligned in both datasets, acquired using the same syllable imagery task. These findings highlight the effectiveness and reliability of adaptive LDA classifiers for real-time imagined speech decoding. Such an improvement can shorten the training time and favor the development of multi-class BCIs, representing a clear interest for non-invasive systems notably characterized by low decoding accuracies. [ABSTRACT FROM AUTHOR]

We propose a Transfer learning-enabled electroencephalography-based intuitive brain-computer interface system by utilizing phase-based brain functional connectivity methods such as phase lag index (PLI) and Intersite phase clustering (ISPC) along with power features to explore both phase and power-based information from electroencephalography (EEG) signals. Time-frequency decomposition using a complex morlet wavelet is applied to analyze the signal components in both the time and frequency domains and extract phase connectivity and power features. Functional connectivity methods aim to recognize functional interactions and statistical mutuality among signals acquired across various brain areas. The phase-based connectivity features are extracted simultaneously for multiple channels to investigate the phase synchronization among EEG signals across the entire brain. Next, Graph theory is adopted to trace connectivity between brain regions by calculating the connectivity degree of extracted PLI and ISPC features with other electrodes. In Parallel, Discrete wavelet convolution is performed to calculate the time variable frequency band’s specific power from the imagined speech EEG data. Finally, Time-frequency images of the above-mentioned PLI, ISPC, and EEG power features are fed as input to DenseNet-121 architecture for classification. Dense Net architecture overcomes the problem of ‘vanishing gradient’ by connecting each layer directly with other layers, making the network densely connected. The maximum classification accuracy achieved is 100%, 99.14%, and 98.72% for binary, three-class, and four-class classifications, respectively. The experimental results indicate that the proposed phase-based connectivity features, EEG power, and the DenseNet-121 model have achieved excellent accuracy for two public datasets, outperforming the state-of-the-art methods. The outstanding results strengthen the possibility of real-time EEG-based intuitive brain-computer interface communication.

Many current brain–computer interface (BCI) applications depend on the quick processing of brain signals. Most researchers strive to create new methods for future implementation and enhance existing models to discover an optimal feature set that can operate independently. This study focuses on four key concepts that will be used to complete future works. The first concept is related to potential future communication models, whereas the others aim to enhance previous models or methodologies. The four concepts are as follows. First, we suggest a new communication imagery model as a substitute for a speech imager that relies on a mental task approach. As speech imagery is intricate, one cannot imagine the sounds of every character in every language. Our study proposes a new mental task model for lip-sync imagery that can be employed in all languages. Any character in any language can be used with this mental task model. In this study, we utilized two lip-sync movements to indicate two sounds, characters, or letters. Second, we considered innovative hybrid signals. Choosing an unsuitable frequency range can lead to ineffective feature extractions. Therefore, the selection of an appropriate frequency range is crucial for processing. The ultimate goal of this method is to accurately discover distinct frequencies of brain imagery activities. The restricted frequency range combination presents an initial proposal for generating fragmented, continuous frequencies. The first model assesses two 4 Hz intervals as filter banks. The primary objective is to discover new combinations of signals at 8 Hz by selecting filter banks with a 4 Hz scale from the frequency range of 4 Hz to 40 Hz. This approach facilitates the acquisition of efficient and clearly defined features by reducing similar patterns and enhancing distinctive patterns of brain activity. Third, we introduce a new linear bond graph classifier as a supplement to a linear support vector machine (SVM) when handling noisy data. The performance of the linear support vector machine (SVM) significantly declines under high-noise conditions. To complement the linear support vector machine (SVM) in noisy-data situations, we introduce a new linear bond graph classifier. Fourth, this paper presents a deep-learning model for formula recognition that converts the first-layer data into a formula extraction model. The primary goal is to decrease the noise in the formula coefficients of the subsequent layers. The output of the final layer comprises coefficients chosen by different functions at various levels. The classifier then extracts the root interval for each formula, and a diagnosis is established based on these intervals. The final goal of the last idea is to explain the main brain imagery activity formula using a combination formula for similar and distinctive brain imagery activities. The results of implementing all of the proposed methods are reported. The results range between 55% and 98%. The lowest result is 55% for the deep detection formula, and the highest result is 98% for new combinations of signals. [ABSTRACT FROM AUTHOR]

Speech imagery is one type of mental imagery specific to processing verbal sequences and plays a vital role in human thought processes. Speech imagery has become an interesting paradigm for researchers as speech imagery has a high similarity to real voice communication. Electroencephalography (EEG) is a noninvasive electrophysiological technique that measures the mental state of the brain directly from the scalp. The nature of the acquired EEG signals is nonlinear and nonstationary. As EEG signals have a low signal-to-noise ratio (SNR), artifacts occur during acquisition. Hence, an efficient framework of pre-processing is required to obtain artifact-free EEG for further applications. Selection of the optimal pre-processing techniques for EEG still remains a challenging task. This work focuses on employing and comparing the different pre-processing techniques and lists out the optimal solutions for pre-processing Speech imagery-based EEG signals. The techniques are compared based on the Mean Square Error and Peak Signal-to-Noise Ratio values. [ABSTRACT FROM AUTHOR]

Brain computer interface (BCI) based on speech imagery can help people with motor disorders communicate their thoughts to the outside world in a natural way. Due to being portable, non-invasive, and safe, functional near-infrared spectroscopy (fNIRS) is preferred for developing BCIs. Previous BCIs based on fNIRS mainly relied on activation information, which ignored the functional connectivity between neural areas. In this study, a 4-class speech imagery BCI based on fNIRS is presented to decode simplified articulation motor imagery (only the movements of jaw and lip were retained) of different vowels. Synchronization information in the motor cortex was extracted as features. In multiclass (four classes) settings, the mean subject-dependent classification accuracies approximated or exceeded 40% in the 0-2.5 s and 0-10 s time windows, respectively. In binary class settings (the average classification accuracies of all pairwise comparisons between two vowels), the mean subject-dependent classification accuracies exceeded 70% in the 0-2.5 s and 0-10 s time windows. These results demonstrate that connectivity features can effectively differentiate different vowels even if the time window size was reduced from 10 s to 2.5 s and the decoding performance in both the time windows was almost the same. This finding suggests that speech imagery BCI based on fNIRS can be further optimized in terms of feature extraction and command generation time reduction. In addition, simplified articulation motor imagery of vowels can be distinguished, and therefore, the potential contribution of articulation motor imagery information extracted from the motor cortex should be emphasized in speech imagery BCI based on fNIRS to improve decoding performance.

Speech imagery recognition from electroencephalograms (EEGs) could potentially become a strong contender among non-invasive brain-computer interfaces (BCIs). In this report, first we extract language representations as the difference of line-spectra of phones by statistically analyzing many EEG signals from the Broca area. Then we extract vowels by using iterative search from hand-labeled short-syllable data. The iterative search process consists of principal component analysis (PCA) that visualizes linguistic representation of vowels through eigen-vectors 0(m), and subspace method (SM) that searches an optimum line-spectrum for redesigning 0(m). The extracted linguistic representation of Japanese vowels /i/ /e/ /a/ /o/ /u/ shows 2 distinguished spectral peaks (P1, P2) in the upper frequency range. The 5 vowels are aligned on the P1-P2 chart. A 5-vowel recognition experiment using a data set of 5 subjects and a convolutional neural network (CNN) classifier gave a mean accuracy rate of 72.6%. [ABSTRACT FROM AUTHOR]

Brain-Computer Interfaces (BCIs) aim to establish a pathway between the brain and an external device without the involvement of the motor system, relying exclusively on neural signals. Such systems have the potential to provide a means of communication for patients who have lost the ability to speak due to a neurological disorder. Traditional methodologies for decoding imagined speech directly from brain signals often deploy static classifiers, that is, decoders that are computed once at the beginning of the experiment and remain unchanged throughout the BCI use. However, this approach might be inadequate to effectively handle the non-stationary nature of electroencephalography (EEG) signals and the learning that accompanies BCI use, as parameters are expected to change, and all the more in a real-time setting. To address this limitation, we developed an adaptive classifier that updates its parameters based on the incoming data in real time. We first identified optimal parameters (the update coefficient, UC) to be used in an adaptive Linear Discriminant Analysis (LDA) classifier, using a previously recorded EEG dataset, acquired while healthy participants controlled a binary BCI based on imagined syllable decoding. We subsequently tested the effectiveness of this optimization in a real-time BCI control setting. Twenty healthy participants performed two BCI control sessions based on the imagery of two syllables, using a static LDA and an adaptive LDA classifier, in randomized order. As hypothesized, the adaptive classifier led to better performances than the static one in this real-time BCI control task. Furthermore, the optimal parameters for the adaptive classifier were closely aligned in both datasets, acquired using the same syllable imagery task. These findings highlight the effectiveness and reliability of adaptive LDA classifiers for real-time imagined speech decoding. Such an improvement can shorten the training time and favor the development of multi-class BCIs, representing a clear interest for non-invasive systems notably characterized by low decoding accuracies.

Speech imagery recognition from electroencephalograms (EEGs) could potentially become a strong contender among non-invasive brain-computer interfaces (BCIs). In this report, first we extract language representations as the difference of line-spectra of phones by statistically analyzing many EEG signals from the Broca area. Then we extract vowels by using iterative search from hand-labeled short-syllable data. The iterative search process consists of principal component analysis (PCA) that visualizes linguistic representation of vowels through eigen-vectors φ(m), and subspace method (SM) that searches an optimum line-spectrum for redesigning φ(m). The extracted linguistic representation of Japanese vowels /i/ /e/ /a/ /o/ /u/ shows 2 distinguished spectral peaks (P1, P2) in the upper frequency range. The 5 vowels are aligned on the P1-P2 chart. A 5-vowel recognition experiment using a data set of 5 subjects and a convolutional neural network (CNN) classifier gave a mean accuracy rate of 72.6%.

Brain computer interface (BCI) based on speech imagery can help people with motor disorders communicate their thoughts to the outside world in a natural way. Due to being portable, non-invasive, and safe, functional near-infrared spectroscopy (fNIRS) is preferred for developing BCIs. Previous BCIs based on fNIRS mainly relied on activation information, which ignored the functional connectivity between neural areas. In this study, a 4-class speech imagery BCI based on fNIRS is presented to decode simplified articulation motor imagery (only the movements of jaw and lip were retained) of different vowels. Synchronization information in the motor cortex was extracted as features. In multiclass (four classes) settings, the mean subject-dependent classification accuracies approximated or exceeded 40% in the 0-2.5 s and 0-10 s time windows, respectively. In binary class settings (the average classification accuracies of all pairwise comparisons between two vowels), the mean subject-dependent classification accuracies exceeded 70% in the 0-2.5 s and 0-10 s time windows. These results demonstrate that connectivity features can effectively differentiate different vowels even if the time window size was reduced from 10 s to 2.5 s and the decoding performance in both the time windows was almost the same. This finding suggests that speech imagery BCI based on fNIRS can be further optimized in terms of feature extraction and command generation time reduction. In addition, simplified articulation motor imagery of vowels can be distinguished, and therefore, the potential contribution of articulation motor imagery information extracted from the motor cortex should be emphasized in speech imagery BCI based on fNIRS to improve decoding performance. [ABSTRACT FROM AUTHOR]

In order to increase the number of active brain computer interface (BCI) instruction sets, an experimental paradigm of sequential coding based on motor imagery and speech imagery is proposed. By dividing one motor imagery and one speech imagery, four kinds of imagination modes are obtained: 1) Motor imagery; 2) Speech imagery; 3) Motor imagery followed by speech imagery; 4) Speech imagery followed by motor imagery. An attention and multi-scale neural network (AMEEGNet) is designed for the data set. Firstly, the robust time representation of the signal is extracted by dilated convolution and three two-dimensional convolution with different size scales. Then, deep convolution and separable convolution are used to extract spatial features and frequency domain features, respectively. In addition, the squeeze-excitation module is added to the model to extract features with high classification accuracy adaptively. Finally, a network layer with full connection is used for classification. The model achieves an average accuracy of 71.1% on a temporal coding experiment dataset with four kinds of imagination. One the same dataset, EEGNet, MMCNN, Shallow ConvNet and TSGL-EEGNet achieved 57.9%, 60.5%, 68.3% and 68.4% accuracy, respectively. It can be seen that the proposed model has the highest recognition accuracy. [ABSTRACT FROM AUTHOR]

Many current brain–computer interface (BCI) applications depend on the quick processing of brain signals. Most researchers strive to create new methods for future implementation and enhance existing models to discover an optimal feature set that can operate independently. This study focuses on four key concepts that will be used to complete future works. The first concept is related to potential future communication models, whereas the others aim to enhance previous models or methodologies. The four concepts are as follows. First, we suggest a new communication imagery model as a substitute for a speech imager that relies on a mental task approach. As speech imagery is intricate, one cannot imagine the sounds of every character in every language. Our study proposes a new mental task model for lip-sync imagery that can be employed in all languages. Any character in any language can be used with this mental task model. In this study, we utilized two lip-sync movements to indicate two sounds, characters, or letters. Second, we considered innovative hybrid signals. Choosing an unsuitable frequency range can lead to ineffective feature extractions. Therefore, the selection of an appropriate frequency range is crucial for processing. The ultimate goal of this method is to accurately discover distinct frequencies of brain imagery activities. The restricted frequency range combination presents an initial proposal for generating fragmented, continuous frequencies. The first model assesses two 4 Hz intervals as filter banks. The primary objective is to discover new combinations of signals at 8 Hz by selecting filter banks with a 4 Hz scale from the frequency range of 4 Hz to 40 Hz. This approach facilitates the acquisition of efficient and clearly defined features by reducing similar patterns and enhancing distinctive patterns of brain activity. Third, we introduce a new linear bond graph classifier as a supplement to a linear support vector machine (SVM) when handling noisy data. The performance of the linear support vector machine (SVM) significantly declines under high-noise conditions. To complement the linear support vector machine (SVM) in noisy-data situations, we introduce a new linear bond graph classifier. Fourth, this paper presents a deep-learning model for formula recognition that converts the first-layer data into a formula extraction model. The primary goal is to decrease the noise in the formula coefficients of the subsequent layers. The output of the final layer comprises coefficients chosen by different functions at various levels. The classifier then extracts the root interval for each formula, and a diagnosis is established based on these intervals. The final goal of the last idea is to explain the main brain imagery activity formula using a combination formula for similar and distinctive brain imagery activities. The results of implementing all of the proposed methods are reported. The results range between 55% and 98%. The lowest result is 55% for the deep detection formula, and the highest result is 98% for new combinations of signals.

We present a transfer learning-based approach for decoding imagined speech from electroencephalogram (EEG). Features are extracted simultaneously from multiple EEG channels, rather than separately from individual channels. This helps in capturing the interrelationships between the cortical regions. To alleviate the problem of lack of enough data for training deep networks, sliding window-based data augmentation is performed. Mean phase coherence and magnitude-squared coherence, two popular measures used in EEG connectivity analysis, are used as features. These features are compactly arranged, exploiting their symmetry, to obtain a three dimensional “image-like” representation. The three dimensions of this matrix correspond to the alpha, beta and gamma EEG frequency bands. A deep network with ResNet50 as the base model is used for classifying the imagined prompts. The proposed method is tested on the publicly available ASU dataset of imagined speech EEG, comprising four different types of prompts. The accuracy of decoding the imagined prompt varies from a minimum of 79.7% for vowels to a maximum of 95.5% for short-long words across the various subjects. The accuracies obtained are better than the state-of-the-art methods, and the technique is good in decoding prompts of different complexities.

Over the past decade, many researchers have come up with different implementations of systems for decoding covert or imagined speech from EEG (electroencephalogram). They differ from each other in several aspects, from data acquisition to machine learning algorithms, due to which, a comparison between different implementations is often difficult. This review article puts together all the relevant works published in the last decade on decoding imagined speech from EEG into a single framework. Every important aspect of designing such a system, such as selection of words to be imagined, number of electrodes to be recorded, temporal and spatial filtering, feature extraction and classifier are reviewed. This helps a researcher to compare the relative merits and demerits of the different approaches and choose the one that is most optimal. Speech being the most natural form of communication which human beings acquire even without formal education, imagined speech is an ideal choice of prompt for evoking brain activity patterns for a BCI (brain-computer interface) system, although the research on developing real-time (online) speech imagery based BCI systems is still in its infancy. Covert speech based BCI can help people with disabilities to improve their quality of life. It can also be used for covert communication in environments that do not support vocal communication. This paper also discusses some future directions, which will aid the deployment of speech imagery based BCI for practical applications, rather than only for laboratory experiments.

Over the past decade, many researchers have come up with different implementations of systems for decoding covert or imagined speech from EEG (electroencephalogram). They differ from each other in several aspects, from data acquisition to machine learning algorithms, due to which, a comparison between different implementations is often difficult. This review article puts together all the relevant works published in the last decade on decoding imagined speech from EEG into a single framework. Every important aspect of designing such a system, such as selection of words to be imagined, number of electrodes to be recorded, temporal and spatial filtering, feature extraction and classifier are reviewed. This helps a researcher to compare the relative merits and demerits of the different approaches and choose the one that is most optimal. Speech being the most natural form of communication which human beings acquire even without formal education, imagined speech is an ideal choice of prompt for evoking brain activity patterns for a BCI (brain-computer interface) system, although the research on developing real-time (online) speech imagery based BCI systems is still in its infancy. Covert speech based BCI can help people with disabilities to improve their quality of life. It can also be used for covert communication in environments that do not support vocal communication. This paper also discusses some future directions, which will aid the deployment of speech imagery based BCI for practical applications, rather than only for laboratory experiments. [ABSTRACT FROM AUTHOR]

Copyright of Journal of Signal Processing is the property of Journal of Signal Processing and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Retrieval of unintelligible speech is a basic need for speech impaired and is under research for several decades. But retrieval of random words from thoughts needs a substantial and consistent approach. This work focuses on the preliminary steps of retrieving vowels from Electroencephalography (EEG) signals acquired while speaking and imagining of speaking a consonant–vowel–consonant (CVC) word. The process, referred to as Speech imagery is imagining of speaking to oneself silently in the mind. Speech imagery is a form of mental imagery. Brain connectivity estimators such as EEG coherence, Partial Directed Coherence, Directed Transfer Function and Transfer Entropy have been used to estimate the concurrency and causal dependence (direction and strength) between different brain regions. From brain connectivity results it has been observed that the left frontal and left temporal electrodes were activated for speech and speech imagery processes. These brain connectivity estimators have been used for training Recurrent Neural Networks (RNN) and Deep Belief Networks (DBN) for identifying the vowel from the subject's thought. Though the accuracy level was found to be varying for each vowel while speaking and imagining of speaking the CVC word, the overall classification accuracy was found to be 72% while using RNN whereas a classification accuracy of 80% was observed while using DBN. DBN was found to outperform RNN in both the speech and speech imagery processes. Thus, the combination of brain connectivity estimators and deep learning techniques appear to be effective in identifying the vowel from EEG signals of subjects' thought. [ABSTRACT FROM AUTHOR]

Speech imagery has been successfully employed in developing Brain-Computer Interfaces because it is a novel mental strategy that generates brain activity more intuitively than evoked potentials or motor imagery. There are many methods to analyze speech imagery signals, but those based on deep neural networks achieve the best results. However, more research is necessary to understand the properties and features that describe imagined phonemes and words. In this paper, we analyze the statistical properties of speech imagery EEG signals from the KaraOne dataset to design a method that classifies imagined phonemes and words. With this analysis, we propose a Capsule Neural Network that categorizes speech imagery patterns into bilabial, nasal, consonant-vocal, and vowels/iy/ and/uw/. The method is called Capsules for Speech Imagery Analysis (CapsK-SI). The input of CapsK-SI is a set of statistical features of EEG speech imagery signals. The architecture of the Capsule Neural Network is composed of a convolution layer, a primary capsule layer, and a class capsule layer. The average accuracy reached is 90.88%±7 for bilabial, 90.15%±8 for nasal, 94.02%±6 for consonant-vowel, 89.70%±8 for word-phoneme, 94.33%± for/iy/ vowel and, 94.21%±3 for/uw/ vowel detection. Finally, with the activity vectors of the CapsK-SI capsules, we generated brain maps to represent brain activity in the production of bilabial, nasal, and consonant-vocal signals., Competing Interests: Declaration of Competing Interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper., (Copyright © 2023 Elsevier Ltd. All rights reserved.)

Humans excel at constructing mental representations of speech streams in the absence of external auditory input: the internal experience of speech imagery. Elucidating the neural processes underlying speech imagery is critical to understanding this higher-order brain function in humans. Here, using functional magnetic resonance imaging, we investigated the shared and distinct neural correlates of imagined and perceived speech by asking participants to listen to poems articulated by a male voice (perception condition) and to imagine hearing poems spoken by that same voice (imagery condition). We found that compared to baseline, speech imagery and perception activated overlapping brain regions, including the bilateral superior temporal gyri and supplementary motor areas. The left inferior frontal gyrus was more strongly activated by speech imagery than by speech perception, suggesting functional specialization for generating speech imagery. Although more research with a larger sample size and a direct behavioral indicator is needed to clarify the neural systems underlying the construction of complex speech imagery, this study provides valuable insights into the neural mechanisms of the closely associated but functionally distinct processes of speech imagery and perception., (© The Author(s) 2022. Published by Oxford University Press. All rights reserved. For permissions, please e-mail: journals.permission@oup.com.)

Brain-computer interfaces (BCIs) based on electroencephalography (EEG) have been attracted enough attention by researchers. In order to determine whether silent reading can improve mental tasks for BCI systems, this paper proposed a two-step experiment: mental tasks with speech imagery and mental tasks without speech imagery. Reading Chinese characters in mind is set as speech imagery. Since Chinese characters are monosyllabic, it is very convenient to read them in mind with related mental tasks simultaneously. Ten Chinese subjects are trained by two steps in this experiment. Feature vectors of EEG signals are extracted and classified by common spatial patterns (CSP) and support vector machine (SVM), respectively. Compared with just mental tasks, the accuracies between two tasks have been significantly improved by appending speech imagery, and the average of accuracies of ten subjects is increased from 76.3% to 82.3%. During the imagery period, the temporal stability of EEG signals is evaluated by Cronbach’s alpha coefficients. The steadiness of signals is different between mental tasks, and EEG signals are more stabilization with speech imagery. The stability of brain activity is conducive to the operation of BCIs. [ABSTRACT FROM AUTHOR]

The perceptual boundaries between speech sounds are malleable and can shift after repeated exposure to contextual information. This shift is known as recalibration. To date, the known inducers of recalibration are lexical (including phonotactic) information, lip-read information and reading. The experiments reported here are a proof-of-effect demonstration that speech imagery can also induce recalibration. [ABSTRACT FROM AUTHOR]

Purpose – The use of imagery in leadership speeches is becoming increasingly important in shaping the beliefs and actions of followers. The purpose of this paper is to investigate the use of speech imagery and linguistic features employed during the 2008 US Presidential Election campaign. Design/methodology/approach – The authors analysed a total of 264 speeches (160 speeches from Obama and 104 speeches from McCain) delivered throughout the 2008 US Presidential Election and identified 15 speech images used by the two candidates. Both descriptive coding and axial coding approaches were applied to the data and speech images common to both candidates were further subjected to Pennebaker et al. (2003) linguistic inquiry methodology. Findings – The analysis revealed a number of important differences with Obama using inclusive language and nurturing communitarian values, whereas McCain focusing on personal actions and strict, conservative individualistic values. The use of more inclusive language by Obama was found to be significant in three of the five speech images common to both candidates. Research limitations/implications – The research acknowledges the difficulty of measuring the effectiveness of speech images without taking into account wider factors such as tone of voice, facial expression and level of conviction. It also recognises the heavy use of speechwriters by presidential candidates whilst on the campaign trail, but argues that candidates still exert a strong influence through instructions to speechwriters and that speeches should reflect the candidate’s values and beliefs. Originality/value – The research findings contribute to the emerging stream of leadership research that addresses language content issues surrounding and embedded in the leadership process. The research argues that leaders’ speeches provide a fertile ground for conducting research and for examining the evolving relationship between leaders and followers. [ABSTRACT FROM AUTHOR]

To what extent is predicting language akin to imagining language? Recently, researchers have argued that covert simulation of the production system underlies both articulation imagery and predicting what somebody is about to say. Moreover, experimental evidence implicates potentially similar production-related mechanisms in prediction during language comprehension and in mental imagery tasks. We discuss evidence in favour of this proposal and argue that imagining others' utterances can also implicate covert simulation. Finally, we briefly review evidence that speakers in jo int language tasks cannot help but mentally represent (i.e„ imagine) whether others are engaging in language production, and that they do so using mechanisms that are also implicated in preparing to speak. [ABSTRACT FROM AUTHOR]

• A novel experimental paradigm based on sequential coding is proposed. • Four imagination tasks are obtained with the sequence combination of motor imagery and speech imagery. • The features of EEG signals are analyzed from the temporal, frequency and spatial domains. • The EEG signals are classified by a multi-classification model based on time sequence. In order to increase the number of instruction sets for the active brain-computer interface (BCI), a novel experimental paradigm based on the sequential coding of motor imagery and speech imagery is proposed in this paper. By dividing one motor imagery and one speech imagery into time series, four imagination tasks are obtained: 1) motor imagery; 2) speech imagery; 3) motor imagery first and then speech imagery; 4) speech imagery first and then motor imagery. After analyzing the temporal, frequency and spatial features of electroencephalography (EEG) signals, four types of signals are classified by a multi-classification model based on time sequence. In this model, feature extraction and classification are accomplished by common spatial pattern (CSP) and support vector machine (SVM), respectively. 12 subjects participate in this experimental paradigm, and their average classification accuracy is 68.94%. The classification results are much higher than random probability, so the proposed experimental paradigm is feasible and valuable. The experimental paradigm based on sequential coding can effectively increase the number of instruction sets of active BCIs, so the practicability of BCIs is also improved. [ABSTRACT FROM AUTHOR]

Objective. Speech imagery (SI) can be used as a reliable, natural, and user-friendly activation task for the development of brain-computer interface (BCI), which empowers individuals with severe disabilities to interact with their environment. The functional near-infrared spectroscopy (fNIRS) is advanced as one of the most suitable brain imaging methods for developing BCI systems owing to its advantages of being non-invasive, portable, insensitive to motion artifacts, and having relatively high spatial resolution. Approach. To improve the classification performance of SI BCI based on fNIRS, a novel paradigm was developed in this work by simplifying the articulation movements in SI to make the articulation movement differences clearer between different words imagery tasks. A SI BCI was proposed to directly answer questions by covertly rehearsing the word '' or '' ('yes' or 'no' in English), and an unconstrained rest task also was contained in this BCI. The articulation movements of SI were simplified by retaining only the movements of the jaw and lips of vowels in Chinese Pinyin for words '' and ''. Main results. Compared with conventional speech imagery, simplifying the articulation movements in SI could generate more different brain activities among different tasks, which led to more differentiable temporal features and significantly higher classification performance. The average 3-class classification accuracies of the proposed paradigm across all 20 participants reached 69.6% and 60.2% which were about 10.8% and 5.6% significantly higher than those of the conventional SI paradigm operated in the 0-10 s and 0-2.5 s time windows, respectively. Significance. These results suggested that simplifying the articulation movements in SI is promising for improving the classification performance of intuitive BCIs based on speech imagery., (© 2023 IOP Publishing Ltd.)

Highlights: [•] In order to complement existing motor-based control paradigms, speech imagery was proposed. [•] Feature vectors of EEG signal were extracted by common spatial patterns (CSP). [•] Feature vectors were classified by support vector machine (SVM). [ABSTRACT FROM AUTHOR]

• A new two-class asynchronous functional near-infrared spectroscopy (fNIRS)-based brain computer interface (BCI) that can detect idle state robustly within a reduced time window in the context of motor imagery of articulation (MIA) is described. • A reduced time window (0–2.5 s) improves the information transfer rate of an fNIRS-based BCI. • Time domain, spatial domain, and connectivity features are analyzed and compared for detecting idle-state in an MIA context. • Compensative information derived from activation, spatial, and information propagation patterns can improve idle-state detection during the performance of MIA. A speech imagery-based brain computer interface (BCI) provides an alternative way for people to interact with the outside world intuitively. Most speech imagery BCIs based on functional near-infrared spectroscopy (fNIRS) are disadvantageous in applications outside the laboratory due to being incapable of detecting asynchronous (self-paced) actions. This work aimed to develop a two-class asynchronous BCI that detects idle state in the context of motor imagery of articulation (MIA). In this study, 19 healthy subjects were asked to rehearse the Chinese vowels/a/and/u/covertly and to accommodate a rest state. A feature selection strategy was designed to combine time domain, spatial domain, and functional connectivity features. All differentiating information was extracted from fNIRS signals in a 0–2.5 s time window. Among single-modality features, the centrality features of brain networks performed better than any other features and yielded a subject-dependent classification accuracy rate of 75.1%. The subject-dependent classification accuracy rate of the combined features at 78.9% was significantly better than that of any single feature. These results demonstrate that it is feasible to detect idle states from active MIA states in a reduced time window size. The proposed combination of information propagation pattern, spatial pattern, and activation pattern was meaningful for extracting discriminative information, hence improving classification accuracy. The high classification accuracy and fast information transfer rate of the presented BCI show promising practicality for real world applications. [ABSTRACT FROM AUTHOR]

Abstract: Inner speech, that little voice that people often hear inside their heads while thinking, is a form of mental imagery. The properties of inner speech errors can be used to investigate the nature of inner speech, just as overt slips are informative about overt speech production. Overt slips tend to create words (lexical bias) and involve similar exchanging phonemes (phonemic similarity effect). We examined these effects in inner and overt speech via a tongue-twister recitation task. While lexical bias was present in both inner and overt speech errors, the phonemic similarity effect was evident only for overt errors, producing a significant overtness by similarity interaction. We propose that inner speech is impoverished at lower (featural) levels, but robust at higher (phonemic) levels. [Copyright &y& Elsevier]

Speech expression is an important high-level cognitive behavior of human beings. The realization of this behavior is closely related to human brain activity. Both true speech expression and speech imagination can activate part of the same brain area. Therefore, speech imagery becomes a new paradigm of brain-computer interaction. Brain-computer interface (BCI) based on speech imagery has the advantages of spontaneous generation, no training, and friendliness to subjects, so it has attracted the attention of many scholars. However, this interactive technology is not mature in the design of experimental paradigms and the choice of imagination materials, and there are many issues that need to be discussed urgently. Therefore, in response to these problems, this article first expounds the neural mechanism of speech imagery. Then, by reviewing the previous BCI research of speech imagery, the mainstream methods and core technologies of experimental paradigm, imagination materials, data processing and so on are systematically analyzed. Finally, the key problems and main challenges that restrict the development of this type of BCI are discussed. And the future development and application perspective of the speech imaginary BCI system are prospected.

The brain–computer interface (BCI) is a promising technology where a user controls a robot or computer by thinking with no movement. There are several underlying principles to implement BCI, such as sensorimotor rhythms, P300, steady-state visually evoked potentials, and directional tuning. Generally, different principles are applied to BCI depending on the application, because strengths and weaknesses vary according to each BCI method. Therefore, BCI should be able to predict a user state to apply suitable principles to the system. This study measured electroencephalography signals in four states (resting, speech imagery, leg-motor imagery, and hand-motor imagery) from 10 healthy subjects. Mutual information from 64 channels was calculated as brain connectivity. We used a convolutional neural network to predict a user state, where brain connectivity was the network input. We applied five-fold cross-validation to evaluate the proposed method. Mean accuracy for user state classification was 88.25 ± 2.34%. This implies that the system can change the BCI principle using brain connectivity. Thus, a BCI user can control various applications according to their intentions. [ABSTRACT FROM AUTHOR]

• Based on motor imagery and speech imagery, an online BCI training system is designed. • The filter range and channels are determined by individuation. • The system is built with anti-interference and robustness. • Online training helps subjects better operate the BCI system. In order to effectively operate brain-computer interfaces (BCI), proper training is required. Currently, the training of BCIs is mainly focused on motor imagery. There is no report of the training combined motor imagery and speech imagery. On the basis of offline experiments, an online BCI training system is designed with motor imagery and speech imagery. The filtering range is analyzed from the offline experimental data. Two most suitable channels are selected by Fisher criterion function for each subject. Power spectral density and sample entropy are combined as the algorithms of feature extraction. The extracted feature vectors are classified by extreme learning machine. As the training progresses, the feature vectors of the updated data are re-extracted and the classifier is retrained, which ensure the adaptability of the system. According to the training results of twelve subjects, the results of six training sessions are gradually improved. The last session yields the best result (83 %). In the sixth session, the results of eight subjects exceed 80 %, and two results achieve and exceed 90 %. With the help of online training, the BCI systems can be better operated by subjects. The classification results of EEG signals are also improved. Therefore, the online training is an effective step for the BCI system based on motor imagery and speech imagery. [ABSTRACT FROM AUTHOR]

• A novel hybrid BCI is proposed: combining motor imagery and speech imagery. • The maximum difference of synchronization is considered as a method to confirm the channel pairs. • The classification results can be improved by combining synchronization and CSP. • A more efficient BCI is designed without long-term training. As a new communication mode between human and computers, brain-computer interfaces (BCIs) based on electroencephalography (EEG) have been widely studied. In order to increase the operational dimension of BCIs, this paper proposes a hybrid BCI based on motor imagery and speech imagery. According to one versus one calculation model, common spatial pattern (CSP) is extended to a three-category algorithm. The energy eigenvalues extracted by CSP are combined with synchronization eigenvalues, which are respectively calculated by cross-correlation function and phase locking value (PLV). The maximum difference of synchronization between two mental tasks is considered as a method to confirm the channel pairs. The experimental results of ten subjects show that: the highest average classification accuracy in three mental tasks is speech imagery (74.3%); imagining left hand movement is followed (71.4%); imagining right hand movement is the last one (69.8%). The classification results can be improved by combining synchronization and CSP, and the synchronization from cross-correlation function is better than PLV. The frequency range and channel pairs both have certain individual differences. By individually adjusting the appropriate settings for each subject, the usage efficiency of BCIs can be improved. [ABSTRACT FROM AUTHOR]