Search

Multimodal emotion recognition in conversational videos (ERC) develops rapidly in recent years. To fully extract the relative context from video clips, most studies build their models on the entire dialogues which make them lack of real-time ERC ability. Different from related researches, a novel multimodal emotion recognition model for conversational videos based on reinforcement learning and domain knowledge (ERLDK) is proposed in this paper. In ERLDK, the reinforcement learning algorithm is introduced to conduct real-time ERC with the occurrence of conversations. The collection of history utterances is composed as an emotion-pair which represents the multimodal context of the following utterance to be recognized. Dueling deep-Q-network (DDQN) based on gated recurrent unit (GRU) layers is designed to learn the correct action from the alternative emotion categories. Domain knowledge is extracted from public dataset based on the former information of emotion-pairs. The extracted domain knowledge is used to revise the results from the RL module and is transformed into other dataset to examine the rationality. The experimental results on datasets show that ERLDK achieves the state-of-the-art results on weighted average and most of the specific emotion categories.

Grant-free non-orthogonal multiple access (NOMA) has recently received wide attention for reducing signaling overhead and transmission latency in massive machine-type communications (mMTC). In grant-free NOMA systems, user activity and data (UAD) has to be detected, which is challenging in practice. As an emerging technique, compressive sensing (CS) shows great promise in solving this problem by exploiting the inherent sparsity nature of user activity. This paper proposes to use the weighted l2,1 minimization (WL21M) to jointly detect UAD in realistic dynamic scenarios. At first, the average recoverability of the WL21M is analyzed. This analysis reveals the fact that the WL21M can improve the detection performance by means of an appropriate weighting and the incorporation of intrinsic temporal correlation. Motivated by the analysis, a collaborative hierarchical match pursuit (C-HiMP) algorithm is proposed for dynamic UAD detection. In the C-HiMP, a sequence of WL21M problems are solved in the subspaces spanned by all of the components in the hierarchical estimated support sets, where the weights are collaboratively updated by the solutions in previous time slots so that an attractive self-correction capacity is obtained. Simulation results demonstrate that the proposed C-HiMP can obtain significant performance improvements, in terms of detection accuracy and computational complexity, compared with several state-of-the-art CS-based detection algorithms.

Stereoelectroencephalography (SEEG) signals can be obtained by implanting deep intracranial electrodes. SEEG depth electrodes can record brain activity from the shallow cortical layer and deep brain structures, which is not achievable through other recording techniques. Moreover, SEEG has the advantage of a high signal-to-noise ratio (SNR). Therefore, it provides a potential way to establish a highly efficient brain-computer interface (BCI) and aid in understanding human brain activity. In this study, we implemented a P300-based BCI using SEEG signals. A single-character oddball paradigm was applied to elicit P300. To predict target characters, we fed the feature vectors extracted from the signals collected by five SEEG contacts into a Bayesian linear discriminant analysis (BLDA) classifier. Thirteen epileptic patients implanted with SEEG electrodes participated in the experiment and achieved an average online spelling accuracy of 93.85%. Moreover, through single-contact decoding analysis and simulated online analysis, we found that the SEEG-based BCI system achieved a high performance even when using a single signal channel. Furthermore, contacts with high decoding accuracies were mainly distributed in the visual ventral pathway, especially the fusiform gyrus (FG) and lingual gyrus (LG), which played an important role in building P300-based SEEG BCIs. These results might provide new insights into P300 mechanistic studies and the corresponding BCIs.

Multimodal sentiment analysis is one of the most attractive interdisciplinary research topics in artificial intelligence (AI). Different from other classification issues, multimodal sentiment analysis of human is a much finer classification problem. However, most current work accept all multimodalities as the input together and then output final results at one time after fusion and decision processes. Rare models try to divide their models into more than one fusion modules with different fusion strategies for better adaption of different tasks. Additionally, most recent multimodal sentiment analysis methods pay great focuses on binary classification, but the accuracy of multi-classification still remains difficult to improve. Inspired by the emotional processing procedure in cognitive science, both binary and multi-classification abilities are improved in our method by dividing the complicated problem into smaller issues which are easier to be handled. In this paper, we propose a Hierarchal Attention-BiLSTM (Bidirectional Long-Short Term Memory) model based on Cognitive Brain limbic system (HALCB). HALCB splits the multimodal sentiment analysis into two modules responsible for two tasks, the binary classification and the multi-classification. The former module divides the input items into two categories by recognizing their polarity and then sends them to the latter module separately. In this module, Hash algorithm is utilized to improve the retrieve accuracy and speed. Correspondingly, the latter module contains a positive sub-net dedicated for positive inputs and a negative sub-nets dedicated for negative inputs. Each of these binary module and two sub-nets in multi-classification module possesses different fusion strategy and decision layer for matching its respective function. We also add a random forest at the final link to collect outputs from all modules and fuse them at the decision-level at last. Experiments are conducted on three datasets and compare the results with baselines on both binary classification and multi-classification tasks. Our experimental results surpass the state-of-the-art multimodal sentiment analysis methods on both binary and multi-classification by a big margin.

Compressive sensing (CS) aims to accurately reconstruct high-dimensional signals from a small number of measurements by exploiting signal sparsity and structural priors. However, signal priors utilized in existing CS reconstruction algorithms rely mainly on hand-crafted design, which often cannot offer the best sparsity-undersampling tradeoff because high-order structural priors of signals are hard to be captured in this manner. In this article, a new recovery guarantee of the unified CS reconstruction model-weighted $\ell _{1}$ minimization (WL1M) is derived, which indicates universal priors could hardly lead to the optimal selection of the weights. Motivated by the analysis, we propose a deep unfolding network for the general WL1M model. The proposed deep unfolding-based WL1M (D-WL1M) integrates universal priors with learning capability so that all of the parameters, including the crucial weights, can be learned from training data. We demonstrate the proposed D-WL1M outperforms several state-of-the-art CS-based methods and deep learning-based methods by a large margin via the experiments on the Caltech-256 image data set.

Achieving high classification performance in electroencephalogram (EEG)-based brain-computer interfaces (BCIs) often entails a large number of channels, which impedes their use in practical applications. Despite the previous efforts, it remains a challenge to determine the optimal subset of channels in a subject-specific manner without heavily compromising the classification performance. In this article, we propose a new method, called spatiotemporal-filtering-based channel selection (STECS), to automatically identify a designated number of discriminative channels by leveraging the spatiotemporal information of the EEG data. In STECS, the channel selection problem is cast under the framework of spatiotemporal filter optimization by incorporating a group sparsity constraints, and a computationally efficient algorithm is developed to solve the optimization problem. The performance of STECS is assessed on three motor imagery EEG datasets. Compared with state-of-the-art spatiotemporal filtering algorithms using full EEG channels, STECS yields comparable classification performance with only half of the channels. Moreover, STECS significantly outperforms the existing channel selection methods. These results suggest that this algorithm holds promise for simplifying BCI setups and facilitating practical utility.

Fusion of multimodal features is a momentous problem for video emotion recognition. As the development of deep learning, directly fusing feature matrixes of each mode through neural networks at feature level becomes mainstream method. However, unlike unimodal issues, for multimodal analysis, finding the correlations between different modal is as important as discovering effective unimodal features. To make up the deficiency in unearthing the intrinsic relationships between multimodal, a novel modularized multimodal emotion recognition model based on deep canonical correlation analysis (MERDCCA) is proposed in this letter. In MERDCCA, four utterances are gathered as a new group and each utterance contains text, audio and visual information as multimodal input. Gated recurrent unit layers are used to extract the unimodal features. Deep canonical correlation analysis based on encoder-decoder network is designed to extract cross-modal correlations by maximizing the relevance between multimodal. The experiments on two public datasets show that MERDCCA achieves the better results.

A brain-computer interface (BCI) measures and analyzes brain activity and converts this activity into computer commands to control external devices. In contrast to traditional BCIs that require a subject-specific calibration process before being operated, a subject-independent BCI learns a subject-independent model and eliminates subject-specific calibration for new users. However, building subject-independent BCIs remains difficult because electroencephalography (EEG) is highly noisy and varies by subject. In this study, we propose an invariant pattern learning method based on a convolutional neural network (CNN) and big EEG data for subject-independent P300 BCIs. The CNN was trained using EEG data from a large number of subjects, allowing it to extract subject-independent features and make predictions for new users. We collected EEG data from 200 subjects in a P300-based spelling task using two different types of amplifiers. The offline analysis showed that almost all subjects obtained significant cross-subject and cross-amplifier effects, with an average accuracy of more than 80%. Furthermore, more than half of the subjects achieved accuracies above 85%. These results indicated that our method was effective for building a subject-independent P300 BCI, with which more than 50% of users could achieve high accuracies without subject-specific calibration.

Spectral–spatial Gabor filtering, which is based on 3-D local harmonic analysis, has been a powerful spectral–spatial feature extraction tool for hyperspectral image (HSI) classification. However, existing spectral–spatial Gabor approaches are prone to oversmoothing, neglecting the existences of edges and negatively affecting the classification. In this article, we propose a new HSI Gabor filtering concept, called spectral–spatial-range Gabor filtering, which intends to restrain edge interference from disturbing local spectral–spatial harmonic components. Contributions and novelties of our work can be identified as follows: 1) an HSI filtering framework is created, which can accommodate various Gabor filtering procedures and hence offer the potential to guide the design of new Gabor filters; 2) following such a unified filtering framework and taking into consideration both local spectral–spatial harmonic characteristics and range domain variations, we develop a new concept of spectral–spatial-range Gabor filtering; and 3) utilizing this proposed Gabor prototype and elaborating mathematical derivations, we achieve a novel discriminative spectral–spatial-range Gabor filtering method, which can deal with discriminative local harmonics and edge interference simultaneously along the spectral–spatial-range domain, obtaining highly discriminative Gabor features while yielding linear computational complexity. Our novel method is evaluated on four real HSI data sets and achieves excellent performances.

Estimating the extents and localizations of extended sources from noninvasive EEG/MEG signals is challenging. In this paper, we have proposed a fully data driven source imaging method, namely Variation Sparse Source Imaging based on Automatic Relevance Determination (VSSI-ARD), to reconstruct extended cortical activities. VSSI-ARD explores the sparseness of current sources on the variation domain by employing ARD prior under empirical Bayesian framework. With convex analysis, the sources are efficiently obtained by solving a series of reweighting L21-norm regularization problems with ADMM. By virtue of the iterative reweighting process and sparse signal processing techniques, VSSI-ARD gets rid of the small amplitude dipoles that are more probably outside the extent of underlying sources. With the sparsity enforced on the edges using ARD prior, the estimations show clear boundaries between active and background regions without subjective thresholds. Validation with both simulated and human experimental data indicates that VSSI-ARD not only estimates the localizations of sources, but also provides relatively useful and accurate information about the extents of cortical activities.

Motion capture is a cross-cutting application field developed in recent years, which comprises electronics, communications, control, computer graphics, ergonomics, navigation, and other disciplines. The accurate application of basketball technical movements in the basketball game is very important. Therefore, it is of great significance to capture and standardize athletes’ movements and improve their training. Unfortunately, there are numerous issues in traditional classroom teaching that largely helps to train the athletes. To solve the issues of traditional basketball classroom teaching, a virtual simulation system for students’ sports training is designed in this paper. Firstly, the information of basketball dribbling movement is captured and simulated in three dimensions. Secondly, we compare it with the standard database to judge the irregularities of athletes’ movements, and carry out digital processing on athletes’ movements and skill improvements statistics in combination with system functions. Thirdly, we set up a gradual training cycle. Finally, the Kinect-based capture technology is adopted to obtain the activity information of different joints of the human body. Through processing the motion data, relevant motion analysis data are fed to the established motion model, to realize the comparative analysis of motion pictures. In our experiments, we observed better training of the physical education.

Instance segmentation is a challenging computer vision problem which lies at the intersection of object detection and semantic segmentation. Motivated by plant image analysis in the context of plant phenotyping, a recently emerging application field of computer vision, this paper presents the Exemplar-Based Recursive Instance Segmentation (ERIS) framework. A three-layer probabilistic model is firstly introduced to jointly represent hypotheses, voting elements, instance labels and their connections. Afterwards, a recursive optimization algorithm is developed to infer the maximum a posteriori (MAP) solution, which handles one instance at a time by alternating among the three steps of detection, segmentation and update. The proposed ERIS framework departs from previous works mainly in two respects. First, it is exemplar-based and model-free, which can achieve instance-level segmentation of a specific object class given only a handful of (typically less than 10) annotated exemplars. Such a merit enables its use in case that no massive manually-labeled data is available for training strong classification models, as required by most existing methods. Second, instead of attempting to infer the solution in a single shot, which suffers from extremely high computational complexity, our recursive optimization strategy allows for reasonably efficient MAP-inference in full hypothesis space. The ERIS framework is substantialized for the specific application of plant leaf segmentation in this work. Experiments are conducted on public benchmarks to demonstrate the superiority of our method in both effectiveness and efficiency in comparison with the state-of-the-art.

To enhance the performance of the brain-actuated robot system, a novel shared controller based on Bayesian approach is proposed for intelligently combining robot automatic control and brain-actuated control, which takes into account the uncertainty of robot perception, action and human control. Based on maximum a posteriori probability (MAP), this method establishes the probabilistic models of human and robot control commands to realize the optimal control of a brain-actuated shared control system. Application on an intelligent Bayesian shared control system based on steady-state visual evoked potential (SSVEP)-based brain machine interface (BMI) is presented for all-time continuous wheelchair navigation task. Moreover, to obtain more accurate brain control commands for shared controller and adapt the proposed system to the uncertainty of electroencephalogram (EEG), a hierarchical brain control mechanism with feedback rule is designed. Experiments have been conducted to verify the proposed system in several scenarios. Eleven subjects participated in our experiments and the results illustrate the effectiveness of the proposed method.

Segmentation of cardiac ventricle from magnetic resonance images is significant for cardiac disease diagnosis, progression assessment, and monitoring cardiac conditions. Manual segmentation is so time consuming, tedious, and subjective that automated segmentation methods are highly desired in practice. However, conventional segmentation methods performed poorly in cardiac ventricle, especially in the right ventricle. Compared with the left ventricle, whose shape is a simple thick-walled circle, the structure of the right ventricle is more complex due to ambiguous boundary, irregular cavity, and variable crescent shape. Hence, effective feature extractors and segmentation models are preferred. In this paper, we propose a dilated-inception net (DIN) to extract and aggregate multi-scale features for right ventricle segmentation. The DIN outperforms many state-of-the-art models on the benchmark database of right ventricle segmentation challenge. In addition, the experimental results indicate that the proposed model has potential to reach expert-level performance in right ventricular epicardium segmentation. More importantly, DIN behaves similarly to clinical expert with high correlation coefficients in four clinical cardiac indices. Therefore, the proposed DIN is promising for automated cardiac right ventricle segmentation in clinical applications.

Brain tissue segmentation from Magnetic resonance (MR) image is significant for assessing both neurologic conditions and brain disease. Manual brain tissue segmentation is time-consuming, tedious and subjective which indicates a need for more efficiently automated approaches. However, due to ambiguous boundaries, anatomically complex structure and individual differences, conventional automated segmentation methods performed poorly. Therefore, more effective feature extraction techniques and advanced segmentation models are in essential demand. Inspired by deep learning concepts, we propose a multi-modality aggregation network (MMAN), which is able to extract multi-scale features of brain tissues and harness complementary information from multi-modality MR images for fast and accurate segmentation. Extensive experiments on the well-known MRBrainS Challenge database corroborate the efficiency of the proposed model. Within approximately thirteen seconds, the MMAN can segment three different brain tissues from MRI data of each individual, that is faster than many existing methods. For the segmentation of gray matter, white matter, and cerebrospinal fluid, the MMAN achieved dice coefficients of 86.40%, 89.70% and 84.86%, respectively. Consequently, the proposed model outperformed many state-of-the-art methods and got the second place in the MRBrainS Challenge. Therefore, the proposed MMAN is promising for automated brain segmentation in clinical applications.

Deep reinforcement learning (DRL) algorithms with experience replays have been used to solve many sequential learning problems. However, in practice, DRL algorithms still suffer from the data inefficiency problem, which limits their applicability in many scenarios, and renders them inefficient in solving real-world problems. To improve the data efficiency of DRL, in this paper, a new multi-step method is proposed. Unlike traditional algorithms, the proposed method uses a new return function, which alters the discount of future rewards while decreasing the impact of the immediate reward when selecting the current state action. This approach has the potential to improve the efficiency of reward data. By combining the proposed method with classic DRL algorithms, deep Q-networks (DQN) and double deep Q-networks (DDQN), two novel algorithms are proposed for improving the efficiency of learning from experience replay. The performance of the proposed algorithms, expected n-step DQN (EnDQN) and expected n-step DDQN (EnDDQN), are validated using two simulation environments, CartPole and DeepTraffic. The experimental results demonstrate that the proposed multi-step methods greatly improve the data efficiency of DRL agents while further improving the performance of existing classic DRL algorithms when incorporated into their training.

Event-related potential (ERP) is bioelectrical activity that occurs in the brain in response to specific events or stimuli, reflecting the electrophysiological changes in the brain during cognitive processes. ERP is important in cognitive neuroscience and has been applied to brain-computer interfaces (BCIs). However, because ERP signals collected on the scalp are weak, mixed with spontaneous electroencephalogram (EEG) signals, and their temporal and spatial features are complex, accurate ERP detection is challenging. Compared to traditional neural networks, the capsule network (CapsNet) replaces scalar-output neurons with vector-output capsules, allowing the various input information to be well preserved in the capsules. In this study, we expect to utilize CapsNet to extract the discriminative spatial-temporal features of ERP and encode them in capsules to reduce the loss of valuable information, thereby improving the ERP detection performance for BCI. Therefore, we propose ERP-CapsNet to perform ERP detection in a BCI speller application. The experimental results on BCI Competition datasets and the Akimpech dataset show that ERP-CapsNet achieves better classification performances than do the state-of-the-art techniques. We also use a decoder to investigate the attributes of ERPs encoded in capsules. The results show that ERP-CapsNet relies on the P300 and P100 components to detect ERP. Therefore, ERP-CapsNet not only acts as an outstanding method for ERP detection, but also provides useful insights into the ERP detection mechanism.

Individual differences of electroencephalogram (EEG) signals can increase calibration difficulty, which is a major challenge in the practical application of brain computer interface (BCI). Transfer learning is an available method to predict the target subject’s EEG signals by learning an effective model from other subjects’ signals. This paper proposes a weight conditional distribution adaptation (WCDA) method, which can enhance feature transferability and discriminability by minimizing the conditional distribution of the same class between domains while maximizing the conditional distribution of different classes between domains. Moreover, a transferable source sample selection (TSSS) method is proposed to improve the transfer learning performance and reduce the computational cost. Experiments on two public motor imagery (MI) datasets demonstrated our approach outperforms the state of the art methods, thus providing an available way to reduce calibration effort for BCI applications.

Achieving high classification performance is challenging due to non-stationarity and low signal-to-noise ratio (low SNR) characteristics of EEG signals. Spatial filtering is commonly used to improve the SNR yet the individual differences in the underlying temporal or frequency information is often ignored. This paper investigates motor imagery signals via orthogonal wavelet decomposition, by which the raw signals are decomposed into multiple unrelated sub-band components. Furthermore, channel-wise spectral filtering via weighting the sub-band components are implemented jointly with spatial filtering to improve the discriminability of EEG signals, with an l2-norm regularization term embedded in the objective function to address the underlying over-fitting issue. Finally, sparse Bayesian learning with Gaussian prior is applied to the extracted power features, yielding an RVM classifier. The classification performance of SEOWADE is significantly better than those of several competing algorithms (CSP, FBCSP, CSSP, CSSSP, and shallow ConvNet). Moreover, scalp weight maps of the spatial filters optimized by SEOWADE are more neurophysiologically meaningful. In summary, these results demonstrate the effectiveness of SEOWADE in extracting relevant spatio-temporal information for single-trial EEG classification.

Autonomous vehicles (AVs) are expected to greatly redefine the future of transportation. However, before people fully realize the benefits of autonomous vehicles, there are still major engineering challenges to be solved. One of the challenges is to build models that reliably predict the movement of the vehicle and its surrounding objects. In this paper, we proposed our ML policy to fully control a Self Driving Vehicle (SDV). The policy is a CNN architecture based on ResNet50 which is invoked by the SDV to obtain the next command to execute. In each step, we predict several different trajectories and their probabilities to assist us in decision-making. Compared with VGG16 and ResNet34, the simulation results demonstrate that our model based on ResN et50 improves the performance by 2.23% and 22.5%, respectively. It also shows that ResNet achieves better performance than VGG in the aspect of motion prediction. What's more, increasing the depth of the network can further improve the performance of the network.

Cancer is a disease that has not been completely overcome in the modern medical field, and skin cancer is one of them. Melanoma in skin cancer is highly lethal. Once the disease is not found in time, it is undoubtedly a disaster for patients. Therefore, early visual detection of melanoma has become a growing concern for researchers. We proposed an online model fusion method based on VGG and DenseNet, which can effectively improve the generalization and performance of the single model. In addition, we analyzed the relationship between the impact of melanoma on different regions of human body, and used data augmentation to effectively augment the dataset to further improve the performance. The experimental results indicate that the AUC-ROC score of data augmentation and online model fusion method on ISIC 2020 challenge dataset is 93.0%, which is higher than that of the single model VGG16 or DenseNet201.

In recent years, deep neural networks have achieved great success in many fields, such as computer vision and natural language processing. Traditional image recommendation algorithms use text-based recommendation methods. The process of displaying images requires a lot of time and labor, and the time-consuming labor is inefficient. Therefore, this article mainly studies image recommendation algorithms based on deep neural networks in social networks. First, according to the time stamp information of the dataset, the interaction records of each user are sorted by the closest time. Then, some feature vectors are created via traditional feature algorithms like LBP, BGC3, RTU, or CNN extraction. For image recommendation, two LSTM neural networks are established, which accept these feature vectors as input, respectively. The compressed output of the two sub-ESTM neural networks is used as the input of another LSTM neural network. The multilayer regression algorithm is adopted to randomly sample some network nodes to obtain the cognitive information of the nodes sampled in the entire network, predict the relationship between all nodes in the network based on the cognitive information, and perform low sampling to achieve relationship prediction. The experiments show that proposed LSTM model together with CNN feature vectors can outperform other algorithms.

Motor imagery (MI) decoding is an important part of brain-computer interface (BCI) research, which translates the subject’s intentions into commands that external devices can execute. The traditional methods for discriminative feature extraction, such as common spatial pattern (CSP) and filter bank common spatial pattern (FBCSP), have only focused on the energy features of the electroencephalography (EEG) and thus ignored the further exploration of temporal information. However, the temporal information of spatially filtered EEG may be critical to the performance improvement of MI decoding. In this paper, we proposed a deep learning approach termed filter-bank spatial filtering and temporal-spatial convolutional neural network (FBSF-TSCNN) for MI decoding, where the FBSF block transforms the raw EEG signals into an appropriate intermediate EEG presentation, and then the TSCNN block decodes the intermediate EEG signals. Moreover, a novel stage-wise training strategy is proposed to mitigate the difficult optimization problem of the TSCNN block in the case of insufficient training samples. Firstly, the feature extraction layers are trained by optimization of the triplet loss. Then, the classification layers are trained by optimization of the cross-entropy loss. Finally, the entire network (TSCNN) is fine-tuned by the back-propagation (BP) algorithm. Experimental evaluations on the BCI IV 2a and SMR-BCI datasets reveal that the proposed stage-wise training strategy yields significant performance improvement compared with the conventional end-to-end training strategy, and the proposed approach is comparable with the state-of-the-art method.

Sleep stage classification is a technique for analyzing sleep quality. Manual sleep staging is time-consuming and laborious. In this paper, we propose an automatic sleep stage classification model combining feature learning and sequence learning, which extract features with convolutional neural network(CNN) and learn the sequence transition rule through multi-layer long short term memory(LSTM) architecture with attention mechanism. In addition, we also noticed that most of the misclassified samples locate in transition period. Therefore, multi-label classification scheme is introduced to provide more label information, so as to improve the classification performance of transition period. We evaluate on two public datasets (Sleep EDF Expanded and Physionet2018), where our framework reaches macro F1-score of 79.7 and 79.8, respectively. The proposed network achieves the state-of-the-art classification performance on Sleep EDF Expanded dataset and sets new benchmark on Physionet2018 dataset.

Epilepsy is a common brain disease that has serious negative effects on patients. Electroencephalogram (EEG) is widely used for detecting epileptic signal. Because epilepsy EEG data set is generally small, in this paper, we use a shallow convolutional neural network (CNN) to classify the raw data. To improve the performance of the model, we propose a layerwise pre-training mechanism. In our experiment, we validate the effectiveness of the method on the public epilepsy EEG data set provided by University of Bonn.

This paper presents a new asynchronous hybrid brain-computer interface (BCI) system that integrates a speller, a web browser, an e-mail client, and a file explorer using electroencephalographic (EEG) and electrooculography (EOG) signals. More specifically, an EOG-based button selection method, which requires the user to blink his/her eyes synchronously with the target button's flashes during button selection, is first presented. Next, we propose a mouse control method by combining EEG and EOG signals, in which the left-/right-hand motor imagery (MI)-related EEG is used to control the horizontal movement of the mouse and the blink-related EOG is used to control the vertical movement of the mouse and to select/reject a target. These two methods are further combined to develop the integrated hybrid BCI system. With the hybrid BCI, users can input text, access the internet, communicate with others via e-mail, and manage files in their computer using only EEG and EOG without any body movements. Ten healthy subjects participated in a comprehensive online experiment, and superior performance was achieved compared with our previously developed P300- and MI-based BCI and some other asynchronous BCIs, therefore demonstrating the system's effectiveness.

Reinforcement learning with appropriately designed reward signal could be used to solve many sequential learning problems. However, in practice, the reinforcement learning algorithms could be broken in unexpected, counterintuitive ways. One of the failure modes is reward hacking which usually happens when a reward function makes the agent obtain high return in an unexpected way. This unexpected way may subvert the designer’s intentions and lead to accidents during training. In this paper, a new multi-step state-action value algorithm is proposed to solve the problem of reward hacking. Unlike traditional algorithms, the proposed method uses a new return function, which alters the discount of future rewards and no longer stresses the immediate reward as the main influence when selecting the current state action. The performance of the proposed method is evaluated on two games, Mappy and Mountain Car. The empirical results demonstrate that the proposed method can alleviate the negative impact of reward hacking and greatly improve the performance of reinforcement learning algorithm. Moreover, the results illustrate that the proposed method could also be applied to the continuous state space problem successfully.

Object: For the shared control systems, how to trade off the control weight between robot autonomy and human operator is an important issue, especially for BCI-based systems. However, most of existing shared controllers have paid less attention to the effects caused by subjects with different levels of brain control ability. Approach: In this paper, a brain state evaluation network, termed BSE-NET, is proposed to evaluate subjects' brain control ability online based on quantized attention-gated kernel reinforcement learning (QAGKRL). With the output of BSE-NET (confidence score), a shared controller is designed to dynamically adjust the control weight between robot autonomy and human operator. Main results: The experimental results show that most of subjects achieved high and stable experimental success rate of approximately 90%. Furthermore, for subjects with different accuracy on EEG decoding, a proper confidence score can be dynamically generated to reflect their levels of brain control ability, and the proposed system can effectively adjust the control weight in all-time shared control. Significance: We discuss how our proposed method shows promise for BCI applications that can evaluate subjects' brain control ability online as well as provide a method for the research on self-adaptive shared control to adaptively balance control weight between subject's instruction and robot autonomy.

Objective: This paper presents an asyn-chronous electrooculography (EOG)-based human–machine interface (HMI) for smart home environmental control with the purpose of providing daily assistance for severe spinal cord injury (SCI) patients. Methods: The proposed HMI allows users to interact with a smart home environment through eye blinking. Specifically, several buttons, each corresponding to a control command, randomly flash on a graphical user interface. Each flash of the buttons functions as a visual cue for the user to blink. To issue a control command, the user can blink synchronously with the flashes of the corresponding button. Through detecting blinks based on the recorded EOG signal, the target button and its corresponding control command are determined. Seven SCI patients participated in an online experiment, during which the patients were required to control a smart home environment including household electrical appliances, an intelligent wheelchair, as well as a nursing bed via the proposed HMI. Results: The average false operation ratio in the control state was 4.1%, whereas during the idle state, no false operations occurred. Conclusion: All SCI patients were able to control the smart home environment using the proposed EOG-based HMI with satisfactory performance in terms of the false operation ratio in both the control and the idle states. Significance: The proposed HMI offers a simple and effective approach for patients with severe SCIs to control a smart home environment. Therefore, it is promising to assist severe SCI patients in their daily lives.

Weighted ℓ1 minimization (WL1M) is a general and powerful framework for reconstructing sparse signals from underdetermined measurements. The performance improvement of WL1M owes to the incorporation of additional structural priors of signals by means of its weights. However, the selection of weights relies on hand-crafted designs in existing works, so that high-order structural priors of signals are hard to be captured. This paper proposes a data-driven method, namely RBM-WL1M, to alleviate this situation. In the RBM-WL1M, restricted Boltzmann machines (RBMs) are employed to learn the prior distribution of the signals from training data; furthermore, utilizing the RBM, high frequency support set and non-zero probabilities for each of the entries in signals can be estimated effectively, which are used to appropriately select the weights. In our experiments, the proposed framework demonstrates superior performance over several state-of-the-art CS methods on the Physikalisch-Technische Bundesanstalt(PTB) Diagnostic ECG Data set.

Few-shot instance segmentation (FSIS) conjoins the few-shot learning paradigm with general instance segmentation, which provides a possible way of tackling instance segmentation in the lack of abundant labeled data for training. This paper presents a Fully Guided Network (FGN) for few-shot instance segmentation. FGN perceives FSIS as a guided model where a so-called support set is encoded and utilized to guide the predictions of a base instance segmentation network (i.e., Mask R-CNN), critical to which is the guidance mechanism. In this view, FGN introduces different guidance mechanisms into the various key components in Mask R-CNN, including Attention-Guided RPN, Relation-Guided Detector, and Attention-Guided FCN, in order to make full use of the guidance effect from the support set and adapt better to the inter-class generalization. Experiments on public datasets demonstrate that our proposed FGN can outperform the state-of-the-art methods., Accepted by CVPR 2020, 10 pages, 6 figures

To solve online continuous time-varying convex quadratic-programming problems constrained by a time-varying linear-equality, a novel varying-parameter convergent-differential neural network (termed as VP-CDNN) is proposed and analyzed. Different from fixed-parameter convergent-differential neural network (FP-CDNN), such as the gradient-based recurrent neural network, the classic Zhang neural network (ZNN), and the finite-time ZNN (FT-ZNN), VP-CDNN is based on monotonically increasing time-varying design-parameters. Theoretical analysis proves that VP-CDNN has super exponential convergence and the residual errors of VP-CDNN converge to zero even under perturbation situations, which are both better than traditional FP-CDNN and FT-ZNN. Computer simulations based on different activation functions are illustrated to verify the super exponential convergence performance and strong robustness characteristics of the proposed VP-CDNN. A robot tracking example is finally presented to verify the effectiveness and availability of the proposed VP-CDNN.

Electromagnetic (E/MEG) brain source imaging involves challenging problems that make it particularly difficult to estimate both the locations and extents of extended sources. In this study, we propose a new method called Variation Sparse Source Imaging based on Conditional Mean of the posterior (VSSI-CM), which is built upon a Bayesian framework, to reconstruct extended E/MEG generators. Based on the proposed framework, VSSI-CM can employ various spatial priors (e.g., the Laplace prior) to explore sparseness of current sources in transform domains (e.g., the variation transform in this study). Considering the complexity of posterior density in the estimated sources, we propose using the posterior mean instead of the typical maximum a posterior (MAP) estimate as a more accurate inverse solution. The posterior mean is obtained by fitting an approximated Gaussian distribution to the intractable true posterior distribution. An efficient double-loop algorithm is also proposed using convex analysis skills. Validation using synthetic and human experimental data sets indicates that VSSI-CM outperforms the well-studied L2-norm methods (i.e., sLORETA and dSPM) and the sparse constrained methods that explore sparseness in the original source domain. The estimates from VSSI-CM are also more accurate than that from MAP.

The emerging compressive sensing (CS) paradigm holds considerable promise for improving the energy efficiency of wireless body sensor networks, which enables nodes to employ a sample rate significantly below Nyquist while still able to accurately reconstruct signals. In this paper, we propose a weighted $\ell _{1,2}$ minimization method for multichannel electrocardiogram (ECG) reconstruction by exploiting both the interchannel correlation and multisource prior in wavelet domain. A sufficient and necessary condition for exact recovery via the proposed method is derived. Based upon the condition, the performance gain of the proposed method is analyzed theoretically. Furthermore, a reconstruction error bound of the proposed method is obtained, which indicates that the proposed method is stable and robust in recovering sparse and compressible signals from noisy measurements. Extensive experiments utilizing Physikalisch-Technische Bundesanstalt diagnostic ECG database and open-source electrophysiological toolbox fetal ECG database show that significant performance improvements, in terms of compression rate and reconstruction quality, can be obtained by the proposed method compared with the state-of-the-art CS-based methods.

Goal : P300 spellers are among the most popular types of brain–computer interfaces (BCIs) and are extremely useful assistive devices that enable severely disabled patients to communicate. However, P300 speller performances should be further improved to translate laboratory designs into practical applications. We aimed to design a new speller paradigm that could evoke higher event-related potentials (ERPs) than traditional P300 spellers, thus improving the performance of BCI systems. Methods : We proposed a new P300 speller paradigm based on three-dimensional (3-D) stereo visual stimuli. In this paradigm, flashing buttons are presented in 3-D stereo form. We designed two experiments, one that tested a traditional two-dimensional (2-D) speller and another that tested the proposed 3-D speller. Twelve healthy volunteers participated in our experiments. We compared the ERPs elicited by the 2-D speller and the 3-D speller, and we also compared the classification accuracy, information transfer rate (ITR), and user workload between the two paradigms. Results : The 3-D P300 speller elicited higher amplitudes of P300 waveforms than the traditional 2-D P300 speller. The online experimental results showed that the classification accuracy and the ITR were significantly improved with the 3-D P300 speller. We also found that the user workload of the 3-D P300 speller was significantly lower than that of the 2-D P300 speller. Conclusion : The proposed 3-D P300 speller based on stereo visual stimuli outperformed a traditional 2-D P300 speller. This finding indicates that our 3-D paradigm offers a new method that will improve the performance of P300 BCI systems.

In order to solve the joint-angular-drift problems of redundant robot manipulators, a novel varying-parameter convergent-differential neural network (VP-CDNN) is proposed and exploited. To do so, a quadratic program (QP)-based feedback-considered joint-angular-drift-free (FC-JADF) scheme is first designed and presented. The FC-JADF scheme adopted in this paper is composed of an optimization criterion simultaneously optimizing quadratic and linear terms, and a velocity layer kinematic equation with adding feedback. Second, the FC-JADF scheme is formulated as a standard QP. Third, the VP-CDNN is proposed to solve the resultant standard QP problem. The Lyapunov theory proves that the proposed VP-CDNN solver can globally converge to an optimal solution to the standard QP problem corresponding to redundant robot manipulators, and the joint-angular-drift problems are solved. Two computer simulations and physical experiments based on a six-degree-of-freedom Kinova Jaco $^2$ robot, i.e., a starfish path and a cardioid path, verify the effectiveness, accuracy, safety, and practicability of the QP-based FC-JADF scheme and the VP-CDNN solver for solving the joint-angular-drift problems of redundant robot manipulators.

In off-line training of motor imagery-based brain-computer interfaces (BCIs), to enhance the generalization performance of the learned classifier, the local information contained in test data could be used to improve the performance of motor imagery as well. Further considering that the covariance matrices of electroencephalogram (EEG) signal lie on Riemannian manifold, in this paper, we construct a Riemannian graph to incorporate the information of training and test data into processing. The adjacency and weight in Riemannian graph are determined by the geodesic distance of Riemannian manifold. Then, a new graph embedding algorithm, called bilinear regularized locality preserving (BRLP), is derived upon the Riemannian graph for addressing the problems of high dimensionality frequently arising in BCIs. With a proposed regularization term encoding prior information of EEG channels, the BRLP could obtain more robust performance. Finally, an efficient classification algorithm based on extreme learning machine is proposed to perform on the tangent space of learned embedding. Experimental evaluations on the BCI competition and in-house data sets reveal that the proposed algorithms could obtain significantly higher performance than many competition algorithms after using same filter process.

Biological signals, including electroencephalography (EEG) and electrooculography (EOG), are often used to develop switches, which represent a class of typical asynchronous human–computer interfaces (HCIs) in which control and idle states need to be distinguished based on a criterion. Determining a satisfactory criterion for rapid and accurate discrimination between control and idle states remains a challenging issue, as EEG signals are highly noisy and nonstationary, and EOG signals are highly affected by unintended/spontaneous eye movements. Therefore, most existing EEG- or EOG-based switches are characterized by disadvantages of long response times (RTs) or high false positive rates (FPRs). The primary contribution of this work is the development of a novel EOG-based switch design, in which a visual trigger mechanism is introduced to guide the users’ blinks and to assist in detecting blinks. Specifically, the graphical user interface (GUI) includes a switch button that flashes once per 1.2 s. The user is instructed to blink synchronously with the flashes of the switch button to issue an on/off command while a single-channel EOG signal is collected. A waveform detection algorithm is applied to the ongoing EOG signal, which discriminates the intended and unintended blinks mainly based on the synchrony between the blink and the switch buttons flash. Once an intended blink, i.e., a blink corresponding to a button’s flash, is detected, the system issues an on/off command. As one application, the proposed EOG-based switch is used to produce start/stop commands for a wheelchair. Several online experiments were conducted with ten healthy subjects. An average accuracy of 99.5%, an RT of 1.3 s for issuing a switch command in the control state, and an average FPR of 0.10/min in the idle state were achieved. The experimental results therefore demonstrate the effectiveness of the single-channel EOG-based switch.

Epileptic seizures are caused by excessive, synchronized activity of large groups of neurons. In human electroencephalograph (EEG), they are reflected by multiple epileptic characteristic waves. Based on the idea of template matching, this paper presents a patient-specific approach for the automatic detection of epileptic seizures. In our method, a set of wavelets are constructed based on the epileptic characteristic waves extracted from training EEG signals, and then continuous wavelet transform (CWT) is performed on the recorded EEG. The coefficients of CWT reflect the similarity of the recorded EEG and the epileptic characteristic waveforms and thus can be used to detect if the epileptic characteristic waveforms exist in the EEG. After applying data fusion to the CWT coefficient matrices corresponding to the multiple constructed wavelets, the boundaries of seizures can be determined. In the experiment, our constructed wavelets performed better in the detection of epileptic characteristic waves compared to the Daubechies wavelet. We analyzed the EEG of 10 patients and our method detected 32 out of 34 seizures and declared five false detections. Therefore, our method is promising for the automatic detection of epileptic seizures and the real-time monitoring of patients’ EEG signal.

Semi-supervised learning has been applied in brain–computer interfaces (BCIs) to reduce calibration time for user. For example, a sequential updated self-training least squares support vector machine (SUST-LSSVM) was devised for online semi-supervised P300 speller. Despite its good performance, the computational complexity becomes too high after several updates, which hinders its practical online application. In this paper, we present a self-training regularized weighted online sequential extreme learning machine (ST-RWOS-ELM) for P300 speller. It achieves much lower complexity compared to SUST-LSSVM without affecting the spelling accuracy performance. The experimental results validate its effectiveness in the P300 system.

Multivariate pattern analysis (MVPA) approach applied to neuroimaging data, such as functional magnetic resonance imaging (fMRI) data, has received a great deal of attention because of its sensitivity to distinguishing patterns of neural activities associated with different stimuli or cognitive states. Generally, when using MVPA approach to decode the mental states or stimuli, a set of discriminative variables (e.g., voxels) is first selected. However, in most of existing MVPA methods, the selected variables do not contain all informative variables, since these selected variables are sufficient for decoding. In this paper, we propose a multivariate pattern analysis method based on sparse representation for decoding the brain states and localizing category-specific brain activation areas corresponding to two experimental conditions/tasks at the same time. Unlike traditional MVPA approaches, this method is designed to find informative variables as many as possible. We applied the proposed method to two judgement experiments: a gender discrimination and an emotion discrimination task, data analysis results demonstrate its effectiveness and potential applications.

Electrooculography (EOG) signals, which can be used to infer the intentions of a user based on eye movements, are widely used in human-computer interface (HCI) systems. Most existing EOG-based HCI systems incorporate a limited number of commands because they generally associate different commands with a few different types of eye movements, such as looking up, down, left, or right. This paper presents a novel single-channel EOG-based HCI that allows users to spell asynchronously by only blinking. Forty buttons corresponding to 40 characters displayed to the user via a graphical user interface are intensified in a random order. To select a button, the user must blink his/her eyes in synchrony as the target button is flashed. Two data processing procedures, specifically support vector machine (SVM) classification and waveform detection, are combined to detect eye blinks. During detection, we simultaneously feed the feature vectors extracted from the ongoing EOG signal into the SVM classification and waveform detection modules. Decisions are made based on the results of the SVM classification and waveform detection. Three online experiments were conducted with eight healthy subjects. We achieved an average accuracy of 94.4% and a response time of 4.14 s for selecting a character in synchronous mode, as well as an average accuracy of 93.43% and a false positive rate of 0.03/min in the idle state in asynchronous mode. The experimental results, therefore, demonstrated the effectiveness of this single-channel EOG-based speller.

Three neural networks and three numerical methods are investigated, developed, and compared to solve a repetitive motion planning (RMP) scheme for remedying joint-drift problems of redundant robot manipulators. Three recurrent neural networks, i.e., a dual neural network, a linear variational inequality (LVI)-based primal-dual neural network, and a simplified LVI-based primal-dual neural network, are recurrent and real time, and they do not need to be trained in advance. Three numerical methods, i.e., the 94LVI method, the E47 method, and the M4 method, are time discrete and ready to conduct in digital computers. All these solutions have global linear convergence. Computer simulations and physical robot experiments verify that they are all effective to solve the RMP scheme. The comparisons show that neural networks are more accurate and faster than numerical methods on the same simulated condition under the majority normal circumstances. Furthermore, numerical methods are easy to be applied in digital computers since they are time discrete.

Objective: A challenging task for an electroencephalography (EEG)-based asynchronous brain-computer interface (BCI) is to effectively distinguish between the idle state and the control state while maintaining a short response time and a high accuracy when commands are issued in the control state. This study proposes a novel hybrid asynchronous BCI system based on a combination of steady-state visual evoked potentials (SSVEPs) in the EEG signal and blink-related electrooculography (EOG) signals. Methods: Twelve buttons corresponding to 12 characters are included in the graphical user interface (GUI). These buttons flicker at different fixed frequencies and phases to evoke SSVEPs and are simultaneously highlighted by changing their sizes. The user can select a character by focusing on its frequency-phase stimulus and simultaneously blinking his/her eyes in accordance with its highlighting as his/her EEG and EOG signals are recorded. A multifrequency band-based canonical correlation analysis (CCA) method is applied to the EEG data to detect the evoked SSVEPs, whereas the EOG data are analyzed to identify the user's blinks. Finally, the target character is identified based on the SSVEP and blink detection results. Results: Ten healthy subjects participated in our experiments and achieved an average information transfer rate (ITR) of 105.52 bits/min, an average accuracy of 95.42%, an average response time of 1.34 s and an average false-positive rate (FPR) of 0.8%. Conclusion: The proposed BCI generates multiple commands with a high ITR and low FPR. Significance: The hybrid asynchronous BCI has great potential for practical applications in communication and control.

Stereo-electroencephalography (SEEG) signals can be obtained by implanting deep intracranial electrodes, which are currently used for epileptic diagnosis. In this study, we implemented a P300-based Brain Computer Interface (BCI) using SEEG signals. 40 buttons corresponding to 40 numbers displayed in a graphical user interface (GUI) were intensified in a random order. To select a number, the user could focus on the corresponding button when it was flashing. Five epileptic patients implanted with SEEG electrodes attended the experiment and achieved an average online accuracy of 97.33%. Moreover, through single contact decoding and simulated online analysis, we found that these subjects achieved an average accuracy of 82.00% using a single channel of signal. These results indicated that our SEEG-based BCI had a high performance, which was mainly because of the high quality of SEEG signals.

Survival prediction is a typical task in computer-aided diagnosis with many clinical applications. Existing approaches to survival prediction are mostly based on the classic Cox model, which mainly focus on learning a hazard or survival function rather than the survival time, largely limiting their practical uses. In this paper, we present a Censoring-Aware Deep Ordinal Regression (CDOR) to directly predict survival time from pathological images. Instead of relying on the Cox model, CDOR formulates survival prediction as an ordinal regression problem, and particularly introduces a censoring-aware loss function to train the deep network in the presence of censored data. Experiment results on publicly available dataset demonstrate that, the proposed CDOR can achieve significant higher accuracy in predicting survival time.

A triple modular redundancy (TMR) based design technique for double cell upsets (DCUs) mitigation is investigated in this paper. This technique adds three extra self-voter circuits into a traditional TMR structure to enable the enhanced error correction capability. Fault-injection simulations show that the soft error rate (SER) of the proposed technique is lower than 3% of that of TMR. The implementation of this proposed technique is compatible with the automatic digital design flow, and its applicability and performance are evaluated on an FIFO circuit.

Most existing brain-computer Interfaces (BCIs) are designed to control a single assistive device, such as a wheelchair, a robotic arm or a prosthetic limb. However, many daily tasks require combined functions which can only be realized by integrating multiple robotic devices. Such integration raises the requirement of the control accuracy and is more challenging to achieve a reliable control compared with the single device case. In this study, we propose a novel hybrid BCI with high accuracy based on electroencephalogram (EEG) and electrooculogram (EOG) to control an integrated wheelchair robotic arm system. The user turns the wheelchair left/right by performing left/right hand motor imagery (MI), and generates other commands for the wheelchair and the robotic arm by performing eye blinks and eyebrow raising movements. Twenty-two subjects participated in a MI training session and five of them completed a mobile self-drinking experiment, which was designed purposely with high accuracy requirements. The results demonstrated that the proposed hBCI could provide satisfied control accuracy for a system that consists of multiple robotic devices, and showed the potential of BCI-controlled systems to be applied in complex daily tasks.