Search

Reconstructing visual experience from brain responses measured by functional magnetic resonance imaging (fMRI) is a challenging yet important research topic in brain decoding, especially it has proved more difficult to decode visually similar stimuli, such as faces. Although face attributes are known as the key to face recognition, most existing methods generally ignore how to decode facial attributes more precisely in perceived face reconstruction, which often leads to indistinguishable reconstructed faces. To solve this problem, we propose a novel neural decoding framework called VSPnet (voxel2style2pixel) by establishing hierarchical encoding and decoding networks with disentangled latent representations as media, so that to recover visual stimuli more elaborately. And we design a hierarchical visual encoder (named HVE) to pre-extract features containing both high-level semantic knowledge and low-level visual details from stimuli. The proposed VSPnet consists of two networks: Multi-branch cognitive encoder and style-based image generator. The encoder network is constructed by multiple linear regression branches to map brain signals to the latent space provided by the pre-extracted visual features and obtain representations containing hierarchical information consistent to the corresponding stimuli. We make the generator network inspired by StyleGAN to untangle the complexity of fMRI representations and generate images. And the HVE network is composed of a standard feature pyramid over a ResNet backbone. Extensive experimental results on the latest public datasets have demonstrated the reconstruction accuracy of our proposed method outperforms the state-of-the-art approaches and the identifiability of different reconstructed faces has been greatly improved. In particular, we achieve feature editing for several facial attributes in fMRI domain based on the multiview (i.e. , visual stimuli and evoked fMRI) latent representations. • A style-oriented decoding framework is proposed to reconstruct perceived faces from brain activity. • Facial information captured through a multi-regression model can be disentangled by feature hierarchy. • Reconstructions of different faces can be distinguished more accurately than the state-of-the-art. • Feature editing can be performed in fMRI domain for the first time to control human face attributes. [ABSTRACT FROM AUTHOR]

State-of-the-art brain imaging studies have recently produced a variety of sometimes contradictory conclusions about the neural systems that support human semantic memory. Multivariate techniques deployed in this work adopt implicit or explicit assumptions that limit the types of signal they can detect, and thus the types of hypotheses they can test. We lay out the space of possible cognitive and neural representations and then critically review contemporary methods to determine which analyses can test which hypotheses. The results account for the heterogeneity of recent findings and identify an important empirical and methodological gap that makes it difficult to connect the imaging literature to neurocomputational models of semantic processing. A key goal for cognitive neuroscience is to understand the neurocognitive systems that support semantic memory. Recent multivariate analyses of neuroimaging data have contributed greatly to this effort, but the rapid development of these novel approaches has made it difficult to track the diversity of findings and to understand how and why they sometimes lead to contradictory conclusions. We address this challenge by reviewing cognitive theories of semantic representation and their neural instantiation. We then consider contemporary approaches to neural decoding and assess which types of representation each can possibly detect. The analysis suggests why the results are heterogeneous and identifies crucial links between cognitive theory, data collection, and analysis that can help to better connect neuroimaging to mechanistic theories of semantic cognition. [ABSTRACT FROM AUTHOR]

Emotion regulation (ER) processes help support well-being, but ineffective ER is implicated in several psychiatric disorders. Engaging ER flexibly by going online and offline as needs and capacities shift may be more effective than engaging ER rigidly across time. Here, we sought to observe the neural temporal dynamics of an ER process, reappraisal, during regulation of responses to negative memories in healthy control subjects (n = 33) and subjects with major depressive disorder (n = 36). To track the temporal dynamics of reappraisal neural systems, we used a functional magnetic resonance imaging neural decoding approach. In task 1, subjects explicitly engaged reappraisal on instruction in response to aversive images, and we used this task to develop the decoder for detecting reappraisal. In task 2, subjects experienced negative autobiographical memories from a distant (third person, ER condition) or immersed (first person, control condition) perspective. The neural decoder, trained to detect reappraisal in task 1, predicted greater reappraisal occurring during the task 2 distance versus immerse trials and was engaged more intensely during memories that were rated as being more negative. Across time, decoder output manifested a temporal dynamic of early engagement followed by disengagement. These results were replicated in an independent subject dataset (n = 59). Relative to healthy control subjects, subjects with major depressive disorder had a comparable initial increase in decoder engagement at the beginning of the trial but an attenuated decrease at the end. Subjects with major depressive disorder evidenced a more rigid neural dynamic of reappraisal compared with healthy control subjects. Rigid ER may indicate diminished ability to flexibly and effectively regulate emotion. [ABSTRACT FROM AUTHOR]

Spectral features of human electroencephalographic (EEG) recordings during learning predict subsequent recall variability. Capitalizing on these fluctuating neural features, we develop a non-invasive closed-loop (NICL) system for real-time optimization of human learning. Participants play a virtual navigation-and-memory game; recording multi-session data across days allowed us to build participant-specific classification models of recall success. In subsequent closed-loop sessions, our platform manipulated the timing of memory encoding, selectively presenting items during periods of predicted good or poor memory function based on EEG features decoded in real time. The induced memory effect (the difference between recall rates when presenting items during predicted good vs. poor learning periods) increased with the accuracy of neural decoding. This study demonstrates greater-than-chance memory decoding from EEG recordings in a naturalistic virtual navigation task with greater real-world validity than basic word-list recall paradigms. Here we modulate memory by timing stimulus presentation based on noninvasive scalp EEG recordings, whereas prior closed-loop studies for memory improvement involved intracranial recordings and direct electrical stimulation. Other noninvasive studies have investigated the use of neurofeedback or remedial study for memory improvement. These findings present a proof-of-concept for using non-invasive closed-loop technology to optimize human learning and memory through principled stimulus timing, but only in those participants for whom classifiers reliably predict out-of-sample memory function. • Classifiers decoded successful vs. unsuccessful memory function from scalp EEG. • Decoded neural features determined the timing of stimulus presentation. • Memory improvement depended on the accuracy of the underlying classifiers. [ABSTRACT FROM AUTHOR]

Despite the well-recognized role of the posterior parietal cortex (PPC) in processing sensory information to guide action, the differential encoding properties of this dynamic processing, as operated by different PPC brain areas, are scarcely known. Within the monkey's PPC, the superior parietal lobule hosts areas V6A, PEc, and PE included in the dorso-medial visual stream that is specialized in planning and guiding reaching movements. Here, a Convolutional Neural Network (CNN) approach is used to investigate how the information is processed in these areas. We trained two macaque monkeys to perform a delayed reaching task towards 9 positions (distributed on 3 different depth and direction levels) in the 3D peripersonal space. The activity of single cells was recorded from V6A, PEc, PE and fed to convolutional neural networks that were designed and trained to exploit the temporal structure of neuronal activation patterns, to decode the target positions reached by the monkey. Bayesian Optimization was used to define the main CNN hyper-parameters. In addition to discrete positions in space, we used the same network architecture to decode plausible reaching trajectories. We found that data from the most caudal V6A and PEc areas outperformed PE area in the spatial position decoding. In all areas, decoding accuracies started to increase at the time the target to reach was instructed to the monkey, and reached a plateau at movement onset. The results support a dynamic encoding of the different phases and properties of the reaching movement differentially distributed over a network of interconnected areas. This study highlights the usefulness of neurons' firing rate decoding via CNNs to improve our understanding of how sensorimotor information is encoded in PPC to perform reaching movements. The obtained results may have implications in the perspective of novel neuroprosthetic devices based on the decoding of these rich signals for faithfully carrying out patient's intentions. [ABSTRACT FROM AUTHOR]

• In this study, we proposed motorSRNN, a recurrent spiking neural network (SNN) that draws inspiration from the neural motor circuit of primates. • The motorSRNN advanced the performance of previously reported SNN method in similar CST-decoding tasks, a feedforward SNN (fSNN), by over 25%. • The energy efficiency of motorSRNN was theoretically approximately 1/50 compared to traditional GRU and LSTM architectures. • The motorSRNN outperformed fSNN, GRU, and LSTM in terms of early-classification capabilities from 2 ms to the end in the 50-ms sample duration. • The motorSRNN elucidates a plausible rationale for the biologically employed topology: to enhance the resilience against Poisson noise from adjacent neurons. Decoding firing rates, averaged from cortical spike trains (CST), has yielded significant progress in invasive brain-machine interfaces (BMI). CSTs are theoretically more informative and efficient than firing rates. By directly decoding CST, spiking neural networks (SNN) exhibit promise for enhancing invasive BMIs due to high compatibility with CST and a low-energy consuming nature. However, whether SNNs can decode CST with applicable performance in terms of classification accuracy and energy consumption remains unclear. In this study, we proposed motorSRNN, a recurrent SNN topologically inspired by the primate motor neural circuit. Employed to decode CST from the primary motor cortex of two monkeys performing 4-direction reaching tasks, the motorSRNN achieved average classification accuracies of 89.44 % and 79.87 % for the 4 directions, respectively. This outperformed previously reported SNN method in similar CST-decoding tasks, a feedforward SNN (fSNN), by more than 25 %. Furthermore, motorSRNN demonstrated superior early-classification capabilities compared to fSNN, GRU, and LSTM from 2 ms to the end in the 50-ms sample duration. Additionally, it only theoretically consumed around 1/50 energy compared to traditional GRU and LSTM. Finally, motorSRNN offers insights into a possible rationale for the biologically employed topology: to enhance the resilience against Poisson noise from neighboring neurons in the biological brains. In conclusion, our proposed motorSRNN is feasible for effective and efficient CST decoding, laying the preliminary groundwork for constructing a fully implanted neuromorphic BMI. [ABSTRACT FROM AUTHOR]

• A novel approach has been developed to improve the robustness of the iBCIs to the changes in neural recording conditions. • This approach leverages contrastive learning and neural data augmentation to learn consistent neural-to-kinematic mapping. • Experimental results demonstrate that this approach can make iBCIs robust to recording condition changes over 83 days on one publicly available nonhuman primate dataset. • This approach outperforms the state-of-the-art iBCIs for long-term use, even when only 11.2 minutes of training data are available. • Our approach reduces the need for frequent calibration of iBCIs and the amount of time required for technicians to gather extensive annotated training data. Intracortical brain-computer interfaces (iBCIs) aim to help paralyzed individuals restore their motor functions by decoding neural activity into intended movement. However, changes in neural recording conditions hinder the decoding performance of iBCIs, mainly because the neural-to-kinematic mappings shift. Conventional approaches involve either training the neural decoders using large datasets before deploying the iBCI or conducting frequent calibrations during its operation. However, collecting data for extended periods can cause user fatigue, negatively impacting the quality and consistency of neural signals. Furthermore, frequent calibration imposes a substantial computational load. This study proposes a novel approach to increase iBCIs' robustness against changing recording conditions. The approach uses three neural augmentation operators to generate augmented neural activity that mimics common recording conditions. Then, contrastive learning is used to learn latent factors by maximizing the similarity between the augmented neural activities. The learned factors are expected to remain stable despite varying recording conditions and maintain a consistent correlation with the intended movement. Experimental results demonstrate that the proposed iBCI outperformed the state-of-the-art iBCIs and was robust to changing recording conditions across days for long-term use on one publicly available nonhuman primate dataset. It achieved satisfactory offline decoding performance, even when a large training dataset was unavailable. This study paves the way for reducing the need for frequent calibration of iBCIs and collecting a large amount of annotated training data. Potential future works aim to improve offline decoding performance with an ultra-small training dataset and improve the iBCIs' robustness to severely disabled electrodes. Proposed L atent F actor learning by D ata A ugmentation (LFDA) aims to increase robustness of intracortical brain-computer interfaces under changing recording conditions. [Display omitted] [ABSTRACT FROM AUTHOR]

The reconstruction of visual stimuli from fMRI data represents a major technological and scientific challenge at the forefront of contemporary neuroscience research. Deep learning techniques have played a critical role in advancing decoding models for visual stimulus reconstruction from fMRI data. Particularly, the use of advanced GANs has resulted in significant improvements in the quality of image generation, providing a powerful tool for addressing the challenges of this complex task. However, none of these studies have taken into account the inherent characteristics of the stimulus contents themselves; This, in turn, leads to unsatisfactory outcomes, as demonstrated by the inconsistent identity between reconstructed faces and ground truth in the decoding of facial images. In order to tackle this challenge, we introduce a new framework aimed at enhancing the accuracy of reconstructing facial images from fMRI data. Our key innovation involves extracting and disentangling multi-level visual information from brain signals in the latent space and optimizing high-level features for facial identity control using identity loss. Specifically, our framework uses StyleGAN inversion to extract hierarchical latent codes from images, which are then bridged to fMRI data through transformation blocks. Additionally, we introduce a multi-stage refinement method to enhance the accuracy of reconstructed faces, which involves progressively updating fMRI latent codes with custom loss functions designed for both feature- and image-wise optimization. Our experimental results demonstrate that our proposed framework effectively achieves two critical objectives: (1) accurate facial image reconstruction from fMRI data and (2) preservation of identity characteristics with a high level of consistency. • We propose a new cross-modal decoding framework for brain-driven image reconstruction. • We leverage StyleGAN inversion to enable realistic visual quality of reconstructions. • We disentangle face representation to make hierarchical multi-modal semantic alignment. • We design progressive refinement learning to iteratively refine recovery accuracy. • We introduce contrastive loss and multi-layer identity loss to improve ID consistency. [ABSTRACT FROM AUTHOR]

Transforming neural activities into language is revolutionary for human–computer interaction as well as functional restoration of aphasia. Present rapid development of artificial intelligence makes it feasible to decode the neural signals of human visual activities. In this paper, a novel Progressive Transfer Language Decoding Model (PT-LDM) is proposed to decode visual fMRI signals into phrases or sentences when natural images are being watched. The PT-LDM consists of an image-encoder, a fMRI encoder and a language-decoder. The results showed that phrases and sentences were successfully generated from visual activities. Similarity analysis showed that three often-used evaluation indexes BLEU, ROUGE and CIDEr reached 0.182, 0.197 and 0.680 averagely between the generated texts and the corresponding annotated texts in the testing set respectively, significantly higher than the baseline. Moreover, we found that higher visual areas usually had better performance than lower visual areas and the contribution curve of visual response patterns in language decoding varied at successively different time points. Our findings demonstrate that the neural representations elicited in visual cortices when scenes are being viewed have already contained semantic information that can be utilized to generate human language. Our study shows potential application of language-based brain–machine interfaces in the future, especially for assisting aphasics in communicating more efficiently with fMRI signals. [ABSTRACT FROM AUTHOR]

• Improving the spike sorting accuracy can improve neural decoding performance. • A spike sorting algorithm is presented that uses optimized continuous wavelet coefficients as features. • In the simulation study, the proposed spike sorting outperformed both the WaveClus and PCA-based spike sorting algorithms. • In real data test, the proposed spike sorting improved neural decoding performance compared to PCA-based spike sorting. Spike sorting is an essential step in extracting neuronal discharge patterns which help to decode different activities in the neural system. Therefore, improving the spike sorting accuracy can improve neural decoding performance subsequently. Although many methods are suggested for spike sorting, few studies have evaluated their effect on neural decoding performance. In this paper, a method of spike sorting based on an optimized selection of the parameters in the continuous wavelet transform (CWT) is proposed. The proposed algorithm was tested on a simulated dataset and two publicly available benchmark datasets to evaluate its performance in spike sorting. To evaluate the effect of utilizing different spike sorting algorithms on neural decoding performance, real data was used in which the aim was to decode the force applied by the rat's hand to a pedal continuously from the intra-cortical data of the primary motor area of the cortex. The extracted neuronal firing rates by the spike sorting algorithms were applied to a partial least squares regression to decode the force signal. In the simulation study, the proposed spike sorting algorithm based on optimized wavelet parameter selection outperformed both the WaveClus spike sorting and traditional PCA-based spike sorting algorithms. The results showed the superiority of the spike sorting algorithm based on optimal wavelet parameters compared to classical discrete wavelet transform (DWT) or PCA-based spike sorting methods in decoding real intracortical data. Overall, the results indicate that it is possible to improve neural decoding performance by improving the spike sorting accuracy. [ABSTRACT FROM AUTHOR]

Recent research has demonstrated that neural and behavioral data acquired in response to viewing face images can be used to reconstruct the images themselves. However, the theoretical implications, promises, and challenges of this direction of research remain unclear. We evaluate the potential of this research for elucidating the visual representations underlying face recognition. Specifically, we outline complementary and converging accounts of the visual content, the representational structure, and the neural dynamics of face processing. We illustrate how this research addresses fundamental questions in the study of normal and impaired face recognition, and how image reconstruction provides a powerful framework for uncovering face representations, for unifying multiple types of empirical data, and for facilitating both theoretical and methodological progress. The visual content of human face representations can be derived through image reconstruction at an unprecedented level of detail. The neural underpinnings of psychological face representations can be elucidated via image reconstruction from behavioral and multiple types of neural data. Facial image reconstruction capitalizes on the structure of face space and also serves to validate its representational properties. The recovery of face representations from both perception and memory can help to clarify the relationship between these two components of cognition. Image reconstruction can pinpoint the loss of information and/or misrepresentations in face perception across a broad population, including individuals with deficits in face processing. [ABSTRACT FROM AUTHOR]

Neural coding is one of the central questions in systems neuroscience for understanding how the brain processes stimulus from the environment, moreover, it is also a cornerstone for designing algorithms of brain–machine interface, where decoding incoming stimulus is highly demanded for better performance of physical devices. Traditionally researchers have focused on functional magnetic resonance imaging (fMRI) data as the neural signals of interest for decoding visual scenes. However, our visual perception operates in a fast time scale of millisecond in terms of an event termed neural spike. There are few studies of decoding by using spikes. Here we fulfill this aim by developing a novel decoding framework based on deep neural networks, named spike-image decoder (SID), for reconstructing natural visual scenes, including static images and dynamic videos, from experimentally recorded spikes of a population of retinal ganglion cells. The SID is an end-to-end decoder with one end as neural spikes and the other end as images, which can be trained directly such that visual scenes are reconstructed from spikes in a highly accurate fashion. Our SID also outperforms on the reconstruction of visual stimulus compared to existing fMRI decoding models. In addition, with the aid of a spike encoder, we show that SID can be generalized to arbitrary visual scenes by using the image datasets of MNIST, CIFAR10, and CIFAR100. Furthermore, with a pre-trained SID, one can decode any dynamic videos to achieve real-time encoding and decoding of visual scenes by spikes. Altogether, our results shed new light on neuromorphic computing for artificial visual systems, such as event-based visual cameras and visual neuroprostheses. [ABSTRACT FROM AUTHOR]

• A systematic baseline evaluation of various machine learning methods and metrics. • The impact of different brain regions varies for each visual task. • Empirical information is provided about which models perform best for each visual task. • Different similarity metrics improve different visual tasks. • Decoding accuracy is 94% for the image presentation task and 74% for the imagery task. Visual representation decoding refers to the task of deciphering what a subject is seeing or visualizing by observing the brain state via neuroimaging. In recent years, there is an increasing interest towards tackling the aforementioned task through the use of machine learning approaches. This study provides an extensive evaluation that will serve as a baseline for visual representation decoding, by exploring a wide range of model configurations, feature representations and evaluation setups. In this way, this work lays the groundwork for developing more sophisticated and accurate decoding pipelines. The evaluation results suggest that neural networks provide, on average, the best performance, while choosing the most appropriate similarity metric for the class decoding process depends mostly on the task at hand. Finally, this work may also assist domain experts to gain high-level insights about the brain's function, through several interesting observations, e.g., our findings hint brain regions that are dominant for specific tasks and back up related claims about potential correspondence of the cortical hierarchy with deep visual representations. [ABSTRACT FROM AUTHOR]

Abstract The speech envelope is essential for speech understanding and can be reconstructed from the electroencephalogram (EEG) recorded while listening to running speech. This so-called "neural envelope tracking" has been shown to relate to speech understanding in normal hearing listeners, but has barely been investigated in persons wearing cochlear implants (CI). We investigated the relation between speech understanding and neural envelope tracking in CI users. EEG was recorded in 8 CI users while they listened to a story. Speech understanding was varied by changing the intensity of the presented speech. The speech envelope was reconstructed from the EEG using a linear decoder. Next, the reconstructed envelope was correlated with the envelope of the speech stimulus as a measure of neural envelope tracking which was compared to actual speech understanding. This study showed that neural envelope tracking increased with increasing speech understanding in every participant. Furthermore behaviorally measured speech understanding was correlated with participant-specific neural envelope tracking results indicating the potential of neural envelope tracking as an objective measure of speech understanding in CI users. This could enable objective and automatic fitting of CIs and pave the way towards closed-loop CIs that adjust continuously and automatically to individual CI users. Highlights • Neural envelope tracking can be measured in CI users in response to running speech. • Neural envelope tracking increases with increasing speech understanding in CI users. • Speech understanding correlates with neural envelope tracking results in CI users. [ABSTRACT FROM AUTHOR]

An intrinsic difficulty in studying cognitive processes is that they are unobservable states that exist in between observable responses to the sensory environment. Cognitive states must be inferred from indirect behavioral measures. Neuroscience potentially provides the tools necessary to measure cognitive processes directly, but it is challenged on two fronts. First, neuroscientific measures often lack the spatiotemporal resolution to identify the neural computations that underlie a cognitive process. Second, the activity of a single neuron, which is the fundamental building block of neural computation, is too noisy to provide accurate measurements of a cognitive process. In this paper, I examine recent developments in neurophysiological recording and analysis methods that provide a potential solution to these problems. Highlights Recent advances in analytic methods and high-channel count recordings have raised the possibility of reading out cognitive processes directly from the brain, as opposed to inferring cognitive processes indirectly from behavior. Decoding neural activity has been used to understand decision making by using place cell activity in the hippocampus or value-selective neural responses in orbitofrontal cortex. Decoding could have broad applications for measuring other cognitive processes directly from neural activity, such as attention, working memory and reasoning. [ABSTRACT FROM AUTHOR]

The primary motor cortex (M1) exhibits a protracted period of development, including the development of a sensory representation long before motor outflow emerges. In rats, this representation is present by postnatal day (P) 8, when M1 activity is "discontinuous." Here, we ask how the representation changes upon the transition to "continuous" activity at P12. We use neural decoding to predict forelimb movements from M1 activity and show that a linear decoder effectively predicts limb movements at P8 but not at P12; instead, a nonlinear decoder better predicts limb movements at P12. The altered decoder performance reflects increased complexity and uniqueness of kinematic information in M1. We next show that M1's representation at P12 is more susceptible to "lesioning" of inputs and "transplanting" of M1's encoding scheme from one pup to another. Thus, the emergence of continuous M1 activity signals the developmental onset of more complex, informationally sparse, and individualized sensory representations. [Display omitted] • In P8 rats, a linear decoder reliably predicts limb movements from M1 activity • At P12, M1 activity is continuous, and a nonlinear decoder is required • M1 activity is more sensitive to "lesions" and encoding scheme "transplants" at P12 • P12 marks the onset of more sparse and individualized M1 movement representations Cortical activity transitions from discontinuous to continuous around postnatal day 12 in rats. Glanz et al. use neural decoding to assess this transition's effect on sensory representations of limb kinematics in the primary motor cortex (M1). After the transition, M1 activity is more informationally dense, and its encoding scheme is more individualized. [ABSTRACT FROM AUTHOR]

Estimating discharge patterns of motor units by electromyography (EMG) decomposition has shown promising perspectives in neurophysiologic investigations and human–machine interfaces. However, the number of decoded motor units is still limited, especially under high excitation and noisy conditions, which is a major barrier to the broad application of EMG decomposition. In this work, we applied a peel-off decomposition strategy to the convolution kernel compensation (CKC) algorithm to extract more motor units non-invasively. EMG signals were firstly decomposed into motor unit spike trains (MUSTs) using the CKC. After each decomposition, the motor unit action potentials (MUAPs) were reconstructed and subtracted from the EMG signals. Then the same CKC decomposition was applied to the residual signals. This peel-off procedure was repeated until no valid motor units were identified. The proposed decomposition strategy was validated on synthetic EMG signals by convolving simulated MUSTs and experimentally extracted MUAPs under multiple excitations and noise conditions. Then the decomposition performance was evaluated on experimental data. Compared with the classic CKC method, the number of the identified motor units from synthetic signals was significantly increased by 7 to 43 in each decomposition. Moreover, the average agreement between identified MUST and ground truth was 0.80 ± 0.20, indicating a high decomposition accuracy. From experimental EMG signals, the peel-off method could identify more motor units (> 20) with high confidence than the classic method (< 5) across three excitation levels. These results demonstrate the efficiency of the proposed method in identifying more motor units from EMG signals, extending the potential applications of surface EMG decomposition for neural decoding. [ABSTRACT FROM AUTHOR]

• The effect of unfamiliar topics and volatile listening on attention was probed. • Topic familiarity had a minimal effect on the cortical tracking of attended speech. • Topics seemed to be less useful when sufficient bottom-up information was available. • Volatile listening was formed by dynamic and irregular changes in auditory scenes. • Volatile listening degraded attention and the neural tracking of attended speech. Selective auditory attention has been shown to modulate the cortical representation of speech. This effect has been well documented in acoustically more challenging environments. However, the influence of top-down factors, in particular topic familiarity, on this process remains unclear, despite evidence that semantic information can promote speech-in-noise perception. Apart from individual features forming a static listening condition, dynamic and irregular changes of auditory scenes—volatile listening environments—have been less studied. To address these gaps, we explored the influence of topic familiarity and volatile listening on the selective auditory attention process during dichotic listening using electroencephalography. When stories with unfamiliar topics were presented, participants' comprehension was severely degraded. However, their cortical activity selectively tracked the speech of the target story well. This implies that topic familiarity hardly influences the speech tracking neural index, possibly when the bottom-up information is sufficient. However, when the listening environment was volatile and the listeners had to re-engage in new speech whenever auditory scenes altered, the neural correlates of the attended speech were degraded. In particular, the cortical response to the attended speech and the spatial asymmetry of the response to the left and right attention were significantly attenuated around 100–200 ms after the speech onset. These findings suggest that volatile listening environments could adversely affect the modulation effect of selective attention, possibly by hampering proper attention due to increased perceptual load. [ABSTRACT FROM AUTHOR]

This review presents a perspective on capturing collective dynamics in recorded neuronal ensembles based on multivariate point process models, inference of low-dimensional dynamics and coarse graining of spatiotemporal measurements. A general probabilistic framework for continuous time point processes reviewed, with an emphasis on multivariate nonlinear Hawkes processes with exogenous inputs. A point process generalized linear model (PP-GLM) framework for the estimation of discrete time multivariate nonlinear Hawkes processes is described. The approach is illustrated with the modeling of collective dynamics in neocortical neuronal ensembles recorded in human and non-human primates, and prediction of single-neuron spiking. A complementary approach to capture collective dynamics based on low-dimensional dynamics (“order parameters”) inferred via latent state-space models with point process observations is presented. The approach is illustrated by inferring and decoding low-dimensional dynamics in primate motor cortex during naturalistic reach and grasp movements. Finally, we briefly review hypothesis tests based on conditional inference and spatiotemporal coarse graining for assessing collective dynamics in recorded neuronal ensembles. [ABSTRACT FROM AUTHOR]

Restoring communication in case of aphasia is a key challenge for neurotechnologies. To this end, brain-computer strategies can be envisioned to allow artificial speech synthesis from the continuous decoding of neural signals underlying speech imagination. Such speech brain-computer interfaces do not exist yet and their design should consider three key choices that need to be made: the choice of appropriate brain regions to record neural activity from, the choice of an appropriate recording technique, and the choice of a neural decoding scheme in association with an appropriate speech synthesis method. These key considerations are discussed here in light of (1) the current understanding of the functional neuroanatomy of cortical areas underlying overt and covert speech production, (2) the available literature making use of a variety of brain recording techniques to better characterize and address the challenge of decoding cortical speech signals, and (3) the different speech synthesis approaches that can be considered depending on the level of speech representation (phonetic, acoustic or articulatory) envisioned to be decoded at the core of a speech BCI paradigm. [ABSTRACT FROM AUTHOR]

A central, unresolved problem in affective neuroscience is understanding how emotions are represented in nervous system activity. After prior localization approaches largely failed, researchers began applying multivariate statistical tools to reconceptualize how emotion constructs might be embedded in large-scale brain networks. Findings from pattern analyses of neuroimaging data show that affective dimensions and emotion categories are uniquely represented in the activity of distributed neural systems that span cortical and subcortical regions. Results from multiple-category decoding studies are incompatible with theories postulating that specific emotions emerge from the neural coding of valence and arousal. This ‘new look’ into emotion representation promises to improve and reformulate neurobiological models of affect. [ABSTRACT FROM AUTHOR]

To gain a better understanding of how neural ensembles communicate and process information, neural decoding algorithms are used to extract information encoded in their spiking activity. Bayesian decoding is one of the most used neural population decoding approaches to extract information from the ensemble spiking activity of rat hippocampal neurons. Recently it has been shown how Bayesian decoding can be implemented without the intermediate step of sorting spike waveforms into groups of single units. Here we extend the approach in order to make it suitable for online encoding/decoding scenarios that require real-time decoding such as brain-machine interfaces. We propose an online algorithm for the Bayesian decoding that reduces the time required for decoding neural populations, resulting in a real-time capable decoding framework. More specifically, we improve the speed of the probability density estimation step, which is the most essential and the most expensive computation of the spike-sorting-less decoding process, by developing a kernel density compression algorithm. In contrary to existing online kernel compression techniques, rather than optimizing for the minimum estimation error caused by kernels compression, the proposed method compresses kernels on the basis of the distance between the merging component and its most similar neighbor. Thus, without costly optimization, the proposed method has very low compression latency with a small and manageable estimation error. In addition, the proposed bandwidth matching method for Gaussian kernels merging has an interesting mathematical property whereby optimization in the estimation of the probability density function can be performed efficiently, resulting in a faster decoding speed. We successfully applied the proposed kernel compression algorithm to the Bayesian decoding framework to reconstruct positions of a freely moving rat from hippocampal unsorted spikes, with significant improvements in the decoding speed and acceptable decoding error. [ABSTRACT FROM AUTHOR]

A recent article shows that the brain automatically estimates the probabilities of possible future actions before it has even received all the information necessary to decide what to do next. [ABSTRACT FROM AUTHOR]

Background Multiple types of neural signals are available for controlling assistive devices through brain–computer interfaces (BCIs). Intracortically recorded spiking neural signals are attractive for BCIs because they can in principle provide greater fidelity of encoded information compared to electrocorticographic (ECoG) signals and electroencephalograms (EEGs). Recent reports show that the information content of these spiking neural signals can be reliably extracted simply by causally band-pass filtering the recorded extracellular voltage signals and then applying a spike detection threshold, without relying on “sorting” action potentials. New method We show that replacing the causal filter with an equivalent non-causal filter increases the information content extracted from the extracellular spiking signal and improves decoding of intended movement direction. This method can be used for real-time BCI applications by using a 4 ms lag between recording and filtering neural signals. Results Across 18 sessions from two people with tetraplegia enrolled in the BrainGate2 pilot clinical trial, we found that threshold crossing events extracted using this non-causal filtering method were significantly more informative of each participant's intended cursor kinematics compared to threshold crossing events derived from causally filtered signals. This new method decreased the mean angular error between the intended and decoded cursor direction by 9.7° for participant S3, who was implanted 5.4 years prior to this study, and by 3.5° for participant T2, who was implanted 3 months prior to this study. Conclusions Non-causally filtering neural signals prior to extracting threshold crossing events may be a simple yet effective way to condition intracortically recorded neural activity for direct control of external devices through BCIs. [ABSTRACT FROM AUTHOR]

Abstract: Future generations of upper limb prosthesis will have dexterous hand with individual fingers and will be controlled directly by neural signals. Neurons from the primary motor (M1) cortex code for finger movements and provide the source for neural control of dexterous prosthesis. Each neuron''s activation can be quantified by the change in firing rate before and after finger movement, and the quantified value is then represented by the neural activity over each trial for the intended movement. Since this neural activity varies with the intended movement, we define the relative importance of each neuron independent of specific intended movements. The relative importance of each neuron is determined by the inter-movement variance of the neural activities for respective intended movements. Neurons are ranked by the relative importance and then a subpopulation of rank-ordered neurons is selected for the neural decoding. The use of the proposed neuron selection method in individual finger movements improved decoding accuracy by 21.5% in the case of decoding with only 5 neurons and by 9.2% in the case of decoding with only 10 neurons. With only 15 highly ranked neurons, a decoding accuracy of 99.5% was achieved. The performance improvement is still maintained when combined movements of two fingers were included though the decoding accuracy fell to 95.7%. Since the proposed neuron selection method can achieve the targeting accuracy of decoding algorithms with less number of input neurons, it can be significant for developing brain–machine interfaces for direct neural control of hand prostheses. [Copyright &y& Elsevier]

Abstract: High performance computation is critical for brain–machine interface (BMI) applications. Current BMI decoding algorithms are always implemented on personal computers (PC) which affect the performance of complex mapping models. In this paper, an FPGA implementation of Kalman filter (KF) algorithm is proposed as a new computational method. The neural ensemble activities are recorded from motor cortex of rats performing a lever-pressing task for water reward. Kalman filter, which is used for mapping neural activities to kinematic variables, is implemented both on PC (MATLAB-based) and FPGA. In FPGA architecture, the row/column-based method is adopted for the matrix operation instead of the traditional element-based method, parallel and pipelined structures are also used for efficient computation at the same time. The results show that the FPGA-based implementation runs 24.45 times faster than the PC-based counterpart while achieving the same accuracy. Such a hardware-based computational method provides a tool for high-performance computation, with profound implications for portable BMI application. [Copyright &y& Elsevier]

Abstract: Generalized linear models (GLMs) have been developed for modeling and decoding population neuronal spiking activity in the motor cortex. These models provide reasonable characterizations between neural activity and motor behavior. However, they lack a description of movement-related terms which are not observed directly in these experiments, such as muscular activation, the subject’s level of attention, and other internal or external states. Here we propose to include a multi-dimensional hidden state to address these states in a GLM framework where the spike count at each time is described as a function of the hand state (position, velocity, and acceleration), truncated spike history, and the hidden state. The model can be identified by an Expectation–Maximization algorithm. We tested this new method in two datasets where spikes were simultaneously recorded using a multi-electrode array in the primary motor cortex of two monkeys. It was found that this method significantly improves the model-fitting over the classical GLM, for hidden dimensions varying from 1 to 4. This method also provides more accurate decoding of hand state (reducing the mean square error by up to 29% in some cases), while retaining real-time computational efficiency. These improvements on representation and decoding over the classical GLM model suggest that this new approach could contribute as a useful tool to motor cortical decoding and prosthetic applications. [Copyright &y& Elsevier]

Abstract: Brain–machine interfaces (BMIs) aim to translate the motor intent of locked-in patients into neuroprosthetic control commands. Electrocorticographical (ECoG) signals provide promising neural inputs to BMIs as shown in recent studies. In this paper, we utilize a broadband spectrum above the fast gamma ranges and systematically study the role of spectral resolution, in which the broadband is partitioned, on the reconstruction of the patients’ hand trajectories. Traditionally, the power of ECoG rhythms (<200–300 Hz) has been computed in short duration bins and instantaneously and linearly mapped to cursor trajectories. Neither time embedding, nor nonlinear mappings have been previously implemented in ECoG neuroprosthesis. Herein, mapping of neural modulations to goal-oriented motor behavior is achieved via linear adaptive filters with embedded memory depths and as a novelty through echo state networks (ESNs), which provide nonlinear mappings without compromising training complexity or increasing the number of model parameters, with up to 85% correlation. Reconstructed hand trajectories are analyzed through spatial, spectral and temporal sensitivities. The superiority of nonlinear mappings in the cases of low spectral resolution and abundance of interictal activity is discussed. [Copyright &y& Elsevier]

In this study, we investigated the use of particle filtering in reconstructing time-varying input visual signals based on Macaque V1 neurons’ responses. A multitude of hypothesis particles are proposed for reconstructing the input stimulus up to time t. A prediction kernel (consisting of the first- and second-order forward Wiener kernels, derived by regression) is used to predict the neural response at time t based on the estimated input signals in the 200 ms prior to t. The fitness of this estimated response in predicting the measured response at time t is used to weigh the importance of the various hypotheses. The hypothesis particle space is collapsed by re-sampling over time. We find this method quite successful in reconstructing the input stimulus for 30 out of 33 V1 neurons measured. It out performs the optimal linear decoder that we have experimented with in the past (Neurocomputing, in press). [Copyright &y& Elsevier]

New research suggests that magnetoencephalography (MEG) contains rich spatial information for decoding neural states. Even small differences in the angle of neighbouring dipoles generate subtle, but statistically separable field patterns. This implies MEG (and electroencephalography: EEG) is ideal for decoding neural states with high-temporal resolution in the human brain. [ABSTRACT FROM AUTHOR]

• The brain tracks dynamic statistical structure of natural sounds along many dimensions. • A predictive inference model, D-REX, tracks statistics to interpret future inputs. • The model is agnostic to experimental paradigms and statistics enabling broad applications. The brain tracks sound sources as they evolve in time, collecting contextual information to predict future sensory inputs. Previous work in predictive coding typically focuses on the perception of predictable stimuli, leaving the implementation of these same neural processes in more complex, real-world environments containing randomness and uncertainty up for debate. To facilitate investigation into the perception of less tightly-controlled listening scenarios, we present a computational model as a tool to ask targeted questions about the underlying predictive processes that connect complex sensory inputs to listener behavior and neural responses. In the modeling framework, observed sound features (e.g. pitch) are tracked sequentially using Bayesian inference. Sufficient statistics are inferred from past observations at multiple time scales and used to make predictions about future observation while tracking the statistical structure of the sensory input. Facets of the model are discussed in terms of their application to perceptual research, and examples taken from real-world audio demonstrate the model's flexibility to capture a variety of statistical structures along various perceptual dimensions. Previous models are often targeted toward interpreting a particular experimental paradigm (e.g., oddball paradigm), perceptual dimension (e.g., pitch processing), or task (e.g., speech segregation), thus limiting their ability to generalize to other domains. The presented model is designed as a flexible and practical tool for broad application. The model is presented as a general framework for generating new hypotheses and guiding investigation into the neural processes underlying predictive coding of complex scenes. [ABSTRACT FROM AUTHOR]

• Introduces novel automated workflow for long-term human brain & behavior recordings. • Develops domain-relevant, robust, temporally precise, queryable pose representation. • Hundreds of upper limb movement events per day detected in epilepsy monitoring data. • Applied to neural correlates & neural decoding of naturalistic movement initiation. • Curated dataset of annotated events & brain data: github.com/BruntonUWBio/mining2021. Recent technological advances in brain recording and machine learning algorithms are enabling the study of neural activity underlying spontaneous human behaviors, beyond the confines of cued, repeated trials. However, analyzing such unstructured data lacking a priori experimental design remains a significant challenge, especially when the data is multi-modal and long-term. Here we describe an automated, behavior-first approach for analyzing simultaneously recorded long-term, naturalistic electrocorticography (ECoG) and behavior video data. We identify and characterize spontaneous human upper-limb movements by combining computer vision, discrete latent-variable modeling, and string pattern-matching on the video. Our pipeline discovers and annotates over 40,000 instances of naturalistic arm movements in long term (7–9 day) behavioral videos, across 12 subjects. Analysis of the simultaneously recorded brain data reveals neural signatures of movement that corroborate previous findings. Our pipeline produces large training datasets for brain–computer interfacing applications, and we show decoding results from a movement initiation detection task. Spontaneous movements capture real-world neural and behavior variability that is missing from traditional cued tasks. Building beyond window-based movement detection metrics, our unsupervised discretization scheme produces a queryable pose representation, allowing localization of movements with finer temporal resolution. Our work addresses the unique analytic challenges of studying naturalistic human behaviors and contributes methods that may generalize to other neural recording modalities beyond ECoG. We publish our curated dataset and believe that it will be a valuable resource for future studies of naturalistic movements. [ABSTRACT FROM AUTHOR]

• We elucidate the neural dynamics of detecting a moving object by a moving observer. • Object detection was predicted by machine learning based on motion-sensitive and fronto-parietal areas. • Connectivity verified the cortical dynamics which machine learning relied on. Relatively little is known about how the human brain identifies movement of objects while the observer is also moving in the environment. This is, ecologically, one of the most fundamental motion processing problems, critical for survival. To study this problem, we used a task which involved nine textured spheres moving in depth, eight simulating the observer's forward motion while the ninth, the target, moved independently with a different speed towards or away from the observer. Capitalizing on the high temporal resolution of magnetoencephalography (MEG) we trained a Support Vector Classifier (SVC) using the sensor-level data to identify correct and incorrect responses. Using the same MEG data, we addressed the dynamics of cortical processes involved in the detection of the independently moving object and investigated whether we could obtain confirmatory evidence for the brain activity patterns used by the classifier. Our findings indicate that response correctness could be reliably predicted by the SVC, with the highest accuracy during the blank period after motion and preceding the response. The spatial distribution of the areas critical for the correct prediction was similar but not exclusive to areas underlying the evoked activity. Importantly, SVC identified frontal areas otherwise not detected with evoked activity that seem to be important for the successful performance in the task. Dynamic connectivity further supported the involvement of frontal and occipital-temporal areas during the task periods. This is the first study to dynamically map cortical areas using a fully data-driven approach in order to investigate the neural mechanisms involved in the detection of moving objects during observer's self-motion. [ABSTRACT FROM AUTHOR]

• Bidirectional neural interfaces are a key component for the functional control of neuroprostheses and FES systems. • Recording of neural signals is useful to obtain motor control or sensory commands. • Nerve recordings can provide: cumulative nerve activity, spike signals from individual fibers, and hybrid signals. • We review the different approaches for nerve signals recording, denoising, processing and classification. Neurointerfaces have acquired major relevance as both rehabilitative and therapeutic tools for patients with spinal cord injury, limb amputations and other neural disorders. Bidirectional neural interfaces are a key component for the functional control of neuroprosthetic devices. The two main neuroprosthetic applications of interfaces with the peripheral nervous system (PNS) are: the refined control of artificial prostheses with sensory neural feedback, and functional electrical stimulation (FES) systems attempting to generate motor or visceral responses in paralyzed organs. The results obtained in experimental and clinical studies with both, extraneural and intraneural electrodes are very promising in terms of the achieved functionality for the neural stimulation mode. However, the results of neural recordings with peripheral nerve interfaces are more limited. In this paper we review the different existing approaches for PNS signals recording, denoising, processing and classification, enabling their use for bidirectional interfaces. PNS recordings can provide three types of signals: i) population activity signals recorded by using extraneural electrodes placed on the outer surface of the nerve, which carry information about cumulative nerve activity; ii) spike activity signals recorded with intraneural electrodes placed inside the nerve, which carry information about the electrical activity of a set of individual nerve fibers; and iii) hybrid signals, which contain both spiking and cumulative signals. Finally, we also point out some of the main limitations, which are hampering clinical translation of neural decoding, and indicate possible solutions for improvement. [ABSTRACT FROM AUTHOR]

Summary Uncovering spatial representations from large-scale ensemble spike activity in specific brain circuits provides valuable feedback in closed-loop experiments. We develop a graphics processing unit (GPU)-powered population-decoding system for ultrafast reconstruction of spatial positions from rodents' unsorted spatiotemporal spiking patterns, during run behavior or sleep. In comparison with an optimized quad-core central processing unit (CPU) implementation, our approach achieves an ∼20- to 50-fold increase in speed in eight tested rat hippocampal, cortical, and thalamic ensemble recordings, with real-time decoding speed (approximately fraction of a millisecond per spike) and scalability up to thousands of channels. By accommodating parallel shuffling in real time (computation time <15 ms), our approach enables assessment of the statistical significance of online-decoded "memory replay" candidates during quiet wakefulness or sleep. This open-source software toolkit supports the decoding of spatial correlates or content-triggered experimental manipulation in closed-loop neuroscience experiments. Graphical Abstract Highlights • Spike-sorting-free decoding reconstructs the rat's position with ultrafast speed • GPU-powered population decoding significantly speeds up multi-core CPU-based system • GPU computing empowers real-time assessment of decoded "memory replay" candidates • Open-source software toolkit supports closed-loop content-triggered intervention The hippocampal and neocortical neuronal ensembles encode rich spatial information in navigation. Hu et al. develop computational techniques that accommodate real-time decoding and assessment of large-scale unsorted neural ensemble place codes during running behavior and sleep. [ABSTRACT FROM AUTHOR]

Objectives Phantom limb pain is neuropathic pain after amputation of a limb and partial or complete deafferentation such as brachial plexus root avulsion. The underlying cause of this pain has been attributed to maladaptive plasticity of the sensorimotor cortex. It has been suggested that experimental reorganization would affect pain, especially if it results in functional restoration. We tested the hypothesis that restoration of hand motor function using a brain–machine interface (BMI) based on magnetoencephalographic (MEG) signals will normalize maladapted cortical representation and relieve pain. Methods This study included 10 phantom limb patients (9 brachial plexus root avulsion and 1 amputee). MEG signals during movements of the phantom hand or intact hand were used to train the decoder inferring movements of each hand. The robotic hand was controlled by the decoder. Patients controlled the robotic hand by moving the phantom hand. The training effects were compared among trainings with the phantom decoder, real hand decoder, and random decoder in a randomized cross-over trial. Results BMI training with the phantom decoder increased the decoding accuracy of phantom hand movements and pain. In contrast, BMI training with the intact hand decoder reduced accuracy and pain. Discussion It was suggested that BMI training to modulate the motor representation of phantom hand controlled pain. The sensorimotor cortical plasticity might induce pain. Conclusions and Significance Phantom limb pain was controlled by BMI training to induce sensorimotor cortical plasticity. [ABSTRACT FROM AUTHOR]