Search

The present fMRI study aimed at highlighting patterns of brain activations and autonomic activity when confronted with high mental workload and the threat of auditory stressors. Twenty participants performed a complex cognitive task in either safe or aversive conditions. Our results showed that increased mental workload induced recruitment of the lateral frontoparietal executive control network (ECN), along with disengagement of medial prefrontal and posterior cingulate regions of the default mode network (DMN). Mental workload also elicited an increase in heart rate and pupil diameter. Task performance did not decrease under the threat of stressors, most likely due to efficient inhibition of auditory regions, as reflected by a large decrement of activity in the superior temporal gyri. The threat of stressors was also accompanied with deactivations of limbic regions of the salience network (SN), possibly reflecting emotional regulation mechanisms through control from dorsal medial prefrontal and parietal regions, as indicated by functional connectivity analyses. Meanwhile, the threat of stressors induced enhanced ECN activity, likely for improved attentional and cognitive processes toward the task, as suggested by increased lateral prefrontal and parietal activations. These fMRI results suggest that measuring the balance between ECN, SN, and DMN recruitment could be used for objective mental state assessment. In this sense, an extra recruitment of task‐related regions and a high ratio of lateral versus medial prefrontal activity may represent a relevant marker of increased but efficient mental effort, while the opposite may indicate a disengagement from the task due to mental overload and/or stressors., We highlighted patterns of brain activations and autonomic activity when confronted with high mental workload and auditory stressors. Mental workload induced recruitment of the fronto‐parietal executive control network (ECN), along with disengagement of the default mode network (DMN). Task performance did not decrease under stressors, most likely due to efficient emotional regulation mechanisms through control from dorsal brain areas.

Mental overload and mental fatigue are two degraded cognitive states that are known to promote cognitive incapacitation. We adopted a neuroergonomics approach to investigate these states that remain difficult to induce under laboratory settings thus impeding their measurement. Two experiments were conducted under real flight conditions to respectively measure the electrophysiological correlates of mental fatigue and mental overload with a 32 channel-dry EEG system. Our findings revealed that the occurrence of mental fatigue was related to higher theta and alpha band power. Mental overload was associated with higher beta band power over frontal sites. We performed single trial classification to detect mental fatigue and over-load states. Classification accuracy reached 76.9% and 89.1%, respectively, in discriminating mental fatigue vs. no fatigue and mental overload vs. low-high load. These preliminary results provide evidence for the feasibility of detecting neural correlates of cognitive fatigue and load during real flight conditions and provide promising perspectives on the implementation of neuroadaptive technology especially in the context of single pilot-operation.

The field of enhancing skill acquisition, particularly in professions necessitating the mastery of a complex combination of physical and mental abilities, is rapidly progressing and amenable to novel training protocols involving both neuroimaging and neurostimulation. Aircraft piloting in particular is an ideal medium for testing new training protocols, because objective performance measures are well-understood and modern flight simulator programs are realistic and high-fidelity. Here, we describe the development of a flight simulator protocol that allows for the analysis of neurostimulation-enhanced skill acquisition both within and between subjects. A three-block design was created to collect data pre-training, during feedback training, and post-training while being recorded in an fMRI. The dual task consists of 30–45 s trials landing a plane on one of two runways, indicated by an arrow displayed on the simulator screen, while simultaneously responding to auditory stimuli played constantly during each trial with button presses. The landing task is presented at two difficulty levels in pseudorandom balanced order, modulated by wind speed and direction. Two auditory conditions, response and control (no response), are used for a two by two design. For the feedback training, subjects are provided with relevant measures of how well they are able to land on the specified runway as well as their accuracy in the auditory task. Subjects will be randomly assigned to tDCS stimulation or sham groups, with stim receiving 30 min of 1.5 mA high definition-tDCS to the right ventrolateral prefrontal cortex during the training block. Altogether, this novel combination of stimulation, neuroimaging, and dual-task training will allow for an in-depth, multi-factor analysis of cognitive workload, behavioral performance, neurostimulation effects, and learning of a complex mental and physical task.

Inattentional deafness is the failure to hear otherwise audible sounds (usually alarms) that may occur under high workload conditions. One potential cause for its occurrence could be an attentional bottleneck that occurs when task demands are high, resulting in lack of resources for processing of additional tasks. In this fMRI experiment, we explore the brain regions active during the occurrence of inattentional deafness using a difficult perceptual-motor task in which the participants fly through a simulated Red Bull air race course and at the same time push a button on the joystick to the presence of audio alarms. Participants were instructed to focus on the difficult piloting task and to press the button on the joystick quickly when they noticed an audio alarm. The fMRI results revealed that audio misses relative to hits had significantly greater activity in the right inferior frontal gyrus IFG and the superior medial frontal cortex. Consistent with an attentional bottleneck, activity in these regions was also present for poor flying performance (contrast of gates missed versus gates passed for the flying task). A psychophysiological interaction analysis from the IFG identified reduced effective connectivity to auditory processing regions in the right superior temporal gyrus for missed audio alarms relative to audio alarms that were heard. This study identifies a neural signature of inattentional deafness in an ecologically valid situation by directly measuring differences in brain activity and effective connectivity between audio alarms that were not heard compared to those that were heard. Hum Brain Mapp 38:5440-5455, 2017. © 2017 Wiley Periodicals, Inc.

This study expores neural activity underlying creative processes through the investigation of music improvisation. Fourteen guitar players with a high level of improvisation skill participated in this experiment. The experimental task involved playing 32-s alternating blocks of improvisation and scales on guitar. electroencephalography (EEG) data was measured continuously throughout the experiment. In order to remove potential artifacts and extract brain-related activity the following signal processing techniques were employed: bandpass filtering, Artifact Subspace Reconstruction, and Independent Component Analysis (ICA). For each participant, artifact related independent components (ICs) were removed from the EEG data and only ICs found to be from brain activity were retained. Source localization using this brain-related activity was carried out using sLORETA. Greater activity for improvisation over scale was found in multiple frequency bands (theta, alpha, and beta) localized primarily in the medial frontal cortex (MFC), Middle frontal gyrus (MFG), anterior cingulate, polar medial prefrontal cortex (MPFC), premotor cortex (PMC), pre and postcentral gyrus (PreCG and PostCG), superior temporal gyrus (STG), inferior parietal lobule (IPL), and the temporal-parietal junction. Together this collection of brain regions suggests that improvisation was mediated by processes involved in coordinating planned sequences of movement that are modulated in response to ongoing environmental context through monitoring and feedback of sensory states in relation to internal plans and goals. Machine-learning using Common Spatial Patterns (CSP) for EEG feature extraction attained a mean of over 75% classification performance for improvisation vs. scale conditions across participants. These machine-learning results are a step towards the development of a brain-computer interface that could be used for neurofeedback training to improve creativity.

A Neuroergonomics Approach to Human Performance in Aviation

International audience; Accident analyses have revealed that pilots can fail to process auditory stimuli such as alarms, a phenomenon known as inattentional deafness. The motivation of this research is to develop a passive brain computer interface that can predict the occurence of this critical phenomenon during real flight conditions. Ten volunteers, equipped with a dry-EEG system, had to fly a challenging flight scenario while responding to auditory alarms by button press. The behavioral results disclosed that the pilots missed 36% of the auditory alarms. ERP analyses confirm that this phenomenon affects auditory processing at an early (N100) and late (P300) stages as the consequence of a potential attentional bottleneck mechanism. Intersubject classification was carried out over frequency features extracted three second epochs before the alarms' onset using sparse representation for classification (SRC), sparse and dense representation (SDR) and more conventional approach such as linear discriminant analysis (LDA), shrinkage LDA and nearest neighbor (1NN). In the best case, SRC and SDR gave respectively a performance of 66.9% and 65.4% of correct mean classification rate to predict the occurrence of inattentional deafness, outperforming LDA (60.6%), sLDA (60%) and 1 NN (59.6%). These results open promising perspectives for the implementation of neuroadaptive automation with as ultimate goal to enhance alarm stimulation delivery so that it is perceived and acted upon.

Noninvasive brain stimulation (NIBS) techniques such as transcranial magnetic stimulation (TMS) and transcranial direct current stimulation (tDCS) provide both a means to investigate neural function and treatment for various pathologies. These techniques also afford the possibility to determine and induce causal influences between stimulated-brain networks and behavior. In this chapter we will review the processes by which TMS and tDCS modulate neural activity with a focus on facilitation or disruption of perceptual, motor, and cognitive functions. An understanding of the various stimulation parameters underlying these NIBS techniques and how they modulate neural processes to enhance performance are discussed to provide insight to potential neuroergonomic applications that can be implemented in real-world situations.

This chapter reviews recent research combining transcranial direct current stimulation (tDCS) and functional magnetic resonance imaging (fMRI) to investigate the neural processes modulated by tDCS that are related to induced enhancement in behavioral performance. Both the effects during tDCS (“active-effects”) and those immediately following tDCS (“after-effects”) on brain activity and connectivity recorded by fMRI are discussed. Modulation of task-related brain activity and resting-state brain activity and connectivity by tDCS is dependent on many factors, including the nature of the task and whether learning has occurred. Experiments using more complex tasks need to be conducted to see how well these results generalize to real-world situations.

Neuroergonomics is the study of how the brain functions in real-world situations with the goal of developing technology to enhance human performance. Neuroergonomics constitutes a paradigm shift away from the standard reductionist approach to neuroscience. The neuroergonomic approach maintains that an understanding of neural processes underlying human behavior can best be understood by investigating the underlying interacting brain networks in the context of carrying out various real-world tasks under investigation, rather than under reduced isolated conditions that only occur in the laboratory. In this chapter we discuss why aerospace cerebral experimental sciences (ACES) is an ideal paradigm to implement this neuroergonomic approach. By using a combination of high resolution and lower resolution portable brain imaging techniques as well as non-invasive brain stimulation the goal of ACES is to determine brain processes underlying complex behavior during aviation and space operations such that neuroergonomic technology can be developed to improve human performance.

The inability to detect auditory alarms is a critical issue in many do- mains such as aviation. An interesting prospect for flight safety is to understand the neural mechanisms underpinning auditory alarm misperception under actual flight condition. We conducted an experiment in which four pilots were to re- spond by button press when they heard an auditory alarm. The 64 channel Cognionics dry-wireless EEG system was used to measure brain activity in a 4 seat light aircraft. An instructor was present on all flights and in charge of initi- ating the various scenarios to induce two levels of task engagement (simple navigation task vs. complex maneuvering task). Our experiment revealed that inattentional deafness to single auditory alarms could take place as the pilots missed a mean number of 12.5 alarms occurring mostly during the complex maneuvering condition, when the EEG engagement index was high.

We designed an aviation pursuit task wherein subjects followed a flying target plane at different speeds and distances, and were tasked to train crosshairs on the target plane by maneuvering their own plane, as well as detect an auditory cue at perceptual threshold. The aim of this project was to collect multimodal neuroimaging data in the form of magnetoencephalography (MEG), functional magnetic resonance imaging (fMRI), and mobile electroenecephalography (mEEG) during such a task, including on a six-degree-of-freedom motion platform. Here we present initial behavioral results to better understand perception–action coupling in the brain, as mediated by neural oscillations and cross-frequency interactions. In so doing, we hope to inform the future design of brain–computer interfaces and adaptive flight systems that respond to the cognitive states of the user/pilot.

International audience; Individuals often have reduced ability to hear alarms in real world situations (e.g., anesthesia monitoring, flying airplanes) when attention is focused on another task, sometimes with devastating consequences. This phenomenon is called inattentional deafness and usually occurs under critical high workload conditions. It is difficult to simulate the critical nature of these tasks in the laboratory. In this study, dry electroencephalography is used to investigate inattentional deafness in real flight while piloting an airplane. The pilots participating in the experiment responded to audio alarms while experiencing critical high workload situations. It was found that missed relative to detected alarms were marked by reduced stimulus evoked phase synchrony in theta and alpha frequencies (6–14 Hz) from 120 to 230 ms poststimulus onset. Correlation of alarm detection performance with intertrial coherence measures of neural phase synchrony showed different frequency and time ranges for detected and missed alarms. These results are consistent with selective attentional processes actively disrupting oscillatory coherence in sensory networks not involved with the primary task (piloting in this case) under critical high load conditions. This hypothesis is corroborated by analyses of flight parameters showing greater maneuvering associated with difficult phases of flight occurring during missed alarms. Our results suggest modulation of neural oscillation is a general mechanism of attention utilizing enhancement of phase synchrony to sharpen alarm perception during successful divided attention, and disruption of phase synchrony in brain networks when attentional demands of the primary task are great, such as in the case of inattentional deafness.

In spatial perception, visual information has higher acuity than auditory information and we often misperceive sound-source locations when spatially disparate visual stimuli are presented simultaneously. Ventriloquists make good use of this auditory illusion. In this study, we investigated neural substrates of the ventriloquism effect to understand the neural mechanism of multimodal integration. This study was performed in 2 steps. First, we investigated how sound locations were represented in the auditory cortex. Secondly, we investigated how simultaneous presentation of spatially disparate visual stimuli affects neural processing of sound locations. Based on the population rate code hypothesis that assumes monotonic sensitivity to sound azimuth across populations of broadly tuned neurons, we expected a monotonic increase of blood oxygenation level-dependent (BOLD) signals for more contralateral sounds. Consistent with this hypothesis, we found that BOLD signals in the posterior superior temporal gyrus increased monotonically as a function of sound azimuth. We also observed attenuation of the monotonic azimuthal sensitivity by spatially disparate visual stimuli. The alteration of the neural pattern was considered to reflect the neural mechanism of the ventriloquism effect. Our findings indicate that conflicting audiovisual spatial information of an event is associated with an attenuation of neural processing of auditory spatial localization.

Inattentional deafness is the failure to hear otherwise audible sounds (usually alarms) that may occur under high workload conditions. One potential cause for its occurrence could be an attentional bottleneck that occurs when task demands are high, resulting in lack of resources for processing of additional tasks. In this fMRI experiment, we explore the brain regions active during the occurrence of inattentional deafness using a difficult perceptual‐motor task in which the participants fly through a simulated Red Bull air race course and at the same time push a button on the joystick to the presence of audio alarms. Participants were instructed to focus on the difficult piloting task and to press the button on the joystick quickly when they noticed an audio alarm. The fMRI results revealed that audio misses relative to hits had significantly greater activity in the right inferior frontal gyrus IFG and the superior medial frontal cortex. Consistent with an attentional bottleneck, activity in these regions was also present for poor flying performance (contrast of gates missed versus gates passed for the flying task). A psychophysiological interaction analysis from the IFG identified reduced effective connectivity to auditory processing regions in the right superior temporal gyrus for missed audio alarms relative to audio alarms that were heard. This study identifies a neural signature of inattentional deafness in an ecologically valid situation by directly measuring differences in brain activity and effective connectivity between audio alarms that were not heard compared to those that were heard. Hum Brain Mapp 38:5440–5455, 2017. © 2017 Wiley Periodicals, Inc.

In this fMRI study we investigate neural processes related to the action observation network using a complex perceptual-motor task in pilots and non-pilots. The task involved landing a glider (using aileron, elevator, rudder, and dive brake) as close to a target as possible, passively observing a replay of one's own previous trial, passively observing a replay of an expert's trial, and a baseline do nothing condition. The objective of this study is to investigate two types of motor simulation processes used during observation of action: imitation based motor simulation and error-feedback based motor simulation. It has been proposed that the computational neurocircuitry of the cortex is well suited for unsupervised imitation based learning, whereas, the cerebellum is well suited for error-feedback based learning. Consistent with predictions, pilots (to a greater extent than non-pilots) showed significant differential activity when observing an expert landing the glider in brain regions involved with imitation based motor simulation (including premotor cortex PMC, inferior frontal gyrus IFG, anterior insula, parietal cortex, superior temporal gyrus, and middle temporal MT area) than when observing one's own previous trial which showed significant differential activity in the cerebellum (only for pilots) thought to be concerned with error-feedback based motor simulation. While there was some differential brain activity for pilots in regions involved with both Execution and Observation of the flying task (potential Mirror System sites including IFG, PMC, superior parietal lobule) the majority was adjacent to these areas (Observation Only Sites) (predominantly in PMC, IFG, and inferior parietal loblule). These regions showing greater activity for observation than for action may be involved with processes related to motor-based representational transforms that are not necessary when actually carrying out the task.

Transcranial direct current stimulation (tDCS) has been shown to enhance cognitive performance on a variety of tasks. It is hypothesized that tDCS enhances performance by affecting task related cortical excitability changes in networks underlying or connected to the site of stimulation facilitating long term potentiation. However, many recent studies have called into question the reliability and efficacy of tDCS to induce modulatory changes in brain activity. In this study, our goal is to investigate the individual differences in tDCS induced modulatory effects on brain activity related to the degree of enhancement in performance, providing insight into this lack of reliability. In accomplishing this goal, we used functional magnetic resonance imaging (fMRI) concurrently with tDCS stimulation (1 mA, 30 minutes duration) using a visual search task simulating real world conditions. The experiment consisted of three fMRI sessions: pre-training (no performance feedback), training (performance feedback which included response accuracy and target location and either real tDCS or sham stimulation given), and post-training (no performance feedback). The right posterior parietal cortex was selected as the site of anodal tDCS based on its known role in visual search and spatial attention processing. Our results identified a region in the right precentral gyrus, known to be involved with visual spatial attention and orienting, that showed tDCS induced task related changes in cortical excitability that were associated with individual differences in improved performance. This same region showed greater activity during the training session for target feedback of incorrect (target-error feedback) over correct trials for the tDCS stim over sham group indicating greater attention to target features during training feedback when trials were incorrect. These results give important insight into the nature of neural excitability induced by tDCS as it relates to variability in individual differences in improved performance shedding some light the apparent lack of reliability found in tDCS research.

Neural correlates of driving and of decision making have been investigated separately, but little is known about the underlying neural mechanisms of decision making in driving. Previous research discusses two types of decision making: reward‐weighted decision making and cost‐weighted decision making. There are many reward‐weighted decision making neuroimaging studies but there are few cost‐weighted studies. Considering that driving involves serious risk, it is assumed that decision making in driving is cost weighted. Therefore, neural substrates of cost‐weighted decision making can be assessed by investigation of driver's decision making. In this study, neural correlates of resolving uncertainty in driver's decision making were investigated. Turning right in left‐hand traffic at a signalized intersection was simulated by computer graphic animation based videos. When the driver's view was occluded by a big truck, the uncertainty of the oncoming traffic was resolved by an in‐car video assist system that presented the driver's occluded view. Resolving the uncertainty reduced activity in a distributed area including the amygdala and anterior cingulate. These results implicate the amygdala and anterior cingulate as serving a role in cost‐weighted decision making. Hum Brain Mapp 2009. © 2008 Wiley‐Liss, Inc.

Recent evidence from neuroscience indicates that the anticipation of external rewards may enhance declarative memory consolidation by increasing dopaminergic-modulated plasticity in the hippo- campus. A number of studies in psychology, however, have shown that external rewards may have null, or even negative, effects on learning. To shed light on this issue, we developed a novel task, in which native Japanese speakers were rewarded to learn unknown English words inside a functional MRI scan- ner. Rewards had no effect on recall performance unless we used a rating of reward-induced anxiety as a covariate. In this case, for highly rewarded words, we found a negative correlation between recall per- formance and anxiety ratings. For those words, high recall performance and low anxiety ratings were associated with enhanced activity in the midbrain dopaminergic centers, the hippocampus, and the amyg- dala. On the other hand, low recall performance and high anxiety ratings were associated with enhanced activity in the anterior cingulate and middle frontal gyrus, brain regions that have been shown to be involved with anxiety and divided attention, respectively. A connectivity analysis indicated positive func- tional connectivity between the midbrain dopaminergic centers and both the hippocampus and the amyg- dala, as well as negative connectivity between the anterior cingulate and the amygdala. Thus, both our be- havioral and imaging results suggest that the anticipation of rewards can, depending on the individual level of reward-induced anxiety, have either a beneficial effect or a negative effect on word learning. Hum Brain Mapp 29:237-249, 2008. V C 2007 Wiley-Liss, Inc.

This commentary builds on a companion article in which Kim et al compare brain activation in elite, expert, and novice archers during a simulated target aiming task (Kim et al. 2014. Cogn Behav Neurol. 27:173-182). With the archery study as our starting point, we address 4 neural processes that may be responsible in general for elite athletes' superior performance over experts and novices: neural efficiency, cortical expansion, specialized processes, and internal models. In Kim et al's study, the elite archers' brains showed more activity in the supplementary motor area and the cerebellum than those of the novices and experts, and showed minimal widespread activity, especially in frontal areas involved with executive control. Kim et al's results are consistent with the idea of specialized neural processes that help coordinate motor planning and control. As athletes become more skilled, these processes may mediate the reduction in widespread activity in regions mapping executive control, and may produce a shift toward more automated processing. Kim et al's finding that activity in the cerebellum rose with increasing skill is consistent both with expansion of the finger representational area in the cerebellum and with internal models that simulate how archers manipulate the bow and arrow when aiming. Kim et al prepare the way for testing of neuromodulation techniques to improve athletic performance, refine highly technical job skills, and rehabilitate patients.

Neuronal operations associated with the top-down control process of shifting attention from one locus to another involve a network of cortical regions, and their influence is deemed fundamental to visual perception. However, the extent and nature of these operations within primary visual areas are unknown. In this paper, we used magnetoencephalography (MEG) in combination with magnetic resonance imaging (MRI) to determine whether, prior to the onset of a visual stimulus, neuronal activity within early visual cortex is affected by covert attentional shifts. Time/frequency analyses were used to identify the nature of this activity. Our results show that shifting attention towards an expected visual target results in a late-onset (600 ms postcue onset) depression of alpha activity which persists until the appearance of the target. Independent component analysis (ICA) and dipolar source modeling confirmed that the neuronal changes we observed originated from within the calcarine cortex. Our results further show that the amplitude changes in alpha activity were induced not evoked (i.e., not phase-locked to the cued attentional task). We argue that the decrease in alpha prior to the onset of the target may serve to prime the early visual cortex for incoming sensory information. We conclude that attentional shifts affect activity within the human calcarine cortex by altering the amplitude of spontaneous alpha rhythms and that subsequent modulation of visual input with attentional engagement follows as a consequence of these localized changes in oscillatory activity. © 2005 Elsevier B.V. All rights reserved.

Left fusiform gyrus and left angular gyrus are considered to be respectively involved with visual form processing and associating visual and auditory (phonological) information in reading. However, there are a number of studies that fail to show the contribution of these regions in carrying out these aspects of reading. Considerable differences in the type of stimuli and tasks used in the various studies may account for the discrepancy in results. This functional magnetic resonance imaging (fMRI) study attempts to control aspects of experimental stimuli and tasks to specifically investigate brain regions involved with visual form processing and character-to-phonological (i.e., simple grapheme-to-phonological) conversion processing for single letters. Subjects performed a two-back identification task using known Japanese, and previously unknown Korean, and Thai phonograms before and after training on one of the unknown language orthographies. Japanese subjects learned either five Korean or five Thai phonograms. Brain regions related to visual form processing were assessed by comparing activity related to native (Japanese) phonograms with that of non-native (Korean and Thai) phonograms. There was no significant differential brain activity for visual form processing. Brain regions related to character-to-phonological conversion processing were assessed by comparing pre- and post-tests of trained non-native phonograms with that of native phonograms and non-trained non-native phonograms. Significant differential activation post-relative to pre-training exclusively for the trained non-native phonograms was found in left angular gyrus. In addition, psychophysiologic interaction (PPI) analysis revealed greater integration of left angular gyrus with primary visual cortex as well as with superior temporal gyrus for the trained phonograms post-relative to pre-training. The results suggest that left angular gyrus is involved with character-to-phonological conversion in letter perception.

Perception of speech is improved when presentation of the audio signal is accompanied by concordant visual speech gesture information. This enhancement is most prevalent when the audio signal is degraded. One potential means by which the brain affords perceptual enhancement is thought to be through the integration of concordant information from multiple sensory channels in a common site of convergence, multisensory integration (MSI) sites. Some studies have identified potential sites in the superior temporal gyrus/sulcus (STG/S) that are responsive to multisensory information from the auditory speech signal and visual speech movement. One limitation of these studies is that they do not control for activity resulting from attentional modulation cued by such things as visual information signaling the onsets and offsets of the acoustic speech signal, as well as activity resulting from MSI of properties of the auditory speech signal with aspects of gross visual motion that are not specific to place of articulation information. This fMRI experiment uses spatial wavelet bandpass filtered Japanese sentences presented with background multispeaker audio noise to discern brain activity reflecting MSI induced by auditory and visual correspondence of place of articulation information that controls for activity resulting from the above-mentioned factors. The experiment consists of a low-frequency (LF) filtered condition containing gross visual motion of the lips, jaw, and head without specific place of articulation information, a midfrequency (MF) filtered condition containing place of articulation information, and an unfiltered (UF) condition. Sites of MSI selectively induced by auditory and visual correspondence of place of articulation information were determined by the presence of activity for both the MF and UF conditions relative to the LF condition. Based on these criteria, sites of MSI were found predominantly in the left middle temporal gyrus (MTG), and the left STG/S (including the auditory cortex). By controlling for additional factors that could also induce greater activity resulting from visual motion information, this study identifies potential MSI sites that we believe are involved with improved speech perception intelligibility.

This fMRI study explores brain regions involved with perceptual enhancement afforded by observation of visual speech gesture information. Subjects passively identified words presented in the following conditions: audio-only, audiovisual, audio-only with noise, audiovisual with noise, and visual only. The brain may use concordant audio and visual information to enhance perception by integrating the information in a converging multisensory site. Consistent with response properties of multisensory integration sites, enhanced activity in middle and superior temporal gyrus/sulcus was greatest when concordant audiovisual stimuli were presented with acoustic noise. Activity found in brain regions involved with planning and execution of speech production in response to visual speech presented with degraded or absent auditory stimulation, is consistent with the use of an additional pathway through which speech perception is facilitated by a process of internally simulating the intended speech act of the observed speaker.

The effects of attentional modulation on activity within the human visual cortex were investigated using magnetoencephalography. Chromatic sinusoidal stimuli were used to evoke activity from the occipital cortex, with attention directed either toward or away from the stimulus using a bar-orientation judgment task. For five observers, global magnetic field power was plotted as a function of time from stimulus onset. The major peak of each function occurred at about 120 ms latency and was well modeled by a current dipole near the calcarine sulcus. Independent component analysis (ICA) on the non-averaged data for each observer also revealed one component of calcarine origin, the location of which matched that of the dipolar source determined from the averaged data. For two observers, ICA revealed a second component near the parieto-occipital sulcus. Although no effects of attention were evident using standard averaging procedures, time-varying spectral analyses of single trials revealed that the main effect of attention was to alter the level of oscillatory activity. Most notably, a sustained increase in alpha-band (7-12 Hz) activity of both calcarine and parieto-occipital origin was evident. In addition, calcarine activity in the range of 13-21 Hz was enhanced, while calcarine activity in the range of 5-6 Hz was reduced. Our results are consistent with the hypothesis that attentional modulation affects neural processing within the calcarine and parieto-occipital cortex by altering the amplitude of alpha-band activity and other natural brain rhythms. © 2003 Elsevier Inc. All rights reserved.

fMRI was used to assess the relationship between brain activation and the degree of audiovisual integration of speech information during a phoneme categorization task. Twelve subjects heard a speaker say the syllable /aba/ paired either with video of the speaker saying the same consonant or a different one (/ava/). In order to manipulate the degree of audiovisual integration, the audio was either synchronous or +/- 400 ms out of phase with the visual stimulus. Subjects reported whether they heard the consonant /b/ or another consonant; fewer /b/ responses when the audio and visual stimuli were mismatched indicated higher levels of visual influence on speech perception (McGurk effect). Active brain regions during presentation of the incongruent stimuli included the superior temporal and inferior frontal gyrus, as well as extrastriate, premotor and posterior parietal cortex. A regression analysis related the strength of the McGurk effect to levels of brain activation. Paradoxically, higher numbers of /b/ responses were positively correlated with activation in the left occipito-temporal junction, an area often associated with processing visual motion. This activation suggests that auditory information modulates visual processing to affect perception.

In this single-sweep electroencephalographic case study, independent component analysis (ICA) was used to investigate multimodal processes underlying the enhancement of speech intelligibility in noise (for monosyllabic English words) by visualizing facial motion concordant with the audio speech signal. Wavelet analysis of the single-sweep IC activation waveforms revealed increased high-frequency energy for two ICs underlying the visual enhancement effect. For one IC, current source density analysis localized activity mainly to the superior temporal gyrus, consistent with principles of multimodal integration. For the other IC, activity was distributed across multiple cortical areas perhaps reflecting global mappings underlying the visual enhancement effect.

This single-sweep electroencephalographic study using independent component analysis was conducted to determine the neural processes underlying both speech perception and production of vowels. The same neural processes located in auditory and motor areas of the brain that significantly distinguish between a speech production and a control mental rehearsal task were found for both auditory evoked responses and speech planning responses. Thus identifying common task dependent neural processes underlying speech production and perception.