Search

Seeing a speaking face can influence observers’ auditory perception of syllables [McGurk and McDonald, Nature 264, 746–748 (1976)]. This effect decreases when the speaker’s face is inverted [e.g., Green, 3014 (1994)]. Face recognition is also inhibited with inverted faces [e.g., Rock, Sci. Am. 230, 78–85 (1974)] suggesting a similar underlying process. To further explore the link between face and audiovisual speech perception, a speech experiment was designed to replicate another face perception effect. In this effect, an inverted face and an inverted face containing upright lips are perceived as looking normal, but an upright face with inverted lips looks grotesque [Thompson, Perception 9, 438–484 (1980)]. An audiovisual speech experiment tested four presentation conditions: Upright face‐upright mouth, upright face‐inverted mouth, inverted face‐inverted mouth, inverted face‐upright mouth. Various discrepant audiovisual syllables were tested in each condition. Visual influences occurred in all but the upr...

The influence of the spatial position of the acoustic signal in audiovisual speech perception was investigated in a series of experiments using the McGurk effect. Subjects viewed video disk recordings of faces producing visual VCV nonsense syllables while simultaneous acoustic VCV stimuli were presented from one of 7 different speakers locations. The speakers were positioned in a semicircular array in front of the subject. In separate studies subjects were required to name the intervocalic consonant, indicate the location of the sound source using a computerized pentiometer system, or perform both tasks. Preliminary results suggest that the subjects’ ability to localize the position of the auditory stimulus was influenced by the presence of a visual stimulus. Specifically, subjects tended to localize the sound closer to the position of the monitor showing the visual stimulus. However the strength of the McGurk effect was not influenced by the spatial position of the sound source. Subjects perceived the visual /g/ and the auditory /b/ combination as /d/ equally often at all sound locations. The independence of the ventriloquist effect and audiovisual integration in the perception of the McGurk effect will be discussed in terms of spatial constraints on the cross‐modal perception of speech.

The development of audiovisual speech perception was examined in this experiment using the McGurk paradigm (McGurk and MacDonald, 1976), in which a visual recording of a person saying a particular syllable is synchronized with the auditory presentation of another syllable. Previous audiovisual speech studies have shown that adult perception is strongly influenced by the visual speech information whereas the perception of young children (5–8 years) shows a very weak influence of visual speech patterns and a strong bias favoring the auditory speech information. In this investigation 46 children in four age groups (5, 7, 9, and 11 year olds) and 15 adults were presented with conflicting audiovisual syllables in which an auditory /ba/ sequence was combined with visual /va/, /θa/, /da/, and /ga/ sequences, respectively. The results indicated that the influence of auditory information decreased with increasing age, while the influence of visual information and the integration of auditory and visual information ...

Oral speech intelligibility tests were conducted with, and without, supplementary visual observation of the speaker's facial and lip movements. The difference between these two conditions was examined as a function of the speech‐to‐noise ratio and of the size of the vocabulary under test. The visual contribution to oral speech intelligibility (relative to its possible contribution) is, to a first approximation, independent of the speech‐to‐noise ratio under test. However, since there is a much greater opportunity for the visual contribution at low speech‐to‐noise ratios, its absolute contribution can be exploited most profitably under these conditions.

Explores perceived audiovisual synchrony in reverberant environments.The temporal smear caused by reverberation can alter a signal's acoustic signature.Reverberating acoustics is a common problem in teleconferencing systems.Reverberation may not have adverse effects on the perceived synchrony for continuous audiovisual speech.The temporal integration of speech syllables and isolated events is affected by the acoustic phenomenon. With teleconferencing becoming more accessible as a communication platform, researchers are working to understand the consequences of the interaction between human perception and this unfamiliar environment. Given the enclosed space of a teleconference room, along with the physical separation between the user, microphone and speakers, the transmitted audio often becomes mixed with the reverberating auditory components from the room. As a result, the audio can be perceived as smeared in time, and this can affect the user experience and perceived quality. Moreover, other challenges remain to be solved. For instance, during encoding, compression and transmission, the audio and video streams are typically treated separately. Consequently, the signals are rarely perfectly aligned and synchronous. In effect, timing affects both reverberation and audiovisual synchrony, and the two challenges may well be inter-dependent. This study explores the temporal integration of audiovisual continuous speech and speech syllables, along with a non-speech event, across a range of asynchrony levels for different reverberation conditions. Non-reverberant stimuli are compared to stimuli with added reverberation recordings. Findings reveal that reverberation does not affect the temporal integration of continuous speech. However, reverberation influences the temporal integration of the isolated speech syllables and the action-oriented event, with perceived subjective synchrony skewed towards audio lead asynchrony and away from the more common audio lag direction. Furthermore, less time is spent on simultaneity judgements for the longer sequences when the temporal offsets get longer and when reverberation is introduced, suggesting that both asynchrony and reverberation add to the demands of the task.

Publisher Summary This chapter discusses about assessing audio-visual speech reception disability. The importance of the role of vision in understanding speech is attested informally by the observation of many older hearing-impaired listeners that they can hear better when they wear their glasses, and by the strategy of most listeners in noisy environments of looking at the face of the talker whose speech they wish to understand. Vision complements audition by providing, in particular, those phonetic details that are lost easily in noise or impairment. The Four-alternative Auditory Disability and Speech-reading test (FADAST) was developed in response to two major needs. First, it would be used as a member of a battery of tests in a population-based survey of deafness. In this role it would contribute to establishing the profile of speech-reception disability in the adult population of Great Britain and relating it to other measures of disability, impairment, and handicap. Secondly, it would be used to assess the benefit obtained from different aids by more impaired adults where the aid is most appropriately construed as an aid to lip-reading. The requirements of the test were that it should: (1) possess sufficient face-validity to ensure the co-operation and motivation of the subject; (2) be straightforward to administer and receive; (3) be relatively free from practice effects; (4) permit a rigorous, informative scoring procedure; (5) be free from phonological dialect variation, particularly between the radically different vowel systems of English as spoken in Scotland and England; and (6) be sensitive to a wide range of disability.

This psychophysics study used musicians as a model to investigate whether musical expertise shapes the temporal integration window for audiovisual speech, sinewave speech, or music. Musicians and non-musicians judged the audiovisual synchrony of speech, sinewave analogs of speech, and music stimuli at 13 audiovisual stimulus onset asynchronies (±360, ±300 ±240, ±180, ±120, ±60, and 0 ms). Further, we manipulated the duration of the stimuli by presenting sentences/melodies or syllables/tones. Critically, musicians relative to non-musicians exhibited significantly narrower temporal integration windows for both music and sinewave speech. Further, the temporal integration window for music decreased with the amount of music practice, but not with age of acquisition. In other words, the more musicians practiced piano in the past 3 years, the more sensitive they became to the temporal misalignment of visual and auditory signals. Collectively, our findings demonstrate that music practicing fine-tunes the audiovisual temporal integration window to various extents depending on the stimulus class. While the effect of piano practicing was most pronounced for music, it also generalized to other stimulus classes such as sinewave speech and to a marginally significant degree to natural speech.

It has been demonstrated that children with specific language impairment (SLI) show difficulties not only with auditory but also with audiovisual speech perception. The goal of this study was to assess whether children with SLI might show reduced attention to the talker's mouth compared to their typically developing (TD) peers. An additional aim was to determine whether the pattern of attention to a talking face would be related to a specific subtype of SLI. We used an eye-tracker methodology and presented a video of a talker speaking the children's native language. Results revealed that children with SLI paid significantly less attention to the mouth than the TD children. More specifically, it was also observed that children with a phonological-syntactic deficit looked less to the mouth as compared to the children with a lexical-syntactic deficit.

There are two main behavioral expressions of multisensory integration (MSI) in speech; the perceptual enhancement produced by the sight of the congruent lip movements of the speaker, and the illusory sound perceived when a speech syllable is dubbed with incongruent lip movements, in the McGurk effect. These two models have been used very often to study MSI. Here, we contend that, unlike congruent audiovisually (AV) speech, the McGurk effect involves brain areas related to conflict detection and resolution. To test this hypothesis, we used fMRI to measure blood oxygen level dependent responses to AV speech syllables. We analyzed brain activity as a function of the nature of the stimuli—McGurk or non‐McGurk—and the perceptual outcome regarding MSI—integrated or not integrated response—in a 2 × 2 factorial design. The results showed that, regardless of perceptual outcome, AV mismatch activated general‐purpose conflict areas (e.g., anterior cingulate cortex) as well as specific AV speech conflict areas (e.g., inferior frontal gyrus), compared with AV matching stimuli. Moreover, these conflict areas showed stronger activation on trials where the McGurk illusion was perceived compared with non‐illusory trials, despite the stimuli where physically identical. We conclude that the AV incongruence in McGurk stimuli triggers the activation of conflict processing areas and that the process of resolving the cross‐modal conflict is critical for the McGurk illusion to arise. Hum Brain Mapp 38:5691–5705, 2017. This research was supported by the Ministerio de Economía y Competitividad (PSI2016–75558‐PAEI/FEDER), AGAUR Generalitat de Catalunya (2014SGR856 and 2012BE100392), and the European Research Council (StG‐2010 263145). The Neuroimaging Laboratory of the Fondazione Santa Lucia is supported by the Italian Ministry of Health. EM is supported by the program “Investissements d'Avenir” (ANR‐11‐IDEX‐0007) and by the program “BQR Accueil EC 16” of University Claude Bernard Lyon 1.

Speech perception is audiovisual, as demonstrated by the McGurk effect in which discrepant visual speech alters the auditory speech percept. We studied the role of visual attention in audiovisual speech perception by measuring the McGurk effect in two conditions. In the baseline condition, attention was focused on the talking face. In the distracted attention condition, subjects ignored the face and attended to a visual distractor, which was a leaf moving across the face. The McGurk effect was weaker in the latter condition, indicating that visual attention modulated audiovisual speech perception. This modulation may occur at an early, unisensory processing stage, or it may be due to changes at the stage where auditory and visual information is integrated. We investigated this issue by conventional statistical testing, and by fitting the Fuzzy Logical Model of Perception (Massaro, 1998) to the results. The two methods suggested different interpretations, revealing a paradox in the current methods of analysis.

Audiovisual identification of sentences was measured as a function of audio delay in untrained listeners with normal hearing; the sound track was replaced by rectangular pulses originally synchronized to the closing of the talker's vocal folds and then subjected to delay. Although group‐mean performance declined monotonically with delay, systematic decrements occurred only when delay exceeded 80 ms. A similar tolerance of delay was found in judgments of audiovisual onset time when observers determined whether a 120‐Hz triangular wave started before or after the opening of a pair of liplike Lissajou figures. Group‐mean 70% DLs were − 78 ms (sound leading) and + 137 ms (sound lagging). This result suggests, first, that most observers possess insufficient sensitivity to intermodal timing cues in audiovisual speech for them to be used analogously to VOT in auditory speech perception, and, second, that the effects found in the first experiment derive from syllabic rather than phonemic interference. However, th...