Search

Although it is clear that sighted listeners use both auditory and visual cues during speech perception, the manner in which multisensory information is combined is a matter of active debate. One approach to measuring multisensory integration is to use variants of the McGurk illusion, in which discrepant auditory and visual cues produce auditory percepts that differ from those based solely on unimodal input. Not all listeners show the same degree of susceptibility to the McGurk illusion, and these individual differences in susceptibility are frequently used as a measure of audiovisual integration ability. However, despite their popularity, we argue that McGurk tasks are ill-suited for studying the kind of multisensory speech perception that occurs in real life: McGurk stimuli are often based on isolated syllables (which are rare in conversations) and necessarily rely on audiovisual incongruence that does not occur naturally. Furthermore, recent data show that susceptibility on McGurk tasks does not correlate with performance during natural audiovisual speech perception. Although the McGurk effect is a fascinating illusion, truly understanding the combined use of auditory and visual information during speech perception requires tasks that more closely resemble everyday communication: namely, words, sentences, and stories with congruent auditory and visual speech cues.

The use of face masks recently has raised awareness about their effects on speech comprehension. In a series of two studies, we assess how different masks speech intelligibility and listening effort. In the first study, we compared four mask types in three noise levels for young and older adult listeners. Stimuli were presented audiovisually. Results showed that, in quiet, speech intelligibility was not greatly affected by any of the masks relative to no-mask conditions. However, in background noise, all masks reduced intelligibility. Masks with transparent windows and cloth masks with filters resulted in the lowest intelligibility scores, while surgical face masks had the least effect. Participants’ subjective effort ratings also reflect that comprehending speech through face masks requires greater effort. Overall, older adults’ listening effort ratings were higher and their intelligibility scores were lower. In the second study, we assess listening effort with pupillometry. We present the same stimuli to a group of young adults in six audio-only conditions: no-mask in quiet, no-mask in moderate noise, and four masks in moderate noise. In addition to intelligibility accuracy and pupil dilation, we also collect subjective effort and performance ratings, pure tone averages, and working memory performance scores.

As listeners age, speech recognition often becomes more challenging, especially in noisy environments. Older adults also appear to have more difficulty resolving lexical competition than do younger adults. Relatively little is known, however, about how such signal- and lexicon-related challenges interact to affect the timing of word recognition and how such effects might change across the lifespan. In this study, we use a visual world paradigm to investigate the effects noise, word frequency, and phonological neighborhood density on the time course of spoken word recognition in young and older adults. On each trial, listeners saw four items on a computer screen and heard “Click on the” followed by a target word that corresponded to one of the images. None of the other items on the display were phonologically or semantically related to the target. Eye-tracking results show that the time course of word recognition is predicted by age, noise, frequency, density, and several of their interactions. In general, high-frequency words and words from dense phonological neighborhoods were fixated more quickly, while noise and age slowed recognition. These results support a framework in which lexical and acoustic factors co-determine the cognitive challenges associated with speech perception.As listeners age, speech recognition often becomes more challenging, especially in noisy environments. Older adults also appear to have more difficulty resolving lexical competition than do younger adults. Relatively little is known, however, about how such signal- and lexicon-related challenges interact to affect the timing of word recognition and how such effects might change across the lifespan. In this study, we use a visual world paradigm to investigate the effects noise, word frequency, and phonological neighborhood density on the time course of spoken word recognition in young and older adults. On each trial, listeners saw four items on a computer screen and heard “Click on the” followed by a target word that corresponded to one of the images. None of the other items on the display were phonologically or semantically related to the target. Eye-tracking results show that the time course of word recognition is predicted by age, noise, frequency, density, and several of their interactions. In general,...

The use of online testing in the behavioral sciences is increasing due to the potential for lower cost and faster completion than traditional in-person laboratory testing. Online testing presents a special challenge for speech research due to the variety of listeners’ acoustic environments. For example, listeners may use headphones or speakers of various types, complete the task in different levels of background noise, or vary in their hearing abilities. Here we presented spoken sentences in speech-shaped noise to participants from the United States recruited online using Amazon Mechanical Turk at SNRs of -2, -5, and -8 dB (n≥50 per SNR). We compared these online results to normative data collected using these same sentences in the lab (SNR of -5 dB; n = 30). Preliminary results suggest a reasonable correspondence between average intelligibility scores for individual sentences presented using the two methods (Spearman rho = 0.79). Standard deviations suggest greater variability in the online responses (average SD across sentences = 1.24) relative to the lab responses (average SD = 0.86). We conclude that despite increased variability, online testing can provide reasonable ratings of intelligibility and in many circumstances is a feasible method to supplement or replace laboratory testing.

The acoustic-phonetic properties of speech signals affect not only their intelligibility, but also how well they are encoded in memory. Recognition memory can be improved, for example, when speakers intentionally speak clearly (Van Engen et al., 2012). One explanation for this result is that enhanced acoustic-phonetic cues reduce the cognitive effort associated with perceptual speech processing, thereby increasing the availability of processing resources for encoding speech content in memory. In the current study, we tested the hypothesis that, following the same logic, recognition memory would be reduced for foreign-accented speech, in which acoustic-phonetic cues deviate from native-language norms. Participants heard English sentences produced by a native speaker of English and a native speaker of Korean. Recognition memory was tested using an old-new judgment task in which participants heard those recordings again, along with an equal number of new items. Surprisingly, recognition memory was higher for Korean-accented English than for native-accented English. A follow-up study in which test sentences were presented visually showed no difference between foreign- and native-accented speech. These results are consistent with an acoustic distinctiveness account in which foreign-accented speech can in some cases be remembered more accurately due to its distinct acoustic features.

With as few as 10–20 sentences of exposure, listeners are able to adapt to speech that is highly distorted compared to that which is encountered in everyday conversation. The current study examines the extent to which adaptation to time-compressed speech can be impeded by disrupting the continuity of the exposure sentences, and whether this differs between young and older adult listeners when they are equated for starting accuracy. In separate sessions conducted one week apart, the degree of adaptation was assessed in four exposure conditions, all of which involved exposure to the same number of time-compressed sentences. A continuous exposure condition involved presentation of the time-compressed sentences without interruption. Two alternation conditions alternated time-compressed speech and uncompressed speech by single sentences or groups of four sentences. A fourth condition presented sentences that were separated by a period of silence but no uncompressed speech. For all conditions, neither young nor older adults’ overall level of learning was influenced by disruptions to the exposure sentences. In addition, participants’ performance showed reliable improvement across the first and subsequent sessions. These results support robust learning mechanisms in speech perception that remain functional throughout the lifespan.

Our perceptual experience is formed by combining incoming sensory information with prior knowledge and expectation. When speech is not fully intelligible, non-acoustic information may be particularly important. Predictions about this degraded acoustic signal can be provided intrinsically (if the speech is still partially intelligible) or extrinsically (for example, by presenting a written cue). I will discuss results from studies in which the neural response to speech was measured using magnetoencephalography (MEG), with speech clarity parametrically manipulated using noise vocoding. In one study, we found that during sentence processing the phase of ongoing cortical oscillations is matched to that of the acoustic speech envelope in the range of the syllable rate (4–8 Hz). Critically, this phase-locking was enhanced in left temporal cortex when speech is intelligible. In a separate study of single word listening, accurate predictions provided by written text enhanced subjective clarity and changed the response in early auditory processing regions of temporal cortex. Both experiments thus highlight neural responses in brain regions associated with relatively low-level speech perception. Together these findings support the ability of linguistic information to provide predictions that shape auditory processing of spoken language, particularly when acoustic clarity is compromised.