Search

This article offers a collection of observations that highlight the value of time course data in pupillometry and points out ways in which these observations create deeper understanding of listening effort. The main message is that listening effort should be considered on a moment-to-moment basis rather than as a singular amount. A review of various studies and the reanalysis of data reveal distinct signatures of effort before a stimulus, during a stimulus, in the moments after a stimulus, and changes over whole experimental testing sessions. Collectively these observations motivate questions that extend beyond the “amount” of effort, toward understanding how long the effort lasts, and how precisely someone can allocate effort at specific points in time or reduce effort at other times. Apparent disagreements between studies are reconsidered as informative lessons about stimulus selection and the nature of pupil dilation as a reflection of decision making rather than the difficulty of sensory encoding.

Purpose: This letter to the editor discusses recommendations and publicity from a recent article by Gallena and Pinto (2021) that appeared in Perspectives of the ASHA Special Interest Groups. We contextualize the recommendations made by Gallena and Pinto that young women actively suppress their use of vocal fry, a normal part of linguistic and sociolinguistic variation, to make themselves more marketable to employers who discriminate on the basis of vocal fry use. By reviewing research on vocal fry, social evaluation, and linguistic discrimination, we show how this recommendation is fundamentally flawed and how it perpetuates sexist tropes about language use. We argue that this is particularly dismaying when publicized by a journal of the American Speech-Language-Hearing Association, which aims to promote the universal use of language, in all its forms, as a human right. Conclusions: People stigmatize linguistic differences to devalue those who have less power. Vocal fry is not a problem; the problem is absurd sexist judgments against those who use vocal fry. The recommendations made in Gallena and Pinto (2021) and Perspectives' publicity about those recommendations both contribute to sexist linguistic discrimination. Rather than recommending that victims of sexism change their behavior to suit the biased views of others, we should use our energy to eradicate the underlying sexism.

The papers in this special issue provide a critical look at some historical ideas that have had an influence on research and teaching in the field of speech communication. They also address widely used methodologies or address long-standing methodological challenges in the areas of speech perception and speech production. The goal is to reconsider and evaluate the need for caution or replacement of historical ideas with more modern results and methods. The contributions provide respectful historical context to the classic ideas, as well as new original research or discussion that clarifies the limitations of the original ideas.

This paper examines some commonly used stimuli in speech perception experiments and raises questions about their use, or about the interpretations of previous results. The takeaway messages are: 1) the Hillenbrand vowels represent a particular dialect rather than a gold standard, and English vowels contain spectral dynamics that have been largely underappreciated, 2) the /ɑ/ context is very common but not clearly superior as a context for testing consonant perception, 3) /ɑ/ is particularly problematic when testing voice-onset-time perception because it introduces strong confounds in the formant transitions, 4) /dɑ/ is grossly overrepresented in neurophysiological studies and yet is insufficient as a generalized proxy for “speech perception,” and 5) digit tests and matrix sentences including the coordinate response measure are systematically insensitive to important patterns in speech perception. Each of these stimulus sets and concepts is described with careful attention to their unique value and also cases where they might be misunderstood or over-interpreted.

The spectral ripple discrimination task is a psychophysical measure that has been found to correlate with speech recognition in listeners with cochlear implants (CIs). However, at ripple densities above a critical value (around 2 RPO, but device-specific), the sparse spectral sampling of CI processors results in stimulus distortions resulting in aliasing and unintended changes in modulation depth. As a result, spectral ripple thresholds above a certain number are not ordered monotonically along the RPO dimension and thus cannot be considered better or worse spectral resolution than each other, thus undermining correlation measurements. These stimulus distortions are not remediated by changing stimulus phase, indicating these issues cannot be solved by spectro-temporally modulated stimuli. Speech generally has very low-density spectral modulations, leading to questions about the mechanism of correlation between high ripple thresholds and speech recognition. Existing data showing correlations between ripple discrimination and speech recognition include many observations above the aliasing limit. These scores should be treated with caution, and experimenters could benefit by prospectively considering the limitations of the spectral ripple test.

Purpose: An extra moment after a sentence is spoken may be important for listeners with hearing loss to mentally repair misperceptions during listening. The current audiologic test battery cannot distinguish between a listener who repaired a misperception versus a listener who heard the speech accurately with no need for repair. This study aims to develop a behavioral method to identify individuals who are at risk for relying on a quiet moment after a sentence. Method: Forty-three individuals with hearing loss (32 cochlear implant users, 11 hearing aid users) heard sentences that were followed by either 2 s of silence or 2 s of babble noise. Both high- and low-context sentences were used in the task. Results: Some individuals showed notable benefit in accuracy scores (particularly for high-context sentences) when given an extra moment of silent time following the sentence. This benefit was highly variable across individuals and sometimes absent altogether. However, the group-level patterns of results were mainly explained by the use of context and successful perception of the words preceding sentence-final words. Conclusions: These results suggest that some but not all individuals improve their speech recognition score by relying on a quiet moment after a sentence, and that this fragility of speech recognition cannot be assessed using one isolated utterance at a time. Reliance on a quiet moment to repair perceptions would potentially impede the perception of an upcoming utterance, making continuous communication in real-world scenarios difficult especially for individuals with hearing loss. The methods used in this study—along with some simple modifications if necessary—could potentially identify patients with hearing loss who retroactively repair mistakes by using clinically feasible methods that can ultimately lead to better patient-centered hearing health care. Supplemental Material: https://doi.org/10.23641/asha.21644801

Perception of word stress is an important aspect of recognizing speech, guiding the listener toward candidate words based on the perceived stress pattern. Cochlear implant (CI) signal processing is likely to disrupt some of the available cues for word stress, particularly vowel quality and pitch contour changes. In this study, we used a cue weighting paradigm to investigate differences in stress cue weighting patterns between participants listening with CIs and those with normal hearing (NH). We found that participants with CIs gave less weight to frequency-based pitch and vowel quality cues than NH listeners but compensated by upweighting vowel duration and intensity cues. Nonetheless, CI listeners' stress judgments were also significantly influenced by vowel quality and pitch, and they modulated their usage of these cues depending on the specific word pair in a manner similar to NH participants. In a series of separate online experiments with NH listeners, we simulated aspects of bimodal hearing by combining low-pass filtered speech with a vocoded signal. In these conditions, participants upweighted pitch and vowel quality cues relative to a fully vocoded control condition, suggesting that bimodal listening holds promise for restoring the stress cue weighting patterns exhibited by listeners with NH.

Voice onset time (VOT) is an acoustic property of stop consonants that is commonly manipulated in studies of phonetic perception. This paper contains a thorough description of the "progressive cutback and replacement" method of VOT manipulation, and comparison with other VOT manipulation techniques. Other acoustic properties that covary with VOT-such as fundamental frequency and formant transitions-are also discussed, along with considerations for testing VOT perception and its relationship to various other measures of auditory temporal or spectral processing. An implementation of the progressive cutback and replacement method in the Praat scripting language is presented, which is suitable for modifying natural speech for perceptual experiments involving VOT and/or related covarying F0 and intensity cues. Justifications are provided for the stimulus design choices and constraints implemented in the script.

Listening effort is a commonly reported difficulty among those who have hearing loss. This project aims to operationalize the concept of effort precision, as the ability to engage or disengage mental resources at strategic times could be an important signature of a person’s capacity to guard against wasted effort. Stimuli were designed to simulate the situation of seeking clarification of a misperceived word or attending to specific information in a sentence. Some stimuli included verbatim repetitions (which would ideally elicit reduced effort) and others featured sentences with a masked word followed by a clear repetition (which would ideally evoke effort aimed only at the newly unmasked word). Pupillometry was used as an index of moment-to-moment changes in listening effort. Data showed pupil dilations linked in time with the unmasked word, enabling precise measurements of the efficiency of effort. Listeners with normal hearing displayed the ability to plan and exert effort at specific times while also reducing effort in situations where it was unnecessary. Conversely, listeners who wear cochlear implants generally did not display these efficient effort characteristics. These results highlight the need to expand the concept of listening effort beyond a “more” or “less” framework, toward a framework of efficiency.

Taylor Swift has changed geographic environments, speaking communities, and singing communities throughout her career, inviting question into changes in her voice acoustics. Capitalizing on the availability of interviews and multiple song recordings, we tracked changes in vowel formant trajectories and other vocal properties from different points in her life to see how her voice might have been influenced by these changes in community (e.g switching from country to pop style, moving from Philadelphia to Nashville). Comparing vowel space from 2012 to 2019, we observe higher tongue position (lower frequency F1) for numerous vowels, especially for /eɪ/ and /aʊ/. Rounding for /ɔ/ was reduced, and /æ/ was pronounced farther back in the mouth. We also observed reduced distinction between the varieties of /ɑɪ/ before voiced and voiceless consonants (e.g., ride versus write), as well as greater front-back movement for /u/. Studio re-recordings show remarkable consistency of word timing and intensity. Conversely, Swift’s live singing voice includes some variation from the recording, including fewer silences between words. Live versions feature extended duration for words that are emphasized for emotion, and/or ends of phrases, compensated by shorter durations for other segments so that the overall song length was very consistent.

Individuals with hearing loss are more likely to miss a word during speech perception, but can use context to retroactively repair the missing word. When sentences need to be repaired, listening effort remains elevated even after the entire sentence is heard; suggesting the repair process continues beyond the stimulus presentation. Importantly, if a listener is stuck processing a previous sentence, they may miss an upcoming utterance, making everyday conversation difficult. By measuring the amount of time that is needed to repair a missing word, we will better understand if processing would interfere with the perception of a subsequent sentence. Individuals with normal hearing and hearing loss heard sentences that did or did not require retroactive repair while performing a secondary case-judgement task. Listeners decided whether a visually presented letter was upper- or lower-case either during the sentence, 0.5 s after, or 2 s after the sentence. Preliminary results indicate that reaction times were faster for intact sentences and decrease (improve) as more time has passed (as the missing word has been resolved) for both sentence types. Our results suggest that the timeline of the repair process can be understood using behavioral measures that complement intelligibility scores.

Semantic context in sentences has been shown to increase intelligibility and reduce listening effort. However, context has typically been tested in situations where it is used to predict upcoming words rather than repair earlier words. The current study attempts to directly compare predictive versus retroactive context for the first time for the same listeners in the same study. We hypothesized an increase in effort for retroactive context compared to the elevated effort of predictive context. Cochlear implant and Normal Hearing listeners heard sentences where a single content word was masked by noise, either early in the sentence (requiring retroactive repair) or later in the sentence (where it was predicted by earlier words). Changes in pupil dilation were measured as an index of listening effort linked with the moment of each word and its context. Effort increases when context is used to repair an earlier missing word, even if intelligibility is perfect. There is a smaller increase of effort when the missing word follows context that enabled prediction of the word. This study provides a novel approach in directly comparing the mental cost of using context to repair versus predict misperceived words in a sentence.

Cochlear implants (CIs) are the most successful sensory implant in history, restoring the sensation of sound to thousands of persons who have severe to profound hearing loss. Implants do not recreate acoustic sound as most of us know it, but they instead convey a rough representation of the temporal envelope of signals. This sparse signal, derived from the envelopes of narrowband frequency filters, is sufficient for enabling speech understanding in quiet environments for those who lose hearing as adults and is enough for most children to develop spoken language skills. The variability between users is huge, however, and is only partially understood. CIs provide acoustic information that is sufficient for the recognition of some aspects of spoken language, especially information that can be conveyed by temporal patterns, such as syllable timing, consonant voicing, and manner of articulation. They are insufficient for conveying pitch cues and separating speech from noise. There is a great need for improving our understanding of functional outcomes of CI success beyond measuring percent correct for word and sentence recognitions. Moreover, greater understanding of the variability experienced by children, especially children and families from various social and cultural backgrounds, is of paramount importance. Future developments will no doubt expand the use of this remarkable device.

Listening effort is a valuable and important notion to measure because it is among the primary complaints of people with hearing loss. It is tempting and intuitive to accept speech intelligibility scores as a proxy for listening effort, but this link is likely oversimplified and lacks actionable explanatory power. This study was conducted to explain the mechanisms of listening effort that are not captured by intelligibility scores, using sentence-repetition tasks where specific kinds of mistakes were prospectively planned or analyzed retrospectively. Effort measured as changes in pupil size among 20 listeners with normal hearing and 19 listeners with cochlear implants. Experiment 1 demonstrates that mental correction of misperceived words increases effort even when responses are correct. Experiment 2 shows that for incorrect responses, listening effort is not a function of the proportion of words correct but is rather driven by the types of errors, position of errors within a sentence, and the need to resolve ambiguity, reflecting how easily the listener can make sense of a perception. A simple taxonomy of error types is provided that is both intuitive and consistent with data from these two experiments. The diversity of errors in these experiments implies that speech perception tasks can be designed prospectively to elicit the mistakes that are more closely linked with effort. Although mental corrective action and number of mistakes can scale together in many experiments, it is possible to dissociate them to advance toward a more explanatory (rather than correlational) account of listening effort.

When hearing an ambiguous speech sound, listeners show a tendency to perceive it as a phoneme that would complete a real word, rather than completing a nonsense/fake word. For example, a sound that could be heard as either /b/ or /ɡ/ is perceived as /b/ when followed by _ack but perceived as /ɡ/ when followed by “_ap.” Because the target sound is acoustically identical across both environments, this effect demonstrates the influence of top-down lexical processing in speech perception. Degradations in the auditory signal were hypothesized to render speech stimuli more ambiguous, and therefore promote increased lexical bias. Stimuli included three speech continua that varied by spectral cues of varying speeds, including stop formant transitions (fast), fricative spectra (medium), and vowel formants (slow). Stimuli were presented to listeners with cochlear implants (CIs), and also to listeners with normal hearing with clear spectral quality, or with varying amounts of spectral degradation using a noise vocoder. Results indicated an increased lexical bias effect with degraded speech and for CI listeners, for whom the effect size was related to segment duration. This method can probe an individual's reliance on top-down processing even at the level of simple lexical/phonetic perception.When hearing an ambiguous speech sound, listeners show a tendency to perceive it as a phoneme that would complete a real word, rather than completing a nonsense/fake word. For example, a sound that could be heard as either /b/ or /ɡ/ is perceived as /b/ when followed by _ack but perceived as /ɡ/ when followed by “_ap.” Because the target sound is acoustically identical across both environments, this effect demonstrates the influence of top-down lexical processing in speech perception. Degradations in the auditory signal were hypothesized to render speech stimuli more ambiguous, and therefore promote increased lexical bias. Stimuli included three speech continua that varied by spectral cues of varying speeds, including stop formant transitions (fast), fricative spectra (medium), and vowel formants (slow). Stimuli were presented to listeners with cochlear implants (CIs), and also to listeners with normal hearing with clear spectral quality, or with varying amounts of spectral degradation using a noise vocod...

Accurate perception of binaural cues is essential for left-right sound localization. Much literature focuses on threshold measures of perceptual acuity and accuracy. This study focused on supra-threshold perception using an anticipatory eye movement (AEM) paradigm designed to capture subtle aspects of perception that might not emerge in behavioral-motor responses, such as the accumulation of certainty, and rapid revisions in decision-making. Participants heard interaural timing differences (ITDs) or interaural level differences in correlated or uncorrelated narrowband noises, respectively. A cartoon ball moved behind an occluder and then emerged from the left or right side, consistent with the binaural cue. Participants anticipated the correct answer (before it appeared) by looking where the ball would emerge. Results showed quicker and more steadfast gaze fixations for stimuli with larger cue magnitudes. More difficult stimuli elicited a wider distribution of saccade times and greater number of corrective saccades before final judgment, implying perceptual uncertainty or competition. Cue levels above threshold elicited some wrong-way saccades that were quickly corrected. Saccades to ITDs were earlier and more reliable for low-frequency noises. The AEM paradigm reveals the time course of uncertainty and changes in perceptual decision-making for supra-threshold binaural stimuli even when behavioral responses are consistently correct.

OBJECTIVES: Cochlear implant (CI) recipients are at a severe disadvantage compared to normal hearing listeners in distinguishing consonants that differ by place of articulation (PoA) because the key relevant spectral differences are degraded by the implant. One component of that degradation is the upward shifting of spectral energy that occurs with a shallow insertion depth of a CI. The present study aimed to systematically measure the effects of spectral shifting on word recognition and phoneme categorization by specifically controlling the amount of shifting and using stimuli whose identification specifically depends on perceiving frequency cues. We hypothesized that listeners would be biased toward perceiving phonemes that contain higher-frequency components because of the upward frequency shift, and that intelligibility would decrease as spectral shifting increased. DESIGN: Normal-hearing listeners (n = 15) heard sinewave-vocoded speech with simulated upward frequency shifts of 0, 2, 4, and 6 mm of cochlear space to simulate shallow CI insertion depth. Stimuli included monosyllabic words and /b/-/d/ and /ʃ/-/s/ continua that varied systematically by formant frequency transitions or frication noise spectral peaks, respectively. Recalibration to spectral shifting was operationally defined as shifting perceptual acoustic-phonetic mapping commensurate with the spectral shift. In other words, adjusting frequency expectations for both phonemes upward so that there is still a perceptual distinction, rather than hearing all upward-shifted phonemes as the higher-frequency member of the pair. RESULTS: For moderate amounts of spectral shifting, group data suggested a general “halfway” recalibration to spectral shifting, but individual data suggested a notably different conclusion: half of the listeners were able to recalibrate fully, while the other halve of the listeners were utterly unable to categorize shifted speech with any reliability. There were no participants who demonstrated a pattern intermediate to these two extremes. Intelligibility of words decreased with greater amounts of spectral shifting, also showing loose clusters of better- and poorer-performing listeners. Phonetic analysis of word errors revealed certain cues were more susceptible to being compromised due to a frequency shift (place and manner of articulation) while voicing was robust to spectral shifting. CONCLUSION: Shifting the frequency spectrum of speech has systematic effects that are in line with known properties of speech acoustics, but the ensuing difficulties cannot be predicted based on tonotopic mismatch alone. Difficulties are subject to substantial individual differences in the capacity to adjust acoustic-phonetic mapping. These results help to explain why speech recognition in CI listeners cannot be fully predicted by peripheral factors like electrode placement and spectral resolution; even among listeners with functionally equivalent auditory input, there is an additional factor of simply being able or unable to flexibly adjust acoustic-phonetic mapping. This individual variability could motivate precise treatment approaches guided by an individual’s relative reliance on wideband frequency representation (even if it is mismatched), or limited frequency coverage whose tonotopy is preserved.

Prosody is used to mark important information in speech yet is not an integral part of speech recognition testing among people with hearing difficulty. While a listener may correctly perceive words spoken, they may not perceive meaningful emphasis on a certain word, which could be critically important in a variety of everyday situations. This study introduces a new paradigm for assessing perception of prosodic cues in listeners with normal hearing and with cochlear implants, who have notorious difficulty perceiving pitch. Stimuli consisted of spoken sentences where one word (in various sentence positions) was emphasized in a manner that indicated that a prior statement was incorrect in a specific way. Participants used a visual analog scale to mark the timing and degree of emphasis aligned with the target words. Here, the perceptual data are linked with acoustic measures of voice pitch contour, intensity, duration, and vocal quality to characterize how contrastive stress cues are recovered by listeners with and without hearing impairment.

Among the common acoustic differences between the voices of women and men are the spectral characteristics of fricatives. Akin to pitch and formant frequencies, the frequency characteristics of fricatives are lower in men than in women. These fricative characteristics are learned social cues that help to express the talker’s gender. Listeners reliably incorporate these differences into perceptual judgments by categorizing the same sound differently depending on what they perceive the gender of the talker to be; a fricative that is intermediate to “s” and “sh” will be labeled “s” when thought to be spoken by a man, but “sh” if by a woman. However, the acoustic descriptions and the perceptual consequences of these gender-related acoustic differences have been explored almost exclusively at syllable-onset position. Here we use both word-onset and -offset positions in both perception and production tasks. The perceptual accommodation of talker gender is weaker when the fricative is at syllable offset, inviting questions about whether the extent of gender-related changes in acoustics are simply commensurate with salience of word position. We will present both the perception and production data in an attempt to unify both phenomena in relation to each other.

Chronic exposure to noise is a well-documented health hazard (Suter, 1991). In this study, we investigated whether an individuals’ self-reported noise history (exposure, sensitivity, and annoyance) correlated with noise-elicited physiologic arousal. We measured pupil dilation responses and loudness judgments to varying types of noise and related those to self-identified noise history. We hypothesized that increased history of exposure or annoyance would be correlated with perceived loudness and arousal from noise. Five colors (spectra) of noise were generated with the same sound pressure levels within the band 1000–4000 Hz but with different slopes of spectral energy below and above the band. Changes in participant pupil dilation responses elicited by each noise type were used as an assessment of physiological stress response to noise stimuli. Participants also adjusted the loudness of two randomized noise spectra until they were determined to be of equal loudness. Preliminary data suggest that the perception of loudness and levels of arousal differed depending on the color spectrum of noise. Relationships between the loudness judgments, self-reported noise history, and observed physiologic data will be further discussed. Results have implications for understanding the behavioral and physiological effects of chronic exposure to noise. [Robert Young, NIDCD R01DC017114 (Winn).]

Most standard speech perception tests use unrelated sentences where the topic changes for each stimulus. This unrealistic paradigm is expedient but might overlook an important aspect of speech processing and effort, which is how listeners use expectations and topic awareness to enable easier repair of misperceived words. We hypothesize that the effort of mentally repairing misperceived words will be reduced when individuals have advance topic awareness of upcoming sentences. In this study, individuals with normal hearing and hearing loss heard sentences with a masked target word that required retroactive mental repair using later context. Sentences were either preceded by the presence or absence of a related priming word presented visually to simulate advance topic knowledge. Listening effort for these sentences was measured by changes in pupil size that were linked with the exact timing of the word being repaired. When the target word is masked in a sentence, pupil dilation increases immediately after the missing word, and that dilation is modulated by the helpfulness of the prime before the sentence.

Supplemental Digital Content is available in the text., Objectives: Slowed speaking rate was examined for its effects on speech intelligibility, its interaction with the benefit of contextual cues, and the impact of these factors on listening effort in adults with cochlear implants. Design: Participants (n = 21 cochlear implant users) heard high- and low-context sentences that were played at the original speaking rate, as well as a slowed (1.4× duration) speaking rate, using uniform pitch-synchronous time warping. In addition to intelligibility measures, changes in pupil dilation were measured as a time-varying index of processing load or listening effort. Slope of pupil size recovery to baseline after the sentence was used as an index of resolution of perceptual ambiguity. Results: Speech intelligibility was better for high-context compared to low-context sentences and slightly better for slower compared to original-rate speech. Speech rate did not affect magnitude and latency of peak pupil dilation relative to sentence offset. However, baseline pupil size recovered more substantially for slower-rate sentences, suggesting easier processing in the moment after the sentence was over. The effect of slowing speech rate was comparable to changing a sentence from low context to high context. The effect of context on pupil dilation was not observed until after the sentence was over, and one of two analyses suggested that context had greater beneficial effects on listening effort when the speaking rate was slower. These patterns maintained even at perfect sentence intelligibility, suggesting that correct speech repetition does not guarantee efficient or effortless processing. With slower speaking rates, there was less variability in pupil dilation slopes following the sentence, implying mitigation of some of the difficulties shown by individual listeners who would otherwise demonstrate prolonged effort after a sentence is heard. Conclusions: Slowed speaking rate provides release from listening effort when hearing an utterance, particularly relieving effort that would have lingered after a sentence is over. Context arguably provides even more release from listening effort when speaking rate is slower. The pattern of prolonged pupil dilation for faster speech is consistent with increased need to mentally correct errors, although that exact interpretation cannot be verified with intelligibility data alone or with pupil data alone. A pattern of needing to dwell on a sentence to disambiguate misperceptions likely contributes to difficulty in running conversation where there are few opportunities to pause and resolve recently heard utterances.

Eye-tracking is widely used throughout the scientific community, from vision science and psycholinguistics to marketing and human-computer interaction. Surprisingly, there is little consistency and transparency in preprocessing steps, making replicability and reproducibility difficult. To increase replicability, reproducibility, and transparency, a package in R (a free and widely used statistical programming environment) called gazeR was created to read and preprocess two types of data: gaze position and pupil size. For gaze position data, gazeR has functions for reading in raw eye-tracking data, formatting it for analysis, converting from gaze coordinates to areas of interest, and binning and aggregating data. For data from pupillometry studies, the gazeR package has functions for reading in and merging multiple raw pupil data files, removing observations with too much missing data, eliminating artifacts, blink identification and interpolation, subtractive baseline correction, and binning and aggregating data. The package is open-source and freely available for download and installation: https://github.com/dmirman/gazer . We provide step-by-step analyses of data from two tasks exemplifying the package's capabilities.

Listeners must accommodate acoustic differences between vocal tracts and speaking styles of conversation partners—a process called normalization or accommodation. This study explores what acoustic cues are used to make this perceptual adjustment by listeners with normal hearing or with cochlear implants, when the acoustic variability is related to the talker's gender. A continuum between /ʃ/ and /s/ was paired with naturally spoken vocalic contexts that were parametrically manipulated to vary by numerous cues for talker gender including fundamental frequency (F0), vocal tract length (formant spacing), and direct spectral contrast with the fricative. The goal was to examine relative contributions of these cues toward the tendency to have a lower-frequency acoustic boundary for fricatives spoken by men (found in numerous previous studies). Normal hearing listeners relied primarily on formant spacing and much less on F0. The CI listeners were individually variable, with the F0 cue emerging as the strongest cue on average.

Purpose Weighting of acoustic cues for perceiving place-of-articulation speech contrasts was measured to determine the separate and interactive effects of age and use of cochlear implants (CIs). It has been found that adults with normal hearing (NH) show reliance on fine-grained spectral information (e.g., formants), whereas adults with CIs show reliance on broad spectral shape (e.g., spectral tilt). In question was whether children with NH and CIs would demonstrate the same patterns as adults, or show differences based on ongoing maturation of hearing and phonetic skills. Method Children and adults with NH and with CIs categorized a /b/–/d/ speech contrast based on two orthogonal spectral cues. Among CI users, phonetic cue weights were compared to vowel identification scores and Spectral-Temporally Modulated Ripple Test thresholds. Results NH children and adults both relied relatively more on the fine-grained formant cue and less on the broad spectral tilt cue compared to participants with CIs. However, early-implanted children with CIs better utilized the formant cue compared to adult CI users. Formant cue weights correlated with CI participants' vowel recognition and in children, also related to Spectral-Temporally Modulated Ripple Test thresholds. Adults and child CI users with very poor phonetic perception showed additive use of the two cues, whereas those with better and/or more mature cue usage showed a prioritized trading relationship, akin to NH listeners. Conclusions Age group and hearing modality can influence phonetic cue-weighting patterns. Results suggest that simple nonlexical categorization tests correlate with more general speech recognition skills of children and adults with CIs.

For cochlear-implant users with near-normal contralateral hearing, a mismatch between the frequency-to-place mapping in the two ears could produce a suboptimal performance. This study assesses tonotopic matches via binaural interactions. Dynamic interaural time-difference sensitivity was measured using bandpass-filtered pulse trains at different rates in the acoustic and implanted ear, creating binaural envelope beats. Sensitivity to beats should peak when the same tonotopic region is stimulated in both ears. All nine participants detected dynamic interaural timing differences and demonstrated some frequency selectivity. This method provides a guide to frequency-to-place mapping without compensation for inherent latency differences between the acoustic and implanted ears.

Speech perception requires accommodation of a wide range of acoustic variability across talkers. A classic example is the perception of "sh" and "s" fricative sounds, which are categorized according to spectral details of the consonant itself, and also by the context of the voice producing it. Because women's and men's voices occupy different frequency ranges, a listener is required to make a corresponding adjustment of acoustic-phonetic category space for these phonemes when hearing different talkers. This pattern is commonplace in everyday speech communication, and yet might not be captured in accuracy scores for whole words, especially when word lists are spoken by a single talker. Phonetic accommodation for fricatives "s" and "sh" was measured in 20 cochlear implant (CI) users and in a variety of vocoder simulations, including those with noise carriers with and without peak picking, simulated spread of excitation, and pulsatile carriers. CI listeners showed strong phonetic accommodation as a group. Each vocoder produced phonetic accommodation except the 8-channel noise vocoder, despite its historically good match with CI users in word intelligibility. Phonetic accommodation is largely independent of linguistic factors and thus might offer information complementary to speech intelligibility tests which are partially affected by language processing.

Accurate perception of word stress is key to segmenting and understanding speech. We examined perception of four acoustic cues to word stress in English—vowel quality (VQ), pitch, syllable duration, and intensity—and how perception of these cues is affected by specific kinds of signal degradation. Cochlear implants (CIs) severely degrade VQ and pitch, which could demand different relative reliance on these cues. We tested NH listeners in unprocessed, vocoded, and simulated bimodal (unprocessed low frequencies, vocoded high frequencies) conditions as well as CI users. Stimuli included stress-contrastive word pairs (DESert versus dessERT, SUBject versus subJECT) with orthogonal manipulations of each cue. For both NH and CI listeners, there were strikingly different patterns of cue weighting between these word pairs, highlighting how stress cue weighting is flexible across words rather than being absolute. Vocoding the stimuli led to a downweighting of VQ and pitch cues and a compensatory upweighting of duration and intensity; perception of pitch and VQ were largely restored in the bimodal conditions with 250 and 800 Hz bands preserved, respectively. CI users relied on duration more than the NH controls in any condition, while also showing a surprising ability to use VQ and pitch cues.

Speech perception is more effortful when the listener needs to fill in a missing word that was masked or misperceived. However, that effort could come at the cost of hearing the next utterance, as conversational speech is continuous. Speech perception is typically tested one utterance at a time, critically overlooking this problem. The current study measured how effort is deployed in the moments of processing one sentence while balancing the need to continue attending or not attending to later speech. Stimuli included sentences where occasionally noise masked a single word that was recoverable from later context. Sentences were followed by silence or three random digits; participants were tested in blocks where they either ignored the digits or repeated them along with the full sentence. Effort was tracked via pupillometry. Listeners with normal hearing front-loaded effort when anticipating subsequent inattention, and spread effort more shallowly across time when anticipating later attended information. Listeners with cochlear implants did not show this local control over the timing of effort. Both groups showed greatest increase in effort in recovering missing words when listening to a single utterance, suggesting that capacity to exert effort is mediated by (and spread across) local demands of the task.

During spoken language comprehension listeners transform continuous acoustic cues into categories (e.g., /b/ and /p/). While long-standing research suggests that phonetic categories are activated in a gradient way, there are also clear individual differences in that more gradient categorization has been linked to various communication impairments such as dyslexia and specific language impairments (Joanisse, Manis, Keating, & Seidenberg, 2000; Lopez-Zamora, Luque, Alvarez, & Cobos, 2012; Serniclaes, Van Heghe, Mousty, Carre, & Sprenger-Charolles, 2004; Werker & Tees, 1987). Crucially, most studies have used 2-alternative forced choice (2AFC) tasks to measure the sharpness of between-category boundaries. Here we propose an alternative paradigm that allows us to measure categorization gradiency in a more direct way. Furthermore, we follow an individual differences approach to (a) link this measure of gradiency to multiple cue integration, (b) explore its relationship to a set of other cognitive processes, and (c) evaluate its role in individuals' ability to perceive speech in noise. Our results provide validation for this new method of assessing phoneme categorization gradiency and offer preliminary insights into how different aspects of speech perception may be linked to each other and to more general cognitive processes. (PsycINFO Database Record

Accepted version to appear in BRM.

ObjectivesSpectral resolution is an important aspect of hearing speech accurately, and a known limiting factor in cochlear implants (CIs). The spectral ripple discrimination task is a psychophysical measure that has been found to correlate with word recognition, music perception, and various aspects of CI processing. It was hypothesized that the limited spectral sampling of CI processors would distort the output of spectral ripple stimuli in ways that could be problematic for their use in CI research. DesignIdealized acoustic ripple spectra were analyzed through synthetic filters matched to CI processor channels, and the output was examined for spectral density and modulation depth cross a wide range of inputs. Existing data showing correlations between ripple discrimination and speech recognition were re-analyzed with special consideration of scores above the spectral aliasing limit. ResultsThe analysis revealed numerous complicating factors in the transmission of spectral ripples through CIs, including spectral aliasing and modulation depth neutralization. These unintended distortions are nonlinear in nature, with the consequence that spectral ripple thresholds above a certain number cannot be ordered monotonically along a single parametric axis/dimension. For the Cochlear device, the critical limit for spectral aliasing is around 2.1 RPO, and the critical limit for modulation saturation is about 3.5 RPO. Neither limitation is remediated by changing stimulus phase. Changes in modulation depth are limited by the single channel with the narrowest input filter octave-scaled bandwidth (channel 9 in the Cochlear device). Correlations between ripple discrimination and various measures of speech recognition are improved when high ripple scores are dismissed or winsorized. An additional difficulty with interpreting discrimination thresholds is that ripple stimuli often do not match the spectral characteristics of modulations found in actual speech sounds.ConclusionsThe authors conclude that the practice of identifying the highest spectral density at which ripple phase inversion can be detected should be abandoned because (1) it lacks the well-behaved mathematical properties a psychophysical experimenter requires, and (2) high spectral densities are a poor representation of ecologically-relevant cues. Alternative testing strategies (ripple depth detection, ripple phase shift, acoustic-phonetic cue perception) should be taken, with the specific consideration of real speech acoustics in their development.

Speech is an attractive stimulus to use for testing the auditory system because it is both highly relevant to everyday life and also replete with acoustic complexity that can test a variety of auditory skills. This presentation shines a light on some commonly used test stimuli or stimulus practices and raises questions about their use, or the interpretations of previous results. For example, the vowel acoustic data published by Hillenbrand et al. (1995) has been adopted as a de facto gold standard even though it was intended as a snapshot of a single dialect at a particular point in time. In that paper, the dynamics of vowel spectra were emphasized, though this aspect of vowels has been often overlooked. In consonant testing, the use of the /a/ context is pervasive, and yet this consonant environment is far from “neutral,” especially in situations where perception of voicing/voice onset time is measured. Perception of consonants and words will be examined with regard to their generalization to perception of longer utterances. Finally, we will examine the balance between convenience and generalizability, including strengths and weaknesses of closed-set speech stimuli.

Suboptimal interfaces between cochlear implant (CI) electrodes and auditory neurons result in a loss or distortion of spectral information in specific frequency regions, which likely decreases CI users' speech identification performance. This study exploited speech acoustics to model regions of distorted CI frequency transmission to determine the perceptual consequences of suboptimal electrode-neuron interfaces. Normal hearing adults identified naturally spoken vowels and consonants after spectral information was manipulated through a noiseband vocoder: either (1) low-, middle-, or high-frequency regions of information were removed by zeroing the corresponding channel outputs, or (2) the same regions were distorted by splitting filter outputs to neighboring filters. These conditions simulated the detrimental effects of suboptimal CI electrode-neuron interfaces on spectral transmission. Vowel and consonant confusion patterns were analyzed with sequential information transmission, perceptual distance, and perceptual vowel space analyses. Results indicated that both types of spectral manipulation were equally destructive. Loss or distortion of frequency information produced similar effects on phoneme identification performance and confusion patterns. Consonant error patterns were consistently based on place of articulation. Vowel confusions showed that perceptions gravitated away from the degraded frequency region in a predictable manner, indicating that vowels can probe frequency-specific regions of spectral degradations.

Objectives This study was conducted to measure auditory perception by cochlear implant users in the spectral and temporal domains, using tests of either categorization (using speech-based cues) or discrimination (using conventional psychoacoustic tests). The authors hypothesized that traditional nonlinguistic tests assessing spectral and temporal auditory resolution would correspond to speech-based measures assessing specific aspects of phonetic categorization assumed to depend on spectral and temporal auditory resolution. The authors further hypothesized that speech-based categorization performance would ultimately be a superior predictor of speech recognition performance, because of the fundamental nature of speech recognition as categorization. Design Nineteen cochlear implant listeners and 10 listeners with normal hearing participated in a suite of tasks that included spectral ripple discrimination, temporal modulation detection, and syllable categorization, which was split into a spectral cue-based task (targeting the /ba/-/da/ contrast) and a timing cue-based task (targeting the /b/-/p/ and /d/-/t/ contrasts). Speech sounds were manipulated to contain specific spectral or temporal modulations (formant transitions or voice onset time, respectively) that could be categorized. Categorization responses were quantified using logistic regression to assess perceptual sensitivity to acoustic phonetic cues. Word recognition testing was also conducted for cochlear implant listeners. Results Cochlear implant users were generally less successful at utilizing both spectral and temporal cues for categorization compared with listeners with normal hearing. For the cochlear implant listener group, spectral ripple discrimination was significantly correlated with the categorization of formant transitions; both were correlated with better word recognition. Temporal modulation detection using 100- and 10-Hz-modulated noise was not correlated either with the cochlear implant subjects' categorization of voice onset time or with word recognition. Word recognition was correlated more closely with categorization of the controlled speech cues than with performance on the psychophysical discrimination tasks. Conclusions When evaluating people with cochlear implants, controlled speech-based stimuli are feasible to use in tests of auditory cue categorization, to complement traditional measures of auditory discrimination. Stimuli based on specific speech cues correspond to counterpart nonlinguistic measures of discrimination, but potentially show better correspondence with speech perception more generally. The ubiquity of the spectral (formant transition) and temporal (voice onset time) stimulus dimensions across languages highlights the potential to use this testing approach even in cases where English is not the native language.

When evaluating a listener's ability to hear speech in a challenging condition, an experimental challenge is determining whether an utterance was heard correctly or whether it was heard incorrectly and then mentally repaired. Mentally transforming incorrect perceptions into “correct” responses is not visible in intelligibility scores, but is likely a major part of what makes listening effortful for people with hearing loss, and therefore there is a need to design tests that track the occurrence and mental cost of making corrections during speech perception. A series of studies is presented which gathers evidence of the mental cost of repairing errors using pupillometry. Stimuli were explicitly designed to induce different types of cognitive repair so that the ensuing mental cost/listening effort can be measured as it unfolds over time, even without major changes in intelligibility. Results show that different errors result in significantly different amounts of effort. Major findings include quick bursts of effort for phonetic repairs, larger prolonged bursts of effort for entire-word repairs constrained by context, and a substantial increase in effort for errors that result in semantic incongruity across an utterance. Results are discussed with regard to implications for clinical interpretation as well as study design.

Within the field of hearing science, pupillometry is a widely used method for quantifying listening effort. Its use in research is growing exponentially, and many labs are (considering) applying pupillometry for the first time. Hence, there is a growing need for a methods paper on pupillometry covering topics spanning from experiment logistics and timing to data cleaning and what parameters to analyze. This article contains the basic information and considerations needed to plan, set up, and interpret a pupillometry experiment, as well as commentary about how to interpret the response. Included are practicalities like minimal system requirements for recording a pupil response and specifications for peripheral, equipment, experiment logistics and constraints, and different kinds of data processing. Additional details include participant inclusion and exclusion criteria and some methodological considerations that might not be necessary in other auditory experiments. We discuss what data should be recorded and how to monitor the data quality during recording in order to minimize artifacts. Data processing and analysis are considered as well. Finally, we share insights from the collective experience of the authors and discuss some of the challenges that still lie ahead.

Objectives: This study measured the impact of auditory spectral resolution on listening effort. Systematic degradation in spectral resolution was hypothesized to elicit corresponding systematic increases in pupil dilation, consistent with the notion of pupil dilation as a marker of cognitive load. Design: Spectral resolution of sentences was varied with two different vocoders: (1) a noise-channel vocoder with a variable number of spectral channels; and (2) a vocoder designed to simulate front-end processing of a cochlear implant, including peak-picking channel selection with variable synthesis filter slopes to simulate spread of neural excitation. Pupil dilation was measured after subject-specific luminance adjustment and trial-specific baseline measures. Mixed-effects growth curve analysis was used to model pupillary responses over time. Results: For both types of vocoder, pupil dilation grew with each successive degradation in spectral resolution. Within each condition, pupillary responses were not related to intelligibility scores, and the effect of spectral resolution on pupil dilation persisted even when only analyzing trials in which responses were 100% correct. Conclusions: Intelligibility scores alone were not sufficient to quantify the effort required to understand speech with poor resolution. Degraded spectral resolution results in increased effort required to understand speech, even when intelligibility is at 100%. Pupillary responses were a sensitive and highly granular measurement to reveal changes in listening effort. Pupillary responses might potentially reveal the benefits of aural prostheses that are not captured by speech intelligibility performance alone as well as the disadvantages that are overcome by increased listening effort.

Vowel perception is influenced by precursor sounds that are resynthesized to shift frequency regions [Ladefoged and Broadbent (1957). J. Acoust. Soc. Am. 29(1), 98-104] or filtered to emphasize narrow [Kiefte and Kluender (2008). J. Acoust. Soc. Am. 123(1), 366-376] or broad frequency regions [Watkins (1991). J. Acoust. Soc. Am. 90(6), 2942-2955]. Spectral differences between filtered precursors and vowel targets are perceptually enhanced, producing spectral contrast effects (e.g., emphasizing spectral properties of /ɪ/ in the precursor elicited more /ɛ/ responses to an /ɪ/-/ɛ/ vowel continuum, and vice versa). Historically, precursors have been processed by high-gain filters, resulting in prominent stable long-term spectral properties. Perceptual sensitivity to subtler but equally reliable spectral properties is unknown. Here, precursor sentences were processed by filters of variable bandwidths and different gains, then followed by vowel sounds varying from /ɪ/-/ɛ/. Contrast effects were widely observed, including when filters had only 100-Hz bandwidth or +5 dB gain. Average filter power was a good predictor of the magnitudes of contrast effects, revealing a close linear correspondence between the prominence of a reliable spectral property and the size of shifts in perceptual responses. High sensitivity to subtle spectral regularities suggests contrast effects are not limited to high-power filters, and thus may be more pervasive in speech perception than previously thought.

Objectives Previous research has found that relative to their peers with normal hearing (NH), children with cochlear implants (CIs) produce the sibilant fricatives /s/ and /∫/ less accurately and with less subphonemic acoustic contrast. The present study sought to further investigate these differences across groups in two ways. First, subphonemic acoustic properties were investigated in terms of dynamic acoustic features that indexed more than just the contrast between /s/ and /∫/. Second, the authors investigated whether such differences in subphonemic acoustic contrast between sibilant fricatives affected the intelligibility of sibilant-initial single word productions by children with CIs and their peers with NH. Design In experiment 1, productions of /s/ and /∫/ in word-initial prevocalic contexts were elicited from 22 children with bilateral CIs (aged 4 to 7 years) who had at least 2 years of CI experience and from 22 chronological age-matched peers with NH. Acoustic features were measured from 17 points across the fricatives: peak frequency was measured to index the place of articulation contrast; spectral variance and amplitude drop were measured to index the degree of sibilance. These acoustic trajectories were fitted with growth-curve models to analyze time-varying spectral change. In experiment 2, phonemically accurate word productions that were elicited in experiment 1 were embedded within four-talker babble and played to 80 adult listeners with NH. Listeners were asked to repeat the words, and their accuracy rate was used as a measure of the intelligibility of the word productions. Regression analyses were run to test which acoustic properties measured in experiment 1 predicted the intelligibility scores from experiment 2. Results The peak frequency trajectories indicated that the children with CIs produced less acoustic contrast between /s/ and /∫/. Group differences were observed in terms of the dynamic aspects (i.e., the trajectory shapes) of the acoustic properties. In the productions by children with CIs, the peak frequency and the amplitude drop trajectories were shallower, and the spectral variance trajectories were more asymmetric, exhibiting greater increases in variance (i.e., reduced sibilance) near the fricative-vowel boundary. The listeners' responses to the word productions indicated that when produced by children with CIs, /∫/-initial words were significantly more intelligible than /s/-initial words. However, when produced by children with NH, /s/-initial words and /∫/-initial words were equally intelligible. Intelligibility was partially predicted from the acoustic properties (Cox & Snell pseudo-R > 0.190), and the significant predictors were predominantly dynamic, rather than static, ones. Conclusions Productions from children with CIs differed from those produced by age-matched NH controls in terms of their subphonemic acoustic properties. The intelligibility of sibilant-initial single-word productions by children with CIs is sensitive to the place of articulation of the initial consonant (/∫/-initial words were more intelligible than /s/-initial words), but productions by children with NH were equally intelligible across both places of articulation. Therefore, children with CIs still exhibit differential production abilities for sibilant fricatives at an age when their NH peers do not.

Purpose This study investigated how listeners' native language affects their weighting of acoustic cues (such as vowel quality, pitch, duration, and intensity) in the perception of contrastive word stress. Method Native speakers ( N = 45) of typologically diverse languages (English, Russian, and Mandarin) performed a stress identification task on nonce disyllabic words with fully crossed combinations of each of the 4 cues in both syllables. Results The results revealed that although the vowel quality cue was the strongest cue for all groups of listeners, pitch was the second strongest cue for the English and the Mandarin listeners but was virtually disregarded by the Russian listeners. Duration and intensity cues were used by the Russian listeners to a significantly greater extent compared with the English and Mandarin participants. Compared with when cues were noncontrastive across syllables, cues were stronger when they were in the iambic contour than when they were in the trochaic contour. Conclusions Although both English and Russian are stress languages and Mandarin is a tonal language, stress perception performance of the Mandarin listeners but not of the Russian listeners is more similar to that of the native English listeners, both in terms of weighting of the acoustic cues and the cues' relative strength in different word positions. The findings suggest that tuning of second-language prosodic perceptions is not entirely predictable by prosodic similarities across languages.