Search

In this article we report on two experiments about the perception of audiovisual cues to emotional speech. The article addresses two questions: 1) how do visual cues from a speaker's face to emotion relate to auditory cues, and (2) what is the recognition speed for various facial cues to emotion? Both experiments reported below are based on tests with video clips of emotional utterances collected via a variant of the well-known Velten method. More specifically, we recorded speakers who displayed positive or negative emotions, which were congruent or incongruent with the (emotional) lexical content of the uttered sentence. In order to test this, we conducted two experiments. The first experiment is a perception experiment in which Czech participants, who do not speak Dutch, rate the perceived emotional state of Dutch speakers in a bimodal (audiovisual) or a unimodal (audio- or vision-only) condition. It was found that incongruent emotional speech leads to significantly more extreme perceived emotion scores than congruent emotional speech, where the difference between congruent and incongruent emotional speech is larger for the negative than for the positive conditions. Interestingly, the largest overall differences between congruent and incongruent emotions were found for the audio-only condition, which suggests that posing an incongruent emotion has a particularly strong effect on the spoken realization of emotions. The second experiment uses a gating paradigm to test the recognition speed for various emotional expressions from a speaker's face. In this experiment participants were presented with the same clips as experiment I, but this time presented vision-only. The clips were shown in successive segments (gates) of increasing duration. Results show that participants are surprisingly accurate in their recognition of the various emotions, as they already reach high recognition scores in the first gate (after only 160 ms). Interestingly, the recognition scores raise faster for positive than negative conditions. Finally, the gating results suggest that incongruent emotions are perceived as more intense than congruent emotions, as the former get more extreme recognition scores than the latter, already after a short period of exposure.

The existence of auditory cues such as intonation, rhythm, and pausing that facilitate end-of-utterance detection is by now well established. It has been argued repeatedly that speakers may also employ visual cues to indicate that they are at the end of their utterance. This raises at least two questions, which are addressed in the current paper. First, which modalities do speakers use for signalling finality and nonfinality, and second, how sensitive are observers to these signals. Our goal is to investigate the relative contribution of three different conditions to end-of-utterance detection: the two unimodal ones, vision only and audio only, and their bimodal combination. Speaker utterances were collected via a novel semicontrolled production experiment, in which participants provided lists of words in an interview setting. The data thus collected were used in two perception experiments, which systematically compared responses to unimodal (audio only and vision only) and bimodal (audio-visual) stimuli. Experiment I is a reaction time experiment, which revealed that humans are significantly quicker in end-of-utterance detection when confronted with bimodal or audio-only stimuli, than for vision-only stimuli. No significant differences in reaction times were found between the bimodal and audio-only condition, and therefore a second experiment was conducted. Experiment II is a classification experiment, and showed that participants perform significantly better in the bimodal condition than in the two unimodal ones. Both the first and the second experiment revealed interesting differences between speakers in the various conditions, which indicates that some speakers are more expressive in the visual and others in the auditory modality.

We describe two experiments on signaling and detecting uncertainty in audiovisual speech by adults and children. In the first study, utterances from adult speakers and child speakers (aged 7-8) were elicited and annotated with a set of six audiovisual features. It was found that when adult speakers were uncertain they were more likely to produce fillers, delays, high intonation, eyebrow movements, and "funny faces." The basic picture for the child speakers was somewhat similar, in that the presence of certain audiovisual cues correlated with uncertainty, but the differences were relatively small and less often significant. In the second study both adult and child judges watched responses from adult and child speakers selected from the first study to find out whether they were able to correctly estimate a speakers' level of uncertainty. It was found that both child and adult judges gave more accurate scores for answers from adult speakers than from child speakers and that child judges overall provided less accurate scores than adult judges.

The current article describes research on whether the goodness of a particular speaking style correlates with the way speakers distribute pitch accents in their speech. Study 1 analyzed two Flemish newsreaders, who, according to poll ratings, had previously been judged to represent a good vs bad speaker. A perception study in which participants had to assess the quality of spoken paragraphs produced by either of the two speakers confirmed that one speaker was rated as significantly and consistently better than the other one. An exploration of the accent distributions in those paragraphs showed that the accent distributions of the better speaker were more similar to the ones of a gold standard, i.e., the accent distributions as predicted by two independent intonation experts. Study 2 compared synthetic versions of a selection of the paragraphs of study 1, generated by a Dutch text-to-speech system. It compared three basically identical versions of the texts, except that they had different accent distributions according to the gold standard, or to distributions as observed in the productions of the two newsreaders. A perception study revealed that the versions of the bad speaker were rated as being significantly worse than the other versions. The two studies thus show that variation in accent distribution can indeed affect the way spoken texts are assessed in terms of their perceived quality. [ABSTRACT FROM AUTHOR]

This article examines potential prosodic predictors of emotional speech in utterances perceived as conveying that good or bad news is about to be delivered. Speakers were asked to call an experimental confederate to inform her about whether or not she had been given a job she had applied for. A perception study was then performed in which initial fragments of the recorded utterances, not containing any explicit lexical cues to emotional content, were presented to listeners who had to rate whether good or bad news would follow the utterance. The utterances were then examined to discover acoustic and prosodic features that distinguished between good and bad news. It was found that speakers in the production study were not simply reflecting their own positive or negative mood during the experiment, but rather appeared to be influenced by the valence of the positive or negative message they were preparing to deliver. Positive and negative utterances appeared to be judged differently with respect to a number of perceived attributes of the speakers’ voices (like sounding hesitant or nervous). These attributes correlated with a number of automatically obtained acoustic features. [ABSTRACT FROM AUTHOR]

This article focuses on the analysis and prediction of corrections, defined as turns where a user tries to correct a prior error made by a spoken dialogue system. We describe our labeling procedure of various corrections types and statistical analyses of their features in a corpus collected from a train information spoken dialogue system. We then present results of machine-learning experiments designed to identify user corrections of speech recognition errors. We investigate the predictive power of features automatically computable from the prosody of the turn, the speech recognition process, experimental conditions, and the dialogue history. Our best-performing features reduce classification error from baselines of 25.70-28.99% to 15.72%. [ABSTRACT FROM AUTHOR]

While considerable research has been done in the past in prosody and conversation, work which combines the two has been lamentably scarce. Yet, to gain real insight into basic characteristics of either phenomena, most researchers would agree that the study of prosody is critical to the understanding of how people converse, and the study of conversational behavior is equally crucial to the understanding of prosodic phenomena outside the laboratory. On the one hand, prosody is known to be a flexible structuring device, which may signal different aspects of dialog structure, such as turn-taking, topic structure. information status, type of speech act, attitudinal and emotional characteristics of utterances, and mutual beliefs of dialog participants. Obviously, for students of prosodic function, the conversational domain should provide a fertile area of research. On the other hand, one intriguing fact about conversation is that it normally proceeds in a smooth and successful manner. Conversational analysts may find new answers to why natural spoken communication works so well by examining the prosodic as well as the textual features of their data.

Previous studies found that repeated references in successful communication are often reduced, not only at the acoustic level, but also in terms of words and manual co-speech gestures. In the present study, we investigated whether repeated references are still reduced in a situation when reduction would not be beneficial for the communicative situation, namely after the speaker receives negative feedback from the addressee. In a director–matcher task (experiment I), we studied gesture rate, as well as the general form of the gestures produced in initial and repeated references. In a separate experiment (experiment II) we studied whether there might (also) be more gradual differences in gesture form between gestures in initial and repeated references, by asking human judges which of two gestures (one from an initial and one from a repeated reference following negative feedback) they considered more precise. In both experiments, mutual visibility was added as a between subjects factor. Results showed that after negative feedback, gesture rate increased in a marginally significant way. With regard to gesture form, we found little evidence for changes in gesture form after negative feedback, except for a marginally significant increase of the number of repeated strokes within a gesture. Lack of mutual visibility only had a significant reducing effect on gesture size, and did not interact with repetition in any way. However, we did find gradual differences in gesture form, with gestures produced after negative feedback being judged as marginally more precise than initial gestures. The results from the present study suggest that in the production of unsuccessful repeated references, a process different from the reduction process as found in previous studies in repeated references takes place, with speakers appearing to put more effort into their gestures after negative feedback, as suggested by the data trending towards an increased gesture rate and towards gestures being judged as more precise after feedback. [ABSTRACT FROM AUTHOR]

Highlights: [•] Previous research has found that speech and gesture are closely related. [•] It has been found that preventing gesture can influence some aspects of speech. [•] In this study, an instructional director–matcher task was set up where directors were not able to gesture for half of the experiment. [•] Contrary to expectations, there was no difference between directors’ speech with and without gesture. [Copyright &y& Elsevier]

Abstract: Studies of perceptually based predictions of upcoming prosodic boundaries in spontaneous Swedish speech, both by native speakers of Swedish and of native speakers of standard American English reveal marked similarity in judgments. We examined whether Swedish and American listeners were able to predict the occurrence and strength of upcoming boundaries in a series of web-based perceptive experiments. Utterance fragments (in both long and short versions) were selected from a corpus of spontaneous Swedish speech, which was first labeled for boundary presence and strength by expert labelers. These fragments were then presented to listeners, who were instructed to guess whether or not they were followed by a prosodic break, and if so, what the strength of the break was. Results revealed that both Swedish and American listening groups were indeed able to predict whether or not a boundary (of a particular strength) followed the fragment. This suggests that acoustic and prosodic, rather than lexico-grammatical and semantic information was being used by listeners as a primary cue. Acoustic and prosodic correlates of these judgments were then examined, with significant correlations found between judgments and the presence/absence of final creak and phrase-final f0 level and slope. [Copyright &y& Elsevier]

In spoken dialogue systems, it is important for the system to know how likely a speech recognition hypothesis is to be correct, so it can reject misrecognized user turns, or, in cases where many errors have occurred, change its interaction strategy or switch the caller to a human attendant. We have identified prosodic features which predict more accurately when a recognition hypothesis contains errors than the acoustic confidence scores traditionally used in automatic speech recognition in spoken dialogue systems. We describe statistical comparisons of features of correctly and incorrectly recognized turns in the TOOT train information corpus and the W99 conference registration corpus, which reveal significant prosodic differences between the two sets of turns. We then present machine learning results showing that the use of prosodic features, alone and in combination with other automatically available features, can predict more accurately whether or not a user turn was correctly recognized, when compared to the use of acoustic confidence scores alone. [ABSTRACT FROM AUTHOR]

Abstract: Interlocutors sometimes repeat each other’s co-speech hand gestures. In three experiments, we investigate to what extent the copying of such gestures’ form is tied to their meaning in the linguistic context, as well as to interlocutors’ representations of this meaning at the conceptual level. We found that gestures were repeated only if they could be interpreted within the meaningful context provided by speech. We also found evidence that the copying of gesture forms is mediated by representations of meaning. That is, representations of meaning are also converging across interlocutors rather than just representations of gesture form. We conclude that the repetition across interlocutors of representational hand gestures may be driven by representations at the conceptual level, as has also been proposed for the repetition of referring expressions across interlocutors (lexical entrainment). That is, adaptation in gesture resembles adaptation in speech, rather than it being an instance of automated motor-mimicry. [Copyright &y& Elsevier]