Search

This paper presents the results of three perceptual experiments investigating the role of auditory and visual channels for the identification of statements and echo questions in Brazilian Portuguese. Ten Brazilian speakers (five male) were video-recorded (frontal view of the face) while they produced a sentence (" Como você sabe "), either as a statement (meaning " As you know. ") or as an echo question (meaning " As you know ?"). Experiments were set up including the two different intonation contours. Stimuli were presented in conditions with clear and degraded audio as well as congruent and incongruent information from both channels. Results show that Brazilian listeners were able to distinguish statements and questions prosodically and visually, with auditory cues being dominant over visual ones. In noisy conditions, the visual channel improved the interpretation of prosodic cues robustly, while it degraded them in conditions where the visual information was incongruent with the auditory information. This study shows that auditory and visual information are integrated during speech perception, also when applied to prosodic patterns.

In this article we report on two experiments about the perception of audiovisual cues to emotional speech. The article addresses two questions: 1) how do visual cues from a speaker's face to emotion relate to auditory cues, and (2) what is the recognition speed for various facial cues to emotion? Both experiments reported below are based on tests with video clips of emotional utterances collected via a variant of the well-known Velten method. More specifically, we recorded speakers who displayed positive or negative emotions, which were congruent or incongruent with the (emotional) lexical content of the uttered sentence. In order to test this, we conducted two experiments. The first experiment is a perception experiment in which Czech participants, who do not speak Dutch, rate the perceived emotional state of Dutch speakers in a bimodal (audiovisual) or a unimodal (audio- or vision-only) condition. It was found that incongruent emotional speech leads to significantly more extreme perceived emotion scores than congruent emotional speech, where the difference between congruent and incongruent emotional speech is larger for the negative than for the positive conditions. Interestingly, the largest overall differences between congruent and incongruent emotions were found for the audio-only condition, which suggests that posing an incongruent emotion has a particularly strong effect on the spoken realization of emotions. The second experiment uses a gating paradigm to test the recognition speed for various emotional expressions from a speaker's face. In this experiment participants were presented with the same clips as experiment I, but this time presented vision-only. The clips were shown in successive segments (gates) of increasing duration. Results show that participants are surprisingly accurate in their recognition of the various emotions, as they already reach high recognition scores in the first gate (after only 160 ms). Interestingly, the recognition scores raise faster for positive than negative conditions. Finally, the gating results suggest that incongruent emotions are perceived as more intense than congruent emotions, as the former get more extreme recognition scores than the latter, already after a short period of exposure.

In spoken conversations, speakers and their addressees constantly seek and provide different forms of audiovisual feedback, also known as backchannels, which include nodding, vocalizations and facial expressions. It has previously been shown that addressees backchannel at specific points during an interaction, namely after a speaker provided a cue to elicit feedback from the addressee. However, addressees may differ in the frequency and type of feedback that they provide, and likewise, speakers may vary the type of cues they generate to signal the backchannel opportunity points (BOPs). Research on the extent to which backchanneling is idiosyncratic is scant. In this article, we quantify and analyze the variability in feedback behavior of 14 addressees who all interacted with the same speaker stimulus. We conducted this research by means of a previously developed experimental paradigm that generates spontaneous interactions in a controlled manner. Our results show that (1) backchanneling behavior varies between listeners (some addressees are more active than others) and (2) backchanneling behavior varies between BOPs (some points trigger more responses than others). We discuss the relevance of these results for models of human–human and human–machine interactions. [ABSTRACT FROM AUTHOR]

The article focuses on a study which examines the effects of social and cultural contexts on smiles displayed by children during gameplay. Topics discussed include the effects of sociality and culture on adult's smiling behavior and on children's expressions, the distribution of smiles and non-smiles as a function of sociality and culture, for video recordings depicting winning and losing, and the frequency and percentages of smiles observed as a function of sociality, culture and game outcome.

We studied the effect of two social settings (collaborative versus competitive) on the visual and auditory expressions of uncertainty by children in two age groups (8 and 11). We conducted an experiment in which children played a quiz game in pairs. They either had to collaborate or compete with each other. We found that the Feeling-of-Knowing of eight-year-old children did not seem to be affected by the social setting, contrary to the Feeling-of-Knowing of 11-year-old children. In addition, we labelled children’s expressions in clips taken from the experiment for various visual and auditory features. We found that children used some of these features to signal uncertainty and that older children exhibited clearer cues than younger children. In a subsequent perception test, adults rated children’s certainty in clips used for labelling. It appeared that older children and children in competition expressed their confidence level more clearly than younger children and children in collaboration. [ABSTRACT FROM PUBLISHER]

Abstract: This paper looks into the extent to which facial expressions may reveal whether a person is telling the truth or not. More specifically, it studies selected clips from video recordings of children who had participated in an interactive story paradigm that naturally elicits minimal pairs of truthful and deceptive utterances in participants. In two perception experiments, these pairs of clips (with audio removed) were shown to adult observers who were given the task to guess which member of the pair contained the deceptive utterance. Experiment 1 tested whether the likelihood of correct detection could be enhanced when clips were slowed down compared to clips at normal speed. Results revealed that this manipulation indeed had a positive effect on lie detection, albeit that the effect interacted with the order of presentation (lies are easier to see when they are shown after rather than before the truthful utterance) and kind of lie (second attempts of children to lie reveal more cues than their first attempts). Experiment 2 explored whether lie detection is different for recordings in which the full face of a child is shown, or for recordings in which either the eye or mouth region is hidden after digital manipulation. This experiment revealed that the partial presentations of the face lead to more correct deception detection than the full face presentation. Implications of the outcomes of Experiment 1 and 2 for lie detection and for a general model of nonverbal communication are discussed. [Copyright &y& Elsevier]

This article focuses on how growing children use prosody for communicative purposes. Prosody refers to the set of expressive features that do not so much determine what speakers say, but rather how they say it. It includes both auditory features, such as intonation and tempo, and visual features, such as facial expressions. Our central hypothesis is that children, as they grow older, become more socially aware - a process which is reflected in the way they express themselves in prosody. To this end, we present the results of three studies that focus on how children use such features (1) to mark their level of uncertainty, (2) to signal a positive or negative emotion, and (3) to show whether they are being truthful or not. All the studies use a game-based experimental paradigm that is especially suited for analyses of child behaviour. The approach combines controlled elicitations of spontaneous interactions with perception tests that explore how children's expressions are being interpreted. Results of such studies are relevant for pedagogical and diagnostic purposes, and will lead to improvements in child-directed communication systems. [ABSTRACT FROM AUTHOR]

Speakers are argued to adapt their language production to their addressee's needs. For instance, speakers produce fewer and smaller hand gestures when interlocutors cannot see each other. Yet is this because speakers know their addressee cannot see them, or because they themselves do not see their addressee? By means of computer-mediated communication we manipulated these factors independently. We found that speakers took into account what their addressee saw. They produced more and larger gestures when they knew the addressee could see them. Seeing the addressee increased gesture production only if speakers could readily interpret the addressee's eye gaze, which is not usually the case in mediated interaction. Adding this affordance resulted in gesturing being similar in mediated and unmediated communication. [ABSTRACT FROM AUTHOR]

Abstract: This article investigates whether newsreaders exploit their expressive style for signalling different types of communicatively relevant information. The study reported upon here investigates whether these speakers use their facial expressions to “package” the content of their messages so that they reflect the relative importance, the emotional connotation and the intended audience of the news items. To this end, Dutch newsreaders (addressing audiences consisting of either adults or children) were analysed in terms of their facial expressions. The first study explores whether eyebrow movements and head nods of newsreaders (only adult news) are correlated with the relative prominence of words in their messages. Consistent with findings in the literature for other styles of speech, it was found that pitch-accented words in the newsreader data tended to be marked by variations in eyebrow movements and head nods. The second study aimed to find out whether newsreaders adapt their facial expressions to the seriousness of the topic they are talking about, and, if so, whether this adaptation differs for newsreaders addressing adults or children. Analyses reveal that both the topic and the intended audience had an effect of the newsreaders’ expressions, especially as the positive topics and the child-directed news items tended to be more expressive. The relevance of the findings of studies 1 and 2 for speech production and recognition is discussed. [Copyright &y& Elsevier]

Abstract: This article addresses two related questions regarding the perception of facial markers of prominence in spoken utterances: (1) how important are visual cues to prominence from the face with respect to auditory cues? and (2) are there differences between different facial areas in their cue value for prosodic prominence? The first perception experiment tackles the relation between auditory and visual cues by means of a reaction-time experiment. For this experiment, recordings of a sentence with three prosodically prominent words were systematically manipulated in such a way that auditory and visual cues to prominence were either congruent (occurring on the same word) or incongruent (in that the auditory and the visual cue were positioned on different words). Participants were instructed to indicate as fast as possible which word they perceived as the most prominent one. Results show that participants can more easily determine prominence when the visual cue occurs on the same word as the auditory cue, while displaced visual cues hinder prominence perception. The second experiment investigates which area of a speaker''s face contains the strongest cues to prominence, using stimuli with either the entire face visible or only parts of it. The task of the participants was to indicate for each stimulus which word they perceived as the most prominent one. Results show that the upper facial area has stronger cue value for prominence detection than the bottom part, and that the left part of the face is more important than the right part. Results of mirror-images of the original fragments show that this latter result is due both to a speaker and an observer effect. [Copyright &y& Elsevier]

Abstract: Recently, some studies have revealed that facial gestures can play an important role in teasing out the meaning of interrogative sentence types in a particular language (Srinivasan and Massaro, 2003; Borràs-Comes and Prieto, 2011; among others). However, less is known about potential cross-linguistic differences. This paper investigates the interaction between facial gestures and intonation in the distinction between information-seeking and incredulity yes/no questions in two languages (i.e., Catalan and Dutch) which use different prosodic strategies to express the distinction between these two types of interrogatives. While Dutch uses two phonologically distinct intonational contours, Catalan uses the same pitch contour with a distinction in pitch range. Twenty listeners of Catalan and twenty listeners of Dutch performed a perception experiment with audio-only, video-only, and audiovisual stimuli in congruent and incongruent intonation and gestural combinations. The results reveal that there is a contrast between Dutch and Catalan listeners in the perceptual processing of these sentences. While Dutch participants rely more on intonational differences, Catalan participants use the facial expression cues to a greater extent. All in all, the results show that both languages express pragmatic contrasts both at the intonation and facial expression levels, and native speakers are highly sensitive to the relative weight of these cues at the perceptual level. [Copyright &y& Elsevier]

The tearing effect refers to the relevance of tears as an important visual cue adding meaning to human facial expression. However, little is known about how people process these visual cues and their mediating role in terms of emotion perception and person judgment. We therefore conducted two experiments in which we measured the influence of tears on the identification of sadness and the perceived need for social support at an early perceptional level. In two experiments (1 and 2), participants were exposed to sad and neutral faces. In both experiments, the face stimuli were presented for 50 milliseconds. In experiment 1, tears were digitally added to sad faces in one condition. Participants demonstrated a significant faster recognition of sad faces with tears compared to those without tears. In experiment 2, tears were added to neutral faces as well. Participants had to indicate to what extent the displayed individuals were in need of social support. Study participants reported a greater perceived need for social support to both sad and neutral faces with tears than to those without tears. This study thus demonstrated that emotional tears serve as important visual cues at an early (pre-attentive) level. [ABSTRACT FROM AUTHOR]

This paper describes research into audiovisual cues to communication problems in interactions between users and a spoken dialogue system. The study consists of two parts. First, we describe a series of three perception experiments in which subjects are offered film fragments (without any dialogue context) of speakers interacting with a spoken dialogue system. In half of these fragments, the speaker is or becomes aware of a communication problem. Subjects have to determine by forced choice which are the problematic fragments. In all three tests, subjects are capable of performing this task to some extent, but with varying levels of correct classifications. Second, we report results of an observational analysis in which we first attempt to relate the perceptual results to visual features of the stimuli presented to subjects, and second to find out which visual features actually are potential cues for error detection. Our major finding is that more problematic contexts lead to more dynamic facial expressions, in line with earlier claims that communication errors lead to marked speaker behaviour. We conclude that visual information from a user's face is potentially beneficial for problem detection. [ABSTRACT FROM AUTHOR]