Search

We present a demonstration system for incremental spoken human–machine dialogue for task-centric domains that includes a controller for verbal and nonverbal behavior for virtual agents. The dialogue management components can handle uncertainty in input and resolve it interactively with high responsivity, and state tracking is aware of momentary events such as interruptions by the user. Aside from adaptable dialogue strategies, such as for grounding, the system includes a multimodal floor management controller that attempts to limit the influence of idiosyncratic dialogue behavior on the part of our primary user groups – older adults and people with cognitive impairments – both of which have previously participated in pilot studies using the platform.

Human interaction has a distinct collaborative quality based on the attribution of communicative intentionality. Two networks in the human brain are often described as part of the “social brain”: the mirror system for recognizing intentional behavior and the mentalizing (theory of mind) system for processing it. We equip virtual agents with both systems and model their interaction during embodied communication. Results of simulation experiments demonstrate how higher orders of theory of mind lead to more robustness of communication by enabling interactive grounding processes.

This paper addresses the semantic coordination of speech and gesture, a major prerequisite when endowing virtual agents with convincing multimodal behavior. Previous research has focused on build- ing rule- or data-based models speci c for a particular language, culture or individual speaker, but without considering the underlying cognitive processes. We present a exible cognitive model in which both linguistic as well as cognitive constraints are considered in order to simulate natu- ral semantic coordination across speech and gesture. An implementation of this model is presented and rst simulation results, compatible with empirical data from the literature are reported.

Gesturing behavior is subject to great variations across situations, individuals, or cultures. These variations make gestures hard for systematic studies and modeling attempts. However, gesture research on real humans and modeling approaches with virtual agents have made significant progress in the last years. In this chapter we discuss the state of research and present results from an extensive empirical study on human iconic gestures in direction giving dialogues. It is described how machine learning methods can be employed to extract different speakers’ gesturing style and to generate individualized language and gestures in ECAs. Evaluations show that human observers rate virtual agents better in terms of competence, human-likeness, or likability when a consistent individual gesture style is produced.

First impressions of others are fundamental for the further develop- ment of a relationship and are thus of major importance for the design of vir- tual agents, too. We addressed the question whether there is a second chance for first impressions with regard to the major dimensions of social cognition–warmth and competence. We employed a novel experimental set-up that combined agent appearance (robot-like vs. human-like) and agent behavior (gestures present vs. absent) of virtual agents as between-subject factors with a repeated measures de- sign. Results indicate that ratings of warmth depend on interaction effects of time and agent appearance, while evaluations of competence seem to depend on the interaction of time and nonverbal behavior. Implications of these results for basic and applied research on intelligent virtual agents will be discussed.

This paper presents a project in which an Embodied Conversational Agent (ECA) attemtps to detect the personality of the user by asking him (or her) a series of questions. The project uses the Elckerlyc platform, a Behavior Markup Language (BML) compliant behavior realizer for ECAs [1]. BML allows the design and implementation of verbal and nonverbal behaviors in an abstract way, without reference to the ECA model in question. This means the designer only needs to decide “what‿ kind of behaviors should be performed by the ECA and to specify “when‿ these behaviors should be performed, without needing to know how the realization engine actually displays these behaviors on the ECA’s embodiment. The aim of this research was to extend the Psychometer [2], a text based dialogue system for personality detection, with a virtual embodiment that uses gaze behavior and facial expressions. The Psychometer asks the user questions, adapted from an existing questionnaire, until it has enough answers to classify the user’s personality according to the well-known Big Five theory of personality traits.

This paper introduces Flipper, an specification language and interpreter for Information State Update rules that can be used for developing spoken dialogue systems and embodied conversational agents. The system uses XML-templates to modify the information state and to select behaviours to perform.

How does a virtual agent’s gesturing behavior influence the user’s perception of communication quality and the agent’s personality? This question was investigated in an evaluation study of co-verbal iconic gestures produced with the Bayesian network-based production model GNetIc. A network learned from a corpus of several speakers was compared with networks learned from individual speaker data, as well as two control conditions. Results showed that automatically GNetIc-generated gestures increased the perceived quality of an object description given by a virtual human. Moreover, the virtual agent showing gesturing behavior generated with individual speaker networks was rated more positively in terms of likeability, competence and human-likeness.

Why an iconic gesture takes its particular form is a largely open question, given the variations one finds across both situations and speakers. We present results of an empirical study that analyzes correlations between contextual factors (referent features, discourse) and gesture features, and tests whether they are systematic (shared among speakers) or idiosyncratic (inter-individually different). Based on this, a computational model of gesture formation is presented that combines data-based, probabilistic and model-based decision making.

This study investigated whether emotional expressions of an ECA influence the participants´ nonverbal and verbal behavior. 70 participants took part in a small talk (10 min.) situation with the ECA MAX who was presented with two different types of feedback: emotional feedback (EMO), which provided a feedback about the emotional state of MAX (including smiles and compliments) and envelope feedback (ENV), which provided a feedback about the comprehension of the participants´ contributions. In general we found that participants showed frequent behavior known from human-human-communication, such as proactive greetings or waving goodbye. Additionally, with regard to some behaviors, the agent´s behavior had an impact: Participants in EMO significantly gave more compliments and exhibited more phrases of politeness (“Thank you!”) than in ENV.

In natural communication, both speakers and listeners are active most of the time. While a speaker contributes new information, a listener gives feedback by producing unobtrusive (usually short) vocal or non-vocal bodily expressions to indicate whether he/she is able and willing to communicate, perceive, and understand the information, and what emotions and attitudes are triggered by this information. The simulation of feedback behavior for artificial conversational agents poses big challenges such as the concurrent and integrated perception and production of multi-modal and multi-functional expressions. We present an approach on modeling feedback for and with virtual humans, based on an approach to study ``embodied feedback'' as a special case of a more general theoretical account of embodied communication. A realization of this approach with the virtual human Max is described and results are presented.

Against the background that recent studies on embodied conversational agents demonstrate the importance of their behavior, an experimental study is presented that assessed the effects of different nonverbal behaviors of an embodied conversational agent on the users Ì experiences and evaluations as well as on their behavior. 50 participants conducted a conversation with different versions of the virtual agent Max, whose nonverbal communication was manipulated with regard to eyebrow movements and self- touching gestures. In a 2x2 between subjects design each behavior was varied in two levels: occurrence of the behavior compared to the absence of the behavior. Results show that self-touching gestures compared to no self-touching gestures have positive effects on the experiences and evaluations of the user, whereas eyebrow raising evoked less positive experiences and evaluations in contrast to no eyebrow raising. The nonverbal behavior of the participants was not affected by the agent's nonverbal behavior.

We describe an anthropomorphic agent that is engaged in an imitation game with the human user. In imitating natural gestures demonstrated by the user, the agent brings together gesture recognition and synthesis on two levels of representation. On the mimicking level, the essential form features of the meaning-bearing gesture phase (stroke) are extracted and reproduced by the agent. Meaning-based imitation requires extracting the semantic content of such gestures and re-expressing it with possibly alternative gestural forms. Based on a compositional semantics for shape-related iconic gestures, we present first steps towards this higher-level gesture imitation in a restricted domain.

FlurMax, a virtual agent, inhabits a hallway at the University of Bielefeld. He resides in a wide-screen panel equipped with a video camera to track and interact with visitors using speech, gesture, and emotional facial expression. For example, FlurMax will detect the presence of visitors and greet them with a friendly wave, saying "Hello, I am Max". FlurMax also recognizes simple gesturing of the by-passer, such as waving, and produces natural multimodal behaviors in response. FlurMax's behavior selection is controlled by a simple emotional/motivational system which gradually changes his mood between states like happy, bored, surprised, and neutral.