Search

Human affect sensing can be obtained from a broad range of behavioral cues and signals that are available via visual, acoustic, and tactual expressions or presentations of emotions. Affective states can thus be recognized from visible/external signals such as gestures (e.g., facial expressions, body gestures, head movements, etc.), and speech (e.g., parameters such as pitch, energy, frequency and duration), or invisible/internal signals such as physiological signals (e.g., heart rate, skin conductivity, salivation, etc.), brain and scalp signals, and thermal infrared imagery. Despite the available range of cues and modalities in human-human interaction (HHI), the mainstream emotion research has mostly focused on facial expressions (Hadjikhani & De Gelder, 2003). In line with this, most of the past research on affect sensing and recognition has also focused on facial expressions and on data that has been posed on demand or acquired in laboratory settings. Additionally, each sense such as vision, hearing, and touch has been considered in isolation. However, natural human-human interaction is multimodal and not occurring in predetermined, restricted and controlled settings. In the day-to-day world people do not present themselves to others as voiceor body-less faces or faceor body-less voices (Walker-Andrews, 1997). Moreover, the available emotional signals such as facial expression, head movement, hand gestures, and voice are unified in space and time (see Figure 1). They inherently share the same spatial location, and their occurrences are temporally synchronized. Cognitive neuroscience research thus claims that information coming from various modalities is combined in our brains to yield multimodally determined percepts (Driver & Spence, 2000). In real life situations, our different senses receive correlated information about the same external event. When assessing each others’ emotional or affective state, we are capable of handling significantly variable conditions in terms of viewpoint (i.e. frontal, profile, even back view), tilt angle, distance (i.e., face to face as well as at a distance) , illumination (i.e., both day and night conditions), occlusions (e.g., even when some body parts are occluded), motion (e.g., both when stationary and moving, walking and talking) and noise (e.g., while many people are chatting and interacting simultaneously).

This paper describes work in the recently started project SEMAINE, which aims to build a set of Sensitive Artificial Listeners – conversational agents designed to sustain an interaction with a human user despite limited verbal skills, through robust recognition and generation of non-verbal behaviour in real-time, both when the agent is speaking and listening. We report on data collection and on the design of a system architecture in view of real-time responsiveness.

Sensitive Artificial Listeners (SAL) are virtual dialogue partners who, despite their very limited verbal understanding, intend to engage the user in a conversation by paying attention to the user's emotions and non-verbal expressions. The SAL characters have their own emotionally defined personality, and attempt to drag the user towards their dominant emotion, through a combination of verbal and non-verbal expression. The demonstrator shows an early version of the fully autonomous SAL system based on audiovisual analysis and synthesis.

Research in Brain-Computer Interface (BCI) has significantly increased during the last few years. In addition to their initial role as assisting devices for the physically challenged, BCIs are now proposed for a wider range of applications. As in any HCI application, BCIs can also benefit from adapting their operation to the emotional state of the user. BCIs have the advantage of having access to brain activity which can provide significant insight into the user's emotional state. This information can be utilized in two manners. 1) Knowledge of the influence of the emotional state on brain activity patterns can allow the BCI to adapt its recognition algorithms, so that the intention of the user is still correctly interpreted in spite of signal deviations induced by the subject's emotional state. 2) The ability to recognize emotions can be used in BCIs to provide the user with more natural ways of controlling the BCI through affective modulation. Thus, controlling a BCI by recollecting a pleasant memory can be possible and can potentially lead to higher information transfer rates. These two approaches of emotion utilization in BCI are elaborated in detail in this paper in the framework of non-invasive EEG based BCIs.

Automatic pain recognition is an evolving research area with promising applications in health care. In this paper, we propose the first fully automatic approach to continuous pain intensity estimation from facial images. We first learn a set of independent regression functions for continuous pain intensity estimation using different shape (facial landmarks) and appearance (DCT and LBP) features, and then perform their late fusion. We show on the recently published UNBC-MacMaster Shoulder Pain Expression Archive Database that late fusion of the afore-mentioned features leads to better pain intensity estimation compared to feature-specific pain intensity estimation.

In this paper we introduce a multi-modal database for the analysis of human interaction, in particular mimicry, and elaborate on the theoretical hypotheses of the relationship between the occurrence of mimicry and human affect. The recorded experiments are designed to explore this relationship. The corpus is recorded with 18 synchronised audio and video sensors, and is annotated for many different phenomena, including dialogue acts, turn-taking, affect, head gestures, hand gestures, body movement and facial expression. Recordings were made of two experiments: a discussion on a political topic, and a role-playing game. 40 participants were recruited, all of whom selfreported their felt experiences. The corpus will be made available to the scientific community.

Recognition and analysis of human emotions have attracted a lot of interest in the past two decades and have been researched extensively in neuroscience, psychology, cognitive sciences, and computer sciences. Most of the past research in machine analysis of human emotion has focused on recognition of prototypic expressions of six basic emotions based on data that has been posed on demand and acquired in laboratory settings. More recently, there has been a shift toward recognition of affective displays recorded in naturalistic settings as driven by real world applications. This shift in affective computing research is aimed toward subtle, continuous, and context-specific interpretations of affective displays recorded in real-world settings and toward combining multiple modalities for analysis and recognition of human emotion. Accordingly, this article explores recent advances in dimensional and continuous affect modeling, sensing, and automatic recognition from visual, audio, tactile, and brain-wave modalities.

A widely accepted prediction is that computing will move to the background, weaving itself into the fabric of our everyday living spaces and projecting the human user into the foreground. If this prediction is to come true, then next generation computing, which we will call human computing, should be about anticipatory user interfaces that should be human-centered, built for humans based on human models. They should transcend the traditional keyboard and mouse to include natural, human-like interactive functions including understanding and emulating certain human behaviors such as affective and social signaling. This article discusses a number of components of human behavior, how they might be integrated into computers, and how far we are from realizing the front end of human computing, that is, how far are we from enabling computers to understand human behavior.

Building systems that are able to analyse communicative behaviours or take part in conversations requires a sound methodology in which the complex organisation of conversations is understood and tested on real-life samples. The data-driven approaches to human computing not only have a value for the engineering of systems, but can also provide feedback to the study of conversations between humans and between human and machines.

Welcome to Geneva and to ACII 2013, “the‿ conference in Affective Computing and Intelligent Interaction. After Beijing in 2005, Lisboa in 2007, Amsterdam in 2009, and Memphis in 2011, researchers from all over the world will converge to Geneva on 2-5 September 2013 for the 5th biannual ACII conference. Since 2005 and under the auspices of the Humaine association, the ACII conference series has become the premier international forum for research on affective human-machine interaction and intelligent affective systems. Affect is a phenomena of substantial importance in most if not all of human activities. This ACII edition therefore strives to emphasize the humanistic side of affective computing by promoting research at the cross-roads between engineering and human sciences (including biological, social and cultural aspects of human life). This will be exemplified by the papers in these proceedings, with topics as varied as computerized emotional modelling, literary and cinema studies, art, gaming, learning, stress and anxiety, avatars and virtual worlds, social computing, pattern recognition, advertisement, and real-time implementations to name a few. All have in affect and emotions in common, with an emphasis on a computational view of emotion. Despite some anxiety in the organizers when there were only about 40 submissions two days before the deadline, in the end we received a record high number of articles: in total, we received 268 submissions (175 regular papers and 93 papers for workshops, the Doctoral Consortium, and demos). In keeping with ACII’s rigorous standards, each paper was reviewed by at least two experts in various subdomains (most papers received three reviews) and vetted by members of the Senior Program Committee and organizers of various events. 55 out of the 175 regular papers were accepted as oral presentations (31 percent acceptance rate), and an additional 48 papers were accepted for poster presentations (an overall acceptance rate of 59 percent). These figures clearly establish the prominent role of ACII in the field.

WE are delighted to introduce this special section featuring expanded versions of the best papers from the 6th Biannual International Conference on Affective Computing and Intelligent Interaction (ACII 2013), held from September 2-5, 2013, in Geneva, Switzerland. The ACII conference series is the flagship conference of the Association for the Advancement of Affective Computing (aaac—formerly the Humaine Association) and is the premier international forum for all areas of affective computing research. The ACII 2013 conference received 175 full-paper submissions, which were rigorously reviewed by members of the Program Committee and further vetted by a Senior Program Committee and the Program Chairs. A total of 55 papers were accepted for oral-presentation at the conference. These 55 papers were ranked based on the scores received during the review process and this special section features enhanced versions of five of the top-ranked papers, which went through additional round(s) of review/revision.

Slow Feature Analysis (SFA) is a subspace learning method inspired by the human visual system, however, it is seldom seen in computer vision. Motivated by its application for unsupervised activity analysis, we develop SFA’s first implementation of online temporal video segmentation to detect episodes of motion changes. We utilize a domainspecific indefinite kernel which takes the data representation into account to introduce robustness. As our kernel is indefinite (i.e. defines instead of a Hilbert, a Krein space), we formulate SFA in Krein space. We propose an incremental kernel SFA framework which utilizes the special properties of our kernel. Finally, we employ our framework to online temporal video segmentation and perform qualitative and quantitative evaluation.

In this study, we investigate an audiovisual approach for classification of vocal outbursts (non-linguistic vocalisations) in noisy conditions using Long Short-Term Memory (LSTM) Recurrent Neural Networks and Support Vector Machines. Fusion of geometric shape features and acoustic low-level descriptors is performed on the feature level. Three different types of acoustic noise are considered: babble, office and street noise. Experiments are conducted on every noise type to asses the benefit of the fusion in each case. As database for evaluations serves the INTERSPEECH 2010 Paralinguistic Challenge’s Audiovisual Interest Corpus of human-to-human natural conversation. The results show that even when training is performed on noise corrupted audio which matches the test conditions the addition of visual features is still beneficial.

During face-to-face interpersonal interaction people have a tendency to mimic each other, that is, they change their own behaviors to adjust to the behavior expressed by a partner. In this paper we describe how behavioral information expressed between two interlocutors can be used to detect and identify mimicry and improve recognition of interrelationship and affect between them in a conversation. To automatically analyze how to extract and integrate this behavioral information into a mimicry detection framework for improving affective computing, this paper addresses the main challenge: mimicry representation in terms of optimal behavioral feature extraction and automatic integration.

During face-to-face interpersonal interaction, people have a tendency to mimic each other. People not only mimic postures, mannerisms, moods or emotions, but they also mimic several speech-related behaviors. In this paper we describe how visual and vocal behavioral information expressed between two interlocutors can be used to detect and identify visual and vocal mimicry. We investigate expressions of mimicry and aim to learn more about in which situation and to what extent mimicry occurs. The observable effects of mimicry can be explored by representing and recognizing mimicry using visual and vocal features. In order to automatically analyze how to extract and integrate this behavioral information into a multimodal mimicry detection framework for improving affective computing, this paper addresses the main challenge: mimicry representation in terms of optimal behavioral feature extraction and automatic integration in both audio and video modalities.

Applications such as surveillance and human behaviour analysis require high-bandwidth recording from multiple cameras, as well as from other sensors. In turn, sensor fusion has increased the required accuracy of synchronisation between sensors. Using commercial off-the-shelf components may compromise quality and accuracy due to several challenges, such as dealing with the combined data rate from multiple sensors; unknown offset and rate discrepancies between independent hardware clocks; the absence of trigger inputs or -outputs in the hardware; as well as the different methods for time-stamping the recorded data. To achieve accurate synchronisation, we centralise the synchronisation task by recording all trigger- or timestamp signals with a multi-channel audio interface. For sensors that don't have an external trigger signal, we let the computer that captures the sensor data periodically generate timestamp signals from its serial port output. These signals can also be used as a common time base to synchronise multiple asynchronous audio interfaces. Furthermore, we show that a consumer PC can currently capture 8-bit video data with 1024 × 1024 spatial- and 59.1 Hz temporal resolution, from at least 14 cameras, together with 8 channels of 24-bit audio at 96 kHz. We thus improve the quality/cost ratio of multi-sensor systems data capture systems.

In this paper, we present a novel software tool designed and implemented to simplify the development process of Multimodal Human-Computer Interaction (MHCI) systems. This tool, which is called the HCI^2 Workbench, exploits a Publish / Subscribe (P/S) architecture [13] [14] to facilitate efficient and reliable inter-module data communication and runtime system management. In addition, through a combination of SDK, software tools, and standardized description / configuration file semantics, the HCI^2 Workbench provides an easy-to-follow procedure for developing highly flexible and reusable modules. Moreover, the HCI^2 Workbench features system persistence and portability by using standardized module packaging method and system configuration files. Last but not least, usability was another major concern. Unlike other similar tool, including Psyclone [9] and ActiveMQ [10], the HCI^2 Workbench provides a complete graphical environment to support every step in a typical MHCI system development process, including module program development and debugging, module packaging, module management, system configuration, module and system testing, in a convenient and intuitive manner. To help demonstrating the HCI^2 Workbench, we also present a readily-applicable system developed using our tool. This open-source demo system, which is called the CamGame, consists of an interactive system allowing users to play a computer game using hand-held marker(s) and low-cost camera(s) instead of keyboard and mouse.

The exploration of how we react to the world and interact with it and each other remains one of the greatest scientific challenges. Latest research trends in cognitive sciences argue that our common view of intelligence is too narrow, ignoring a crucial range of abilities that matter immensely for how people do in life. This range of abilities is called social intelligence and includes the ability to express and recognise social signals produced during social interactions like agreement, politeness, empathy, friendliness, conflict, etc., coupled with the ability to manage them in order to get along well with others while winning their cooperation. Social Signal Processing (SSP) is the new research domain that aims at understanding and modelling social interactions (human-science goals), and at providing computers with similar abilities in human-computer interaction scenarios (technological goals). SSP is in its infancy, and the journey towards artificial social intelligence and socially-aware computing is still long. This research agenda is a twofold, a discussion about how the field is understood by people who are currently active in it and a discussion about issues that the researchers in this formative field face.

Human affective behavior is multimodal, continuous and complex. Despite major advances within the affective computing research field, modeling, analyzing, interpreting and responding to human affective behavior still remains a challenge for automated systems as affect and emotions are complex constructs, with fuzzy boundaries and with substantial individual differences in expression and experience [7]. Therefore, affective and behavioral computing researchers have recently invested increased effort in exploring how to best model, analyze and interpret the subtlety, complexity and continuity (represented along a continuum e.g., from −1 to +1) of affective behavior in terms of latent dimensions (e.g., arousal, power and valence) and appraisals, rather than in terms of a small number of discrete emotion categories (e.g., happiness and sadness). This chapter aims to (i) give a brief overview of the existing efforts and the major accomplishments in modeling and analysis of emotional expressions in dimensional and continuous space while focusing on open issues and new challenges in the field, and (ii) introduce a representative approach for multimodal continuous analysis of affect from voice and face, and provide experimental results using the audiovisual Sensitive Artificial Listener (SAL) Database of natural interactions. The chapter concludes by posing a number of questions that highlight the significant issues in the field, and by extracting potential answers to these questions from the relevant literature. The chapter is organized as follows. Section 10.2 describes theories of emotion, Sect. 10.3 provides details on the affect dimensions employed in the literature as well as how emotions are perceived from visual, audio and physiological modalities. Section 10.4 summarizes how current technology has been developed, in terms of data acquisition and annotation, and automatic analysis of affect in continuous space by bringing forth a number of issues that need to be taken into account when applying a dimensional approach to emotion recognition, namely, determining the duration of emotions for automatic analysis, modeling the intensity of emotions, determining the baseline, dealing with high inter-subject expression variation, defining optimal strategies for fusion of multiple cues and modalities, and identifying appropriate machine learning techniques and evaluation measures. Section 10.5 presents our representative system that fuses vocal and facial expression cues for dimensional and continuous prediction of emotions in valence and arousal space by employing the bidirectional Long Short-Term Memory neural networks (BLSTM-NN), and introduces an output-associative fusion framework that incorporates correlations between the emotion dimensions to further improve continuous affect prediction. Section 10.6 concludes the chapter.

Human mimicry is one of the important behavioral cues displayed during social interaction that inform us about the interlocutors' interpersonal states and attitudes. For example, the absence of mimicry is usually associated with negative attitudes. A system capable of analyzing and understanding mimicry behavior could enhance social interaction, both in human-human and human-machine interaction, by informing the interlocutors about each other's interpersonal attitudes and feelings of affiliation. Hence, our research focus is the investigation of mimicry in social human-human and human-machine interactions with the aim to help improve the quality of these interactions. In particular, we aim to develop automatic multimodal mimicry analyzers, to enhance affect recognition and social signal understanding systems through mimicry analysis, and to implement mimicry behavior in Embodied Conversational Agents. This paper surveys and discusses the recent work we have carried out regarding these aims. It is meant to serve as an ultimate goal and a guide for determining recommendations for the development of automatic mimicry analyzers to facilitate affective computing and social signal processing.

We present a novel framework for the recognition of facial expressions at arbitrary poses that is based on 2D geometric features. We address the problem by first mapping the 2D locations of landmark points of facial expressions in non-frontal poses to the corresponding locations in the frontal pose. Then, recognition of the expressions is performed by using any state-of-the-art facial expression recognition method (in our case, multi-class SVM). To learn the mappings that achieve pose normalization, we use a novel Gaussian Process Regression (GPR) model which we name Coupled Gaussian Process Regression (CGPR) model. Instead of learning single GPR model for all target pairs of poses at once, or learning one GPR model per target pair of poses independently of other pairs of poses, we propose CGPR model, which also models the couplings between the GPR models learned independently per target pairs of poses. To the best of our knowledge, the proposed method is the first one satisfying all: (i) being face-shape-model-free, (ii) handling expressive faces in the range from -45° to +45° pan rotation and from -30° to +30° tilt rotation, and (iii) performing accurately for continuous head pose despite the fact that the training was conducted only on a set of discrete poses.

This paper aims to give a brief overview of the current state-of-the-art in automatic measurement of affect signals in dimensional and continuous spaces (a continuous scale from -1 to +1) by seeking answers to the following questions: i) why has the field shifted towards dimensional and continuous interpretations of affective displays recorded in real-world settings? ii) what are the affect dimensions used, and the affect signals measured? and iii) how has the current automatic measurement technology been developed, and how can we advance the field?

In this issue, “Best of the Web‿ focuses on introducing the social signal processing network (SSPNet), a large European collaboration aimed at establishing a research community in social signal processing (SSP), the new, emerging domain aimed at bringing social intelligence in computers.

This paper focuses on dimensional prediction of emotions from spontaneous conversational head gestures. It maps the amount and direction of head motion, and occurrences of head nods and shakes into arousal, expectation, intensity, power and valence level of the observed subject as there has been virtually no research bearing on this topic. Preliminary experiments show that it is possible to automatically predict emotions in terms of these five dimensions (arousal, expectation, intensity, power and valence) from conversational head gestures. Dimensional and continuous emotion prediction from spontaneous head gestures has been integrated in the SEMAINE project [1] that aims to achieve sustained emotionally-colored interaction between a human user and Sensitive Artificial Listeners.

Tagging is the annotation of multimedia data with userspecified keywords known as tags, with the aim of facilitating fast and accurate data retrieval based on these tags. In contrast to this process, also referred to as explicit tagging, implicit human-centered tagging (IHCT) refers to exploiting the information on user's nonverbal reactions (e.g., facial expressions like smiles or head gestures like shakes) to multimedia data, with which he or she interacts, to assign new or improve the existing tags associated with the target data. Thus, implicit tagging allows that a data item gets tagged each time a user interacts with it based on the reactions of the user to the data (e.g., laughter when seeing a funny video), in contrast to explicit tagging paradigm in which a data item gets tagged only if a user is requested (or chooses) to associate tags with it. As nonverbal reactions to observed multimedia are displayed naturally and spontaneously, no purposeful explicit action (effort) is required from the user; hence, the resulting tagging process is said to be "implicit" and "human centered" (in contrast to being dictated by computer and being "computer-centered").

It is a pleasure and an honor to have organized the Third International Conference on Affective Computing and Intelligent Interaction (ACII). The conference will be held from 10th – 12th September 2009 in Amsterdam, The Netherlands. The conference series is the premier forum for presenting research in affective computing and related topics. The conference is organized under the auspices of Humaine Association, and continues the tradition of the previous conferences by maintaining the high standard set by its predecessors.

It is a pleasure and an honor to have organized the Third International Conference on Affective Computing and Intelligent Interaction (ACII). The conference will be held from 10th – 12th September 2009 in Amsterdam, The Netherlands. The conference series is the premier forum for presenting research in affective computing and related topics. The conference is organized under the auspices of Humaine Association, and continues the tradition of the previous conferences by maintaining the high standard set by its predecessors. The conference program includes oral papers, posters, invited presentations, sessions on special topics, mini-tutorials, doctoral consortium, demos, and pre- and post-conference workshops. For the conference, we received 157 contributed papers for review. Each of these was assessed by no fewer than two reviewers, with a large majority of papers being assessed by three reviewers; 60 submissions (38%) were selected for oral presentation and 33 submissions for poster presentation (an overall acceptance rate of 59%). These papers will be accompanied by 3 invited presentations, by Qiang Ji (Rensselaer Polytechnic Institute, US); Klaus Scherer (University of Geneva, Switzerland), and Justine Cassell (Northwestern University, US). Three mini tutorials will address a number of challenging issues in the field including modeling emotions in game characters, mental state modeling for brain-computer interfaces, and preference learning for affective modeling. Four special sessions will highlight new trends and challenges in affective computing and intelligent interaction including recognizing non-prototypic emotions from speech, processing real-life affective signals, analyzing emotional content of music and the musical performance, and developing games for training and entertainment (sponsored by the GATE project). A panel session will address one more novel and important challenge – social interaction with robots and virtual agents. The technical program will also present 15 stunning interactive demos. The Doctoral Consortium program consists of 13 student paper presentations. The pre- and post-conference workshops will bring together behavioral, neuro- and computer scientists to purse links between automatic human behavior recognition and related sciences and applications. The Workshop on Affective Brain-Computer Interfaces explores advantages and limitations of using neuro-physiological signals for automatic affect recognition and related applications. The Workshop on Social Signal Processing explores cognitive models, automatic analysis, and machine synthesis of social signals, and possibilities for their usage towards the realization of socially-aware computing (sponsored by the SSPNet project, [FP7/2007-2013] grant agreement 231287). The Behavior Markup Language Workshop will discuss open issues in defining the “core��? of BML. The oral presentations, posters, invited addresses, and workshops bring together related communities to share the latest findings and ideas and pursue continuing and new collaborations in research on affective computing and intelligent interaction. By papers, location, and content, this promises to be an excellent edition of the Affective Computing and Intelligent Interaction conference series. The conference continues to provide a leading forum for cutting-edge research in human affect modeling, affective computing, and intelligent interaction. We wish all delegates a most enjoyable and productive conference.

Embodied Conversational Agents can be viewed as spoken dialogue systems with a graphical representation of a human body. But the embodiment is not the only difference. Whereas Spoken Dialogue Systems are mostly focused on computing the linguistic dimensions of communication, conversational agents are conceived as intelligent agents that have an identity, a persona. Thus, cognitive modeling is often more involved in ECAs including the modeling of emotion. Whereas spoken dialogue systems are focused on the task, virtual humans are also equipped with social skills involved in interaction. This can take various forms. In this paper we review some of the approaches that have been taken in the first decade of ECA research, by presenting the social signaling skills of three agents we have developed in our group.

We introduce and test an affective music player (AMP) that selects music for mood enhancement. Through a concise overview of content, construct, and ecological validity, we elaborate five considerations that form the foundation of the AMP. Based on these considerations, computational models are developed, using regression and kernel density estimation. We show how these models can be used for music selection and how they can be extended to fit in other systems. Subsequently, the success of the models is illustrated with a user test. The AMP augments music listening, where its techniques, in general, enable automated affect guidance. Finally, we argue that our AMP is readily applicable to real-world situations as it can 1) cope with noisy situations, 2) handle the large inter-individual differences apparent in the musical domain, and 3) integrate context or other information, all in real-time.

This paper describes work toward an affective music player (AMP), which is able to direct affect to a goal state by selecting music. Repeatedly, music has been shown to modulate affect; however, precise guidelines for the use of music characteristics in an AMP have not been defined. To explore these, we investigated the influence of music characteristics on 32 participants who listened to 16 songs, testing effects of tempo (slow/high), mode (minor/major), and per-cussiveness (low/high). Subjective measures of affect (i.e., arousal, tension, and positive and negative valence) and physiology (i.e., skin conductance level, skin conductance responses, and heart rate variability) were measured during listening. Results show main and interaction effects of music characteristics on both subjective affect and physiology, implying that the characteristics are mutually dependent in modulating affect. Based on these results, guidelines are presented for AMPs, which can effectively direct affect through music.

This article presents the rationale behind ACII2009’s special session: Guidelines for Affective Signal Processing (ASP): From lab to life. Although affect is embraced by both science and engineering, its recognition has not reached a satisfying level. Through a concise overview of ASP and the automatic classification of affect, we provide understanding for the problems encountered. Next, we identify guidelines for ASP: 1) four approaches to validation: content, criteria-related, construct, and ecological, 2) identification of users, 3) triangulation, and 4) signal processing issues. Each of these guidelines is briefly touched upon in this paper. A more exhaustive discussion on these guidelines, in perspective of the invited speakers’ experience, will be provided through the session and its accompanying papers.

This paper introduces social signal processing (SSP), the domain aimed at automatic understanding of social interactions through analysis of nonverbal behavior. The core idea of SSP is that nonverbal behavior is machine detectable evidence of social signals, the relational attitudes exchanged between interacting individuals. Social signals include (dis-)agreement, empathy, hostility, and any other attitude towards others that is expressed not only by words but by nonverbal behaviors such as facial expression and body posture as well. Thus, nonverbal behavior analysis is used as a key to automatic understanding of social interactions. This paper presents not only a survey of the related literature and the main concepts underlying SSP, but also an illustrative example of how such concepts are applied to the analysis of conflicts in competitive discussions.

Facial expression recognition is a process performed by humans or computers, which consists of: 1. Locating faces in the scene (e.g., in an image; this step is also referred to as face detection), 2. Extracting facial features from the detected face region (e.g., detecting the shape of facial components or describing the texture of the skin in a facial area; this step is referred to as facial feature extraction), 3. Analyzing the motion of facial features and/or the changes in the appearance of facial features and classifying this information into some facialexpression-interpretative categories such as facial muscle activations like smile or frown, emotion (affect) categories like happiness or anger, attitude categories like (dis)liking or ambivalence, etc. (this step is also referred to as facial expression interpretation).

The seven articles in this special issue focus on human computing. Most focus on two challenging issues in human computing, namely, machine analysis of human behavior in group interactions and context-sensitive modeling.

Here is a conversation between an interviewer and a subject occurring in an Adult Attachment Interview (Roisman, Tsai, & Chiang, 2004). AUs are facial action units defined in Ekman, Friesen, and Hager (2002). The interviewer asked: “Now, let you choose five adjective words to describe your childhood relationship with your mother when you were about five years old, or as far back as you remember.��? The subject kept smiling (lip corner raiser AU12) when listening. After the interviewer finished the question, the subject looked around and lowered down her head (AU 54) and eyes (AU 64). Then she lowered and drew together the eyebrows (AU4) so that severe vertical wrinkles and skin bunching between the eyebrows appeared. Then her left lip raise[d] (Left AU10), and finger scratched chin. After about 50 second silence, the subject raise her head (AU53) and brow (AU1+AU2), and asked with a smile (AU12): “Should I . . . give what I have now?��? The interviewer response with smiling (AU12): “I guess, those will be when you were five years old. Can you remember?��? The subject answered with finger touching chin: “Yeap. Ok. Happy (smile, AU6+AU12), content, dependent, (silence, then lower her voice) what is next (silent, AU4+left AU 10), honest, (silent, AU 4), innocent.��?

One interest of our research group is in (1) developing Ambient Entertainment technologies and applications that (2) interact in a coordinated way with human partners using (3) a multitude of different sensors observing many characteristics of the partner and using (4) many different channels of expression (such as sound, visuals, speech and embodiment with gestures and facial expressions). The Virtual Conductor project [1, 2] concerns the development of the first properly interactive virtual orchestra conductor that can conduct a piece of music through interaction with musicians, leading and following them while they are playing. Its observations consist of different manners of musical processing of the incoming signal from a microphone. The forms of its expressions are defined by the possible conducting patterns and by the timing, speed, amplitude and smoothness with which those patterns are expressed. The interaction is focused on the tempo of the musicians and includes a correction module that interactively corrects the tempo when the musicians are playing too slow or too fast. We describe our motivation for developing such a system; related work in the areas of ambient entertainment and coordinated timing, automatic music processing, virtual humans and conducting; the design, implementation and evaluation of the Virtual Conductor; and, finally, contains a discussion of the resulting system, general applicability of the ideas and technology developed in this project, and expected developments in the (ongoing) Virtual Conductor project.

This book contains the proceedings of the 20th edition of the Belgian-Netherlands Conference on Artificial Intelligence. The conference was organized by the Human Media Interaction group of the University of Twente. As usual, the conference was under the auspices of the Belgian-Dutch Association for Artificial Intelligence (BNVKI) and the Dutch Research School for Information and Knowledge Systems (SIKS). The conference aims at presenting an overview of state-of-the-art research in artificial intelligence in Belgium and the Netherlands, but does not exclude contributions from other countries. The received submissions show that AI researchers in Belgium and the Netherlands continue to work actively in many different areas of artificial intelligence and are open for new developments in technology and society. The annual BNAIC conference is the main meeting place for artificial intelligence researchers and practioners in Belgium and the Netherlands. Therefore we did not change the tradition that beside the sessions with accepted regular papers describing original work, there are also short papers describing work published elsewhere and papers describing demonstrations. We received 108 submissions consisting of 44 regular full papers, 53 short papers, and 11 system demonstrations. We are grateful to the programme committee members who carefully reviewed all submissions. A small committee chaired by the conference chairs made the final decisions. The acceptance rate of the regular papers was 80%. Of the short papers 75% was presented in oral sessions, the others were presented in poster sessions.

Many interesting phenomena in conversations require interpretative judgements by the annotators. This leads to data which is annotated with lower levels of agreement due to the differences in how annotators interpret conversations. Instead of throwing away this data we show how and when we can exploit it. We analyse the (dis)agreements between annotators for two different cases in a multimodal annotated corpus and explicitly relate the results to the way machine-learning algorithms perform on the annotated data.

Unobtrusive and multiple-sensor interfaces enable observation of natural human behavior in smart environments. Being able to detect, analyze and interpret this activity allows for the implementation of various applications, including real-time surveillance and real-time support in smart home and office environments. We focus on smart meeting environments, where nonverbal behavioral cues sometimes tell more about issues such as discussion participation, involvement and contribution, than information obtained from verbal contributions. An important aspect of this behavior is the interaction between meeting participants. We regard the special case where some of the participants are at physically different locations, which hinders natural interaction. We discuss how we can exploit the ability of a sensor-equipped environment to detect nonverbal interaction cues and use these to allow and improve natural interaction between collaborating participants at distributed locations. We focus on research efforts to detect nonverbal interaction cues by looking at various modalities. Also, we discuss how information obtained from the fusion of these modalities allows us to generate and display behavioral cues that allow remote participants to take part in a distributed meeting in a natural way.

In this workshop of the 8th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2008), the emphasis is on research on facial and bodily expressions for the control and adaptation of games [3]. We distinguish between two forms of expressions, depending on whether the user has the initiative and consciously uses his or her movements and expressions to control the interface, or whether the application takes the initiative to adapt itself to the affective state of the user as it can be interpreted from the user’s expressive behavior.

No abstract.

This book constitutes the thoroughly refereed post-proceedings of two events discussing AI for Human Computing: one Special Session during the Eighth International ACM Conference on Multimodal Interfaces (ICMI 2006), held in Banff, Canada, in November 2006, and a Workshop organized in conjunction with the 20th International Joint Conference on Artificial Intelligence (IJCAI 2007), held in Hyderabad, India, in January 2007. A large number of the contributions in this state-of-the-art survey are updated and extended versions of the papers presented during these two events. In order to obtain a more complete overview of research efforts in the field of human computing, a number of additional invited contributions are also included in this book on AI for human computing. The 17 revised papers presented were carefully selected from numerous submissions to and presentations made at the two events and include invited articles to round off coverage of all relevant topics of the emerging topic. The papers are organized in three parts: a part on foundational issues of human computing, a part on sensing humans and their activities, and a part on anthropocentric interaction models.

Ambient intelligence promises to enable humans to smoothly interact with their environment, mediated by computer technology. In the literature on ambient intelligence, empirical scientists are not often mentioned. Yet they form an interesting target group for this technology. In this position paper, we describe a project aimed at realising an ambient intelligence environment for face-to-face meetings of researchers with different academic backgrounds involved in molecular biology “omics��? experiments. In particular, microarray experiments are a focus of attention because these experiments require multidisciplinary collaboration for their design, analysis, and interpretation. Such an environment is characterised by a high degree of complexity that has to be mitigated by ambient intelligence technology. By experimenting in a real-life setting, we will learn more about life scientists as a user group.

Virtual Humans are on the border of fiction and realism: while it is obvious that they do not exist in reality and function on different principles than real people, they have been endowed with human features such as being emotionally sensitive. In this article we argue that many dimensions, both human-like ones and ones made possible by the computer technology, are still unexploited to increase the effectivity and engagement of interaction with VHs.

This paper describes an intelligent system that we developed to support affective multimodal human-computer interaction (AMM-HCI) where the user’s actions and emotions are modeled and then used to adapt the interaction and support the user in his or her activity. The proposed system, which we named Gaze-X, is based on sensing and interpretation of the human part of the computer’s context, known as W5+ (who, where, what, when, why, how). It integrates a number of natural human communicative modalities including speech, eye gaze direction, face and facial expression, and a number of standard HCI modalities like keystrokes, mouse movements, and active software identification, which, in turn, are fed into processes that provide decision making and adapt the HCI to support the user in his or her activity according to his or her preferences. A usability study conducted in an office scenario with a number of users indicates that Gaze-X is perceived as effective, easy to use, useful, and affectively qualitative.