Search

Emotion recognition is usually achieved by collecting features (physiological signals, events, facial expressions, etc.) to predict an emotional ground truth. This ground truth is arguably unreliable due to its subjective nature. In this paper, we introduce a new approach to measure the magnitude of an emotion in the latent space of a Neural Network without the need for a subjective ground truth. Our data consists of physiological measurements during video gameplay, game events, and subjective rankings of game events for the validation of our model. Our model encodes physiological features into a latent variable which is then decoded into video game events. We show that the events are ranked in the latent space similarly to the participants' subjective ranks. For instance, our model's ranking is correlated (Kendall $\tau$ of 0.91) with the predictability rankings.

Horror games represent a very specific genre specifically designed to elicit fear. These games provide a tremendous emotional experience balanced between stress and satisfaction. Yet, over time, the player acquires further insight of the mechanisms of the game, dissipating the creepy climate that reduces the emotional impact intended. This article hypothesises that exploring existing facets within a game such as visuals and sounds might establish a good approach to renew the gaming experience. To understand the players’ emotional reactions toward context alteration, an adaptation of a published game ( P.T. by Konami 2014) was used. This context refers mainly to light effects, sounds, and in-game events. To learn which game effects induce the strongest physiological reactions, an experiment was conducted, and correlation between the physiological data, collected through the measure of the galvanic skin response, and that of the perceived emotion provided by participants, was investigated. Results show that the order in which effects are arranged can produce extensive emotional responses. They also suggest that psychological impact can be increased not only by the visual horror itself, but also through the process that slowly builds up to it, in particular the usage of sounds.

The role of emotions in (computer-supported) collaborative learning has become an important research topic. However, research has mainly focused on the relationships between emotions and socio-emotional processes. Despite an extensive literature on the role of emotions in cognitive processes, socio-cognitive processes have been scarcely explored at this point. This article aims to investigate, using control-value theory (Pekrun, 2006) as a frame of reference, how task-achievement appraisals impact both self-experienced emotions and the perception of socio-cognitive processes. An experimental study involving 28 pairs of participants playing a collaborative problem-solving computer puzzle game was carried out. Depending on the experimental conditions, participants received different false feedback about group task mastery and ranking aiming at skewing in different ways how they appraised task achievement. At the end of the task, they reported self and partner's emotions as well as their perception of their socio-cognitive exchanges. The results indicate that 1) participants experienced a large variety of achievement emotions and some of them appear more closely related to specific task-achievement appraisals, 2) achievement emotions do not mediate the relationship between task-achievement appraisals and the perception of socio-cognitive processes, 3) the level of self-experienced activation as well as the perception of activation in the partner mediate the relationship between task-achievement appraisals and the perception of socio-cognitive processes and could be used as a heuristic of socio-cognitive collaborative involvement by problem solvers, 4) partner's emotions such as gratitude appear to increase the perception of socio-cognitive processes through the reinforcement of group mastery goal.

Challenging players is a fundamental element when designing games, but finding the perfect balance between a frustrating and boring experience can become a challenge in itself. This paper proposes the usage of deep data-driven affective models capable of detecting anxiety and boredom from raw human physiological signals to adapt the fluctuation of difficulty in a game of Tetris. The first phase of this work was to construct several emotion detection models for performance comparison, where the most accurate model achieved an average accuracy of \(73.2\%\). A user evaluation was subsequently conducted on a total of 56 participants to validate the efficiency of the most accurate model obtained using two diverging difficulty adaptation types. One method adapts difficulty according to raw values directly outputted by the affective model (ABS), while another compares the current value with the previous one to influence difficulty (REA). Results show that the model using the ABS adaptation was capable of effectively adjusting the overall difficulty based on a player’s physiological state during a game of Tetris.

Formed impressions are crucial for human-human interaction (e.g. a job interview) and an interaction with a virtual agent/robot, since they can impact people's perceptions and willingness to be involved in the interaction. There are studies on how facial features (e.g. skin color, face shape), acoustic signals and non-verbal behaviors (e.g. gestures, postures) create/leave certain impressions. However there is little research focusing on how our bodies disclose our already formed impression of someone. Forming an impression leads to emotions and behaviors which can be measured. In this paper, we investigate recognition of evoked impression of warmth and competence from the non- verbal behaviors expressed by the person forming the impression. We conducted an experiment in which participants were watching impression stimuli. We measured participant's facial expressions, eye movements and physiological reactions (electrocardiography and galvanic skin response). To recognize impressions, we tested 2 multivariate regression models with the aforementioned mul- timodal recordings. Our best results demonstrate the possibility to detect impressions along warmth and competence dimensions with a concordance correlation coefficient of 0.838 and 0.864. Facial expressions and eye movements are more reliable for impression detection compared with physiological signals. Finally, the higher the Berkeley emotion expressivity scores the partic- ipants have, the more accurately the impressions are detected.

Recognizing emotional reactions of movie audiences to affective movie content is a challenging task in affective computing. Previous research on induced emotion recognition has mainly focused on using audio-visual movie content. Nevertheless, the relationship between the perceptions of the affective movie content (perceived emotions) and the emotions evoked in the audiences (induced emotions) is unexplored. In this work, we studied the relationship between perceived and induced emotions of movie audiences. Moreover, we investigated multimodal modelling approaches to predict movie induced emotions from movie content based features, as well as physiological and behavioral reactions of movie audiences. To carry out analysis of induced and perceived emotions, we first extended an existing database for movie affect analysis by annotating perceived emotions in a crowd-sourced manner. We find that perceived and induced emotions are not always consistent with each other. In addition, we show that perceived emotions, movie dialogues, and aesthetic highlights are discriminative for movie induced emotion recognition besides spectators’ physiological and behavioral reactions. Also, our experiments revealed that induced emotion recognition could benefit from including temporal information and performing multimodal fusion. Moreover, our work deeply investigated the gap between affective content analysis and induced emotion recognition by gaining insight into the relationships between aesthetic highlights, induced emotions, and perceived emotions.

The papers in this special section focus on commercial applications for affective computing. One of the main goals of affective computing is to create machines which can adapt to users’ emotions in order to produce more natural and efficient interaction. Emotion recognition is thus a central component of the field, and is based on a variety of measurements (facial expressions, speech, gait patterns, physiology, eye tracking, etc.) that are analyzed using advanced pattern recognition techniques. Furthermore, researchers and entrepreneurs have identified countless possible applications of affect-aware technology, from health and driver monitoring to exercise [6] and computer game adaptation. However, although great scientific advances have been made and many applications have been proposed, only few robust implementations have been presented or validated, and commercial adoption of affect-aware technology has been marginal. One rare example of a successful application is the smile detector used in digital cameras to automatically take pictures when the subject is smiling. This weak adoption of the technology can be attributed to several unsolved challenges in the domain of affective computing.

Predicting the emotional response of movie audi- ences to affective movie content is a challenging task in affective computing. Previous work has focused on using audiovisual movie content to predict movie induced emotions. However, the relationship between the audience’s perceptions of the affective movie content (perceived emotions) and the emotions evoked in the audience (induced emotions) remains unexplored. In this work, we address the relationship between perceived and in- duced emotions in movies, and identify features and modelling approaches effective for predicting movie induced emotions. First, we extend the LIRIS-ACCEDE database by annotating perceived emotions in a crowd-sourced manner, and find that perceived and induced emotions are not always consistent. Second, we show that dialogue events and aesthetic highlights are effective predictors of movie induced emotions. In addition to movie based features, we also study physiological and be- havioural measurements of audiences. Our experiments show that induced emotion recognition can benefit from including temporal context and from including multimodal information. Our study bridges the gap between affective content analysis and induced emotion prediction.

Biased feedbacks of control and value were used to influence the emotional appraisal during a collaborative problem-solving game. We investigated how these feedbacks modulate the intensity of experienced emotions as well as the relations between emotions, perceived collaboration and group performance. Different correlational patterns between emotions, socio-cognitive processes and performance occurred depending on the perceived control and value., Des feedbacks biaisés de contrôle et de valeur ont été utilisés pour influencer l’évaluation émotionnelle pendant un jeu collaboratif de résolution de problème. Nous avons étudié comment ces feedbacks ont modulé l’intensité des émotions ressenties ainsi que les relations entre les émotions, la collaboration perçue et la performance du groupe. Les résultats montrent des patterns de corrélations entre émotions, processus socio-cognitifs et performance différents en fonction des perceptions de contrôle et de valeur., Avry Sunny, Chanel Guillaume, Bétrancourt Mireille, Pun Thierry, Molinari Gaëlle. Effet des antécédents émotionnels de contrôle et de valeur sur la résolution de problème dans un jeu vidéo collaboratif. In: Sciences et Technologies de l'Information et de la Communication pour l'Éducation et la Formation, volume 25 n°1, 2018. Numéro spécial : Sélection de la conférence EIAH 2017. pp. 195-220.

Detection of aesthetic highlights is a challenge for understanding the affective processes taking place during movie watching. In this article, we study spectators’ responses to movie aesthetic stimuli in a social context. Moreover, we look for uncovering the emotional component of aesthetic highlights in movies. Our assumption is that synchronized spectators’ physiological and behavioral reactions occur during these highlights because: ( i ) aesthetic choices of filmmakers are made to elicit specific emotional reactions (e.g., special effects, empathy, and compassion toward a character) and ( ii ) watching a movie together causes spectators’ affective reactions to be synchronized through emotional contagion. We compare different approaches to estimation of synchronization among multiple spectators’ signals, such as pairwise, group, and overall synchronization measures to detect aesthetic highlights in movies. The results show that the unsupervised architecture relying on synchronization measures is able to capture different properties of spectators’ synchronization and detect aesthetic highlights based on both spectators’ electrodermal and acceleration signals. We discover that pairwise synchronization measures perform the most accurately independently of the category of the highlights and movie genres. Moreover, we observe that electrodermal signals have more discriminative power than acceleration signals for highlight detection.

Over the last years, affective computing has been strengthening its ties with the humanities, exploring and building understanding of people’s responses to specific artistic multimedia stimuli. “Aesthetic experience” is acknowledged to be the subjective part of some artistic exposure, namely, the inner affective state of a person exposed to some artistic object. In this work, we describe ongoing research activities for studying the aesthetic experience of people when exposed to movie artistic stimuli. To do so, this work is focused on the definition of emotional and aesthetic highlights in movies and studies the people responses to them using physiological and behavioral signals, in a social setting. In order to examine the suitability of multimodal signals for detecting highlights, we initially evaluate a supervised highlight detection system. Further, in order to provide an insight on the reactions of people, in a social setting, during emotional and aesthetic highlights, we study two unsupervised systems. Those systems are able to (a) measure the distance among the captured signals of multiple people using the dynamic time warping algorithm and (b) create a reaction profile for a group of people that would be indicative of whether that group reacts or not at a given time. The results indicate that the proposed systems are suitable for detecting highlights in movies and capturing some form of social interactions across different movie genres. Moreover, similar social interactions during exposure to emotional and some types of aesthetic highlights, such as those corresponding to technical or lightening choices of the director, can be observed. The utilization of electrodermal activity measurements yields in better performances than those achieved when using acceleration measurements, whereas fusion of the modalities does not appear to be beneficial for the majority of the cases.

This paper studies emotion recognition in the context of collaboration. When people are interacting with each other they tend to reach a similar emotional state through mechanisms like empathy and emotion contagion. We thus investigated if participants' emotions could be determined from the affective reactions and behaviors of their partner. Two types of emotional expressions were studied: physiological reactions and speech. Results show that emotions could be recognized with similar performance when employing affective features from the self or the partner. In addition, performance was improved when combining self and partner information. The results demonstrate that in social situations an emotion recognition model should include information about partners.

Physiological response is an important component of an emotional episode. In this paper, we introduce a Toolbox for Emotional feAture Extraction from Physiological signals (TEAP). This open source toolbox can preprocess and calculate emotionally relevant features from multiple physiological signals, namely, electroencephalogram (EEG), galvanic skin response (GSR), electromyogram (EMG), skin temperature, respiration pattern, and blood volume pulse. The features from this toolbox are tested on two publicly available databases, i.e., MAHNOB-HCI and DEAP. We demonstrate that we achieve similar performance to the original work with the features from this toolbox. The toolbox is implemented in MATLAB and is also compatible with Octave. We hope this toolbox to be further developed and accelerate research in affective physiological signal analysis.

Detection of emotional and aesthetic highlights is a challenge for the affective understanding of movies. Our assumption is that synchronized spectators' physiological and behavioral reactions occur during these highlights. We propose to employ the periodicity score to capture synchronization among groups of spectators' signals. To uncover the periodicity score's capabilities, we compare it with baseline synchronization measures, such as the nonlinear interdependence and the windowed mutual information. The results show that the periodicity score and the pairwise synchronization measures are able to capture different properties of spectators' synchronization, and they indicate the presence of some types of emotional and aesthetic highlights in a movie based on spectators' electro-dermal and acceleration signals.