Search

• We design, develop, deploy an Android application TouchSense, which traces different touch interactions and develop a personalized machine learning model by jointly modeling the different types of touch interaction features. • We trace different touch interactions like typing, swyping in a smartphone and perform a detailed analysis to find the role of individual and combined influence of these interactions on emotion detection. • We carry out a detail user study to find the suitability of this model. We also perform a post-study user analysis to find the usability of the proposed application. There are different modes of interaction with a software keyboard on a smartphone, such as typing and swyping. Patterns of such touch interactions on a keyboard may reflect emotions of a user. Since users may switch between different touch modalities while using a keyboard, therefore, automatic detection of emotion from touch patterns must consider both modalities in combination to detect the pattern. In this paper, we focus on identifying different features of touch interactions with a smartphone keyboard that lead to a personalized model for inferring user emotion. Since distinguishing typing and swyping activity is important to record the correct features, we designed a technique to correctly identify the modality. The ground truth labels for user emotion are collected directly from the user by periodically collecting self-reports. We jointly model typing and swyping features and correlate them with user provided self-reports to build a personalized machine learning model, which detects four emotion states (happy, sad, stressed, relaxed). We combine these design choices into an Android application TouchSense and evaluate the same in a 3-week in-the-wild study involving 22 participants. Our key evaluation results and post-study participant assessment demonstrate that it is possible to predict these emotion states with an average accuracy (AUCROC) of 73% (std dev. 6%, maximum 87%) combining these two touch interactions only. [ABSTRACT FROM AUTHOR]

The paper presents an empirical study centred on a participatory game design activity with 8–10 years old primary-school children, split in different sessions. The study assesses how children perform in game design and whether they are engaged in design tasks. To this end, the study gathers data concerning the quality of children's game design products, regarded as indicators of children's performance in game design. It collects data concerning children's emotions, which are taken as indicators of their engagement in game design. The paper statistically analyses and discusses how emotions and quality of products evolve across the game design experience, and how emotions are related to children's quality of products. Results of this work can help researchers, educators and practitioners manage a complex design experience with and for children, and identify key emotions for promoting quality of design work. [ABSTRACT FROM AUTHOR]

The ability of detecting emotions is essential in different fields such as user experience (UX), affective computing, and psychology. This paper explores the possibility of detecting emotions through user-generated bodily sensation maps (BSMs). The theoretical basis that inspires this work is the proposal by Nummenmaa et al. (2014) of BSMs for 14 emotions. To make it easy for users to create a BSM of how they feel, and convenient for researchers to acquire and classify users’ BSMs, we created a mobile app, called EmoPaint. The app includes an interface for BSM creation, and an automatic classifier that matches the created BSM with the BSMs for the 14 emotions. We conducted a user study aimed at evaluating both components of EmoPaint. First, it shows that the app is easy to use, and is able to classify BSMs consistently with the considered theoretical approach. Second, it shows that using EmoPaint increases accuracy of users’ emotion classification when compared with an adaptation of the well-known method of using the Affect Grid with the Circumplex Model, focused on the same set of 14 emotions of Nummenmaa et al. Overall, these results indicate that the novel approach of using BSMs in the context of automatic emotion detection is promising, and encourage further developments and studies of BSM-based methods. [ABSTRACT FROM AUTHOR]

Abstract: While nowadays the most usual way to show emotions in digital contexts is via virtual characters, its use may raise false expectations (the user attributes human abilities to the virtual character). This paper proposes and explores an approach to express emotions which intends to minimize the user's expectations by using a non-anthropomorphic model. Emotions are represented in terms of arousal and valence dimensions. They are visualized in a simple way through the behaviour and appearance of a series of cartoonish clouds. In particular, the arousal value is expressed through the movement of these clouds (controlled by a flocking algorithm), while the valence value is expressed through their degree of darkness. Furthermore, the paper describes a user experiment which investigated whether the arousal and valence expressed by our model are appropriately interpreted by the users or not. The results suggest that movement and darkness are interpreted as arousal and valence respectively and that they are independent of each other. [Copyright &y& Elsevier]

Although computers could offer emotional support as well as task support when aiding a user for a complex task, there is little current understanding of how they might do this. Moreover existing demonstrations of emotional support, though promising, only cover a small number of types of support and investigate a limited number of algorithms designed by hand. In this paper, we present an empirical investigation that starts from first principles, determining different categories of stressors for which emotional support might be useful, different categories of emotional support utterances and promising algorithms for deciding the content and form of textual emotional support messages according to the stressors present. At each stage, the results are validated through empirical experiments with human participants who, for instance, are required to place statements into categories, evaluate possible support messages in different imagined situations and compose their own emotional support from options offered. This development methodology allows us to avoid potentially challenging ethical issues in presenting people with stressful situations. Although our algorithms are attempting to choose emotional support based on the general, “naive” competence of human speakers, we use as a running example situations that can arise when attending a medical emergency and awaiting expert help. [ABSTRACT FROM AUTHOR]

Abstract: Emotional human–computer interactions are attracting increasing interest with the improvement in the available technology. Through presenting affective stimuli and empathic communication, computer agents are able to adjust to users'' emotional states. As a result, users may produce better task performance. Existing studies have mainly focused on the effect of only a few basic emotions, such as happiness and frustration, on human performance. Furthermore, most research explored this issue from the psychological perspective. This paper presents an emotion and performance relation model in the context of vehicle driving. This general emotion–performance model is constructed on an arousal–valence plane and is not limited to basic emotions. Fifteen paid participants took part in two driving simulation experiments that induced 115 pairs of emotion–performance sample. These samples revealed the following: (1) driving performance has a downward U-shaped relationship with both intensities of arousal and valence. It deteriorates at extreme arousal and valence. (2) Optimal driving performance, corresponding to the appropriate emotional state, matches the “sweet spot” phenomenon of the engagement psychology. (3) Arousal and valence are not perfectly independent across the entire 2-D emotion plane. Extreme valence is likely to stimulate a high level of arousal, which, in turn, deteriorates task performance. The emotion–performance relation model proposed in the paper is useful in designing emotion-aware intelligent systems to predict and prevent task performance degradation at an early stage and throughout the human–computer interactions. [Copyright &y& Elsevier]

• People behave more socially toward an intimate virtual counselor than a nonintimate one. • The perception of virtual intimacy is a strong predictor of social presence and user experience. • Perceiving virtual intimacy in human-agent interactions elicits positive emotions in users. • The social characteristics of the interaction could have a direct effect on the user's emotions. Professional embodied conversational agents (ECAs) deployed in the market are dedicated to satisfying the digital customer relationship. However, ECAs still suffer from a lack of user adoption, in particular because most of them have few social skills. Inspired by social presence theories and based on the ergonomic components of user experience (CUE) model (Mahlke and Lindgaard, 2007), the paper focuses on one social skill, i.e. , virtual intimacy, a 3-dimensional concept including honesty and genuineness, positivity and mutual comprehension. Virtual intimacy may be a way to reinforce the social dimension of human-agent interactions and to provide a better user experience. We therefore propose an interactive experiment that incorporates natural interactions between real tourists and an autonomously intimate virtual counselor who is an expert in tourism and able to express intimacy-related behaviors in verbal and nonverbal communication. The paper studies the impact of the agent's expression of intimacy-related behaviors on the perception of virtual intimacy, social presence and the user experience. The results show that users adopt a social attitude toward the intimate counselor, and although they do not significantly perceive virtual intimacy, they clearly perceive the dimension of honesty and genuineness. Moreover, the agent's expression of intimacy only enhances copresence, the first perceptive level of social presence. Except for user social status, which is enhanced, the user experience is also not significantly influenced by intimate expression. The study results further demonstrate that perceiving virtual intimacy is a good predictor of social presence and user experience, especially user emotional reactions. Perceiving virtual intimacy influences user experience independently of social presence, which thereby indicates that independent intimacy-related mechanisms, such as emotional contagion, may be involved. Mediation analyses also underline that perceiving virtual intimacy has a direct effect on user emotional reactions, but this finding is not supported by the CUE model. The findings in this study provide new evidence that perceiving virtual intimacy in human-agent interaction elicits emotions in users and enhances user experience. In line with this, we propose an outline for an ECA-adapted, user experience model based on the CUE model. [ABSTRACT FROM AUTHOR]

Humans are reflective, adaptive, and social. They recall the past, sometimes discuss these recollections with others and become nostalgic; yet, previous research has not examined nostalgia in social media. This paper investigates the expression of nostalgia within Facebook conversations. The specific themes of nostalgic longing and the expression of personal emotions are investigated in two studies. Study 1 examines 375,857 Facebook posts, and Study 2 expands on this by comparing a sample of 10,000 Facebook and 10,000 general (non-nostalgic) posts. Content analyses of these posts reveal significant evidence of nostalgic expressions in Facebook conversations. Cluster analysis reveals newer themes of nostalgic longing related to family, life stories, historical events (presidential elections, man on the moon and Gandhi), spirituality, appreciation of life, romanticism and fun. General posts tend to focus on spontaneous, spur-of-the-moment greetings, emotions, and day of the week. Nostalgic posts, in contrast, are more reflective, more emotional, and frequently include both positive and negative emotions, which is consistent with a deeper, bittersweet character to nostalgia. The concurrent utilization of past- and present-tense words in nostalgic posts suggests that, for some Facebook users, nostalgia helps interpret and navigate present circumstances. The research concludes with theoretical and managerial implications. [ABSTRACT FROM AUTHOR]

This paper describes Oudjat , a new software program which has been developed in order to conduct recognition experiments. Oudjat is dedicated to the manual annotation facial expressions of emotion (FEE). Considering the existence of other software applications in that field, Oudjat provides a compromise solution between the currently existing tools. For the investigators, it is an easy-to-configure interface to set up relevant behaviors to be annotated. For the annotators, it is an easy-to-use interface. This tool can perform complex annotations procedures utilizing multiple responses panels such as buttons, scales (e.g., Likert scales), and free labelling. Oudjat also allows to chain response panels or to conduct sequence marking annotations (i.e., two-steps temporal annotation). As it can be configured in any language, Oudjat is particularly suited for intercultural experiments. Four annotation procedures are presented to illustrate Oudjat’ s possibilities with FEE annotation. Oudjat is an open source software available to the scientific community, and can be freely be obtained on request. [ABSTRACT FROM AUTHOR]

• We designed ten virtual environments able to elicit a specific emotional reaction. • We describe the methodology we used to design the emotional virtual environments. • The methodology is based on the use of semantic elements and valence, arousal and dominance. • The results show that the designed virtual environments elicit the desired emotions and arousal-valence-dominance levels. In the era of 'metaverse', virtual environments are gaining popularity among new multimedia contents and are also recognized as a valuable means to deliver emotional content. This is favoured by cost reduction, availability and acceptance by end-users of virtual reality technology. Creating effective virtual environments can be achieved by exploiting several opportunities: creating artificial worlds able to generate different stories, mixing sensory cues, and making the whole interactive. The design space for creating emotional virtual environments is ample, and no clear idea of how to integrate the various components exists. This paper discusses how to combine multiple design elements to elicit five distinct emotions. We developed and tested two scenarios per emotion. We present the methodology, the development of the case studies, and the results of the testing. [ABSTRACT FROM AUTHOR]

In order to improve intelligent Human-Computer Interaction it is important to create a personalized adaptive emotion estimator that is able to learn over time emotional response idiosyncrasies of individual person and thus enhance estimation accuracy. This paper, with the aim of identifying preferable methods for such a concept, presents an experiment-based comparative study of seven feature reduction and seven machine learning methods commonly used for emotion estimation based on physiological signals. The analysis was performed on data obtained in an emotion elicitation experiment involving 14 participants. Specific discrete emotions were targeted with stimuli from the International Affective Picture System database. The experiment was necessary to achieve the uniformity in the various aspects of emotion elicitation, data processing, feature calculation, self-reporting procedures and estimation evaluation, in order to avoid inconsistency problems that arise when results from studies that use different emotion-related databases are mutually compared. The results of the performed experiment indicate that the combination of a multilayer perceptron (MLP) with sequential floating forward selection (SFFS) exhibited the highest accuracy in discrete emotion classification based on physiological features calculated from ECG, respiration, skin conductance and skin temperature. Using leave-one-session-out crossvalidation method, 60.3% accuracy in classification of 5 discrete emotions (sadness, disgust, fear, happiness and neutral) was obtained. In order to identify which methods may be the most suitable for real-time estimator adaptation, execution and learning times of emotion estimators were also comparatively analyzed. Based on this analysis, preferred feature reduction method for real-time estimator adaptation was minimum redundancy - maximum relevance (mRMR), which was the fastest approach in terms of combined execution and learning time, as well as the second best in accuracy, after SFFS. In combination with mRMR, highest accuracies were achieved by k-nearest neighbor (kNN) and MLP with negligible difference (50.33% versus 50.54%); however, mRMR+kNN is preferable option for real-time estimator adaptation due to considerably lower combined execution and learning time of kNN versus MLP. [ABSTRACT FROM AUTHOR]

Abstract: This paper presents preliminary efficacy data in a controlled study of the use of a virtual reality (VR) system for treating stress-related disorders (Post-Traumatic Stress Disorder, or PTSD; Pathological Grief, or PG; and Adjustment Disorders, or AD). “EMMA''s World” is a VR application in which patients can explore negative experiences to the degree required for their specific therapeutic needs. To accomplish therapeutic goals, a series of virtual elements is customized to be meaningful to the user; the elements contain the fundamental emotional components that the person must confront. Thirty-nine participants diagnosed with PTSD (N=10), PG (N=16), and AD (N=13) were randomly assigned to a standard cognitive-behavioral program (CBT) (N=20) or a CBT program driven by EMMA''s World (N=19). Participants were assessed before and after treatment. Measurements related to anxiety, depression and other emotions, maladjustment and interference were applied. Results indicate that CBT with EMMA''s World was as effective as the standard CBT program for the treatment of these disorders, and the statistically significant differences (depression, relaxation intensity and social area interference) were in favor of EMMA''s World. We expect VR to provide a positive alternative that will draw in clients who do not seek traditional forms of treatment. [Copyright &y& Elsevier]

Communication of affect across a distance is not well supported by current technology, despite its importance to interpersonal interaction in modern lifestyles. Touch is a powerful conduit for emotional connectedness, and thus mediating haptic (touch) displays have been proposed to address this deficiency; but suitable evaluative methodology has been elusive. In this paper, we offer a first, structured examination of a design space for haptic support of remote affective communication, by analyzing the space and then comparing haptic models designed to manipulate its key dimensions. In our study, dyads (intimate pairs or strangers) are asked to communicate specified emotions using a purely haptic link that consists of virtual models rendered on simple knobs. These models instantiate both interaction metaphors of varying intimacy, and representations of virtual interpersonal distance. Our integrated objective and subjective observations imply that emotion can indeed be communicated through this medium, and confirm that the factors examined influence emotion communication performance as well as preference, comfort and connectedness. The proposed design space and the study results have implications for future efforts to support affective communication using the haptic modality, and the study approach comprises a first model for systematic evaluation of haptically expressed affect. [Copyright &y& Elsevier]

This paper presents experiences on using five different self-report methods, two adopted from literature and three self-created, for collecting information about emotional responses to mobile applications. These methods were used in nine separate field experiments done in naturalistic settings. Based on our experiments, we can argue that all of these methods can be successfully used for collecting emotional responses to evaluate mobile applications in mobile settings. However, differences can be identified in the suitability of the methods for different research setups. Even though the self-report instruments provide a feasible alternative for evaluating emotions evoked by mobile applications, several challenges were identified, for example, in capturing the dynamic nature of mobile interaction usage situations and contexts. To summarise our results, we propose a framework for selecting and comparing these methods for different usage purposes. [Copyright &y& Elsevier]

The popularity of computer games has exploded in recent years, yet methods of evaluating user emotional state during play experiences lag far behind. There are few methods of assessing emotional state, and even fewer methods of quantifying emotion during play. This paper presents a novel method for continuously modeling emotion using physiological data. A fuzzy logic model transformed four physiological signals into arousal and valence. A second fuzzy logic model transformed arousal and valence into five emotional states relevant to computer game play: boredom, challenge, excitement, frustration, and fun. Modeled emotions compared favorably with a manual approach, and the means were also evaluated with subjective self-reports, exhibiting the same trends as reported emotions for fun, boredom, and excitement. This approach provides a method for quantifying emotional states continuously during a play experience. [Copyright &y& Elsevier]

This paper reports on the results of two studies that used facial electromyography (EMG) measures combined with verbal and performance measures to provide feedback in the software design process on the user's emotional state. The first study assessed 16 participant's emotional responses while they passively viewed mock ups of proposed new operating system features. The second study measured the emotional responses of 15 participants while they actively used one of two versions of a media player. This multimodal assessment method was able to provide a sensitive measure of the desirability of the proposed software features, and a measure of emotional tension and mental effort expended in the interactive tasks. [Copyright &y& Elsevier]

In this paper we describe the development and testing of a tool for self-assessment of affect while interacting with computer systems, meant to be used in many cultures. We discuss our research approach within the context of existing cultural, affective and HCI theory, and describe testing of its effectiveness in the US and Sweden. [Copyright &y& Elsevier]

Emotional aspects of homepages are becoming more important as people spend more time in cyberspace. This research aims to identify quantitative relationships between key design factors and generic dimensions of secondary emotions so that we may develop homepages which target emotions more effectively. In order to achieve this goal, we conducted three related studies. In the first study, we identified 13 generic dimensions of secondary emotions that people usually feel when viewing diverse homepages. In the second study, we identified key design factors that professional designers frequently use in their attempts to develop emotionally evocative homepages. Finally, in the third study, we identified quantitative relationships between the key design factors and the 13 emotional dimensions. This paper describes these three studies and concludes with the implications and limitations of the study results. [Copyright &y& Elsevier]

Models of affective behavior are critical for the development of training systems that are designed to exercise social interactions. Potential applications include various security-oriented operations such as police interrogation, airport security, border crossings, and military peacekeeping. Aside from speech, humans also communicate through vocalizations and inflections, as well as with body language. Such nonverbal communication can convey affect such as anger or nervousness that is important in identifying deception. In this research, a trainee is asked to perform checkpoint duty and question drivers of vehicles about their identity and reasons for entering a secured area. Most of the encounters are routine and innocuous, but occasionally a scenario unfolds that requires additional interrogation and rapid decision-making the part of the trainee. These special scenarios require the individual to draw upon his/her knowledge of social interactions in order to make the proper decisions and act appropriately. Virtual environments that address this form of training are few. Accordingly, the present paper describes an ongoing program of research designed to generate affective states for intelligent agents, create affective component behaviors to convey cues for anger, nervousness, and deception, and provide a complex interrogative training environment to exercise judgment-based decision-making. [Copyright &y& Elsevier]

This paper presents algorithms that allow a robot to express its emotions by modulating the intonation of its voice. They are very simple and efficiently provide life-like speech thanks to the use of concatenative speech synthesis. We describe a technique which allows to continuously control both the age of a synthetic voice and the quantity of emotions that are expressed. Also, we present the first large-scale data mining experiment about the automatic recognition of basic emotions in informal everyday short utterances. We focus on the speaker-dependent problem. We compare a large set of machine learning algorithms, ranging from neural networks, Support Vector Machines or decision trees, together with 200 features, using a large database of several thousands examples. We show that the difference of performance among learning schemes can be substantial, and that some features which were previously unexplored are of crucial importance. An optimal feature set is derived through the use of a genetic algorithm. Finally, we explain how this study can be applied to real world situations in which very few examples are available. Furthermore, we describe a game to play with a personal robot which facilitates teaching of examples of emotional utterances in a natural and rather unconstrained manner. [Copyright &y& Elsevier]

This paper focuses on the role of emotion and expressive behavior in regulating social interaction between humans and expressive anthropomorphic robots, either in communicative or teaching scenarios. We present the scientific basis underlying our humanoid robot''s emotion models and expressive behavior, and then show how these scientific viewpoints have been adapted to the current implementation. Our robot is also able to recognize affective intent through tone of voice, the implementation of which is inspired by the scientific findings of the developmental psycholinguistics community. We first evaluate the robot''s expressive displays in isolation. Next, we evaluate the robot''s overall emotive behavior (i.e. the coordination of the affective recognition system, the emotion and motivation systems, and the expression system) as it socially engages nave human subjects face-to-face. [Copyright &y& Elsevier]

• Pen pressure parameters serve as indicators of enjoyment and frustration. • Higher frustration leads to more pressure on a pen-based tablet. • Higher enjoyment leads to less pressure on a pen-based tablet. • The relation between pen pressure and writing performance is mediated by engagement. In this paper, two studies were conducted to understand the link between students' emotions and their engagement and performance when learning to write Japanese letters with a learning technology involving a pen-based tablet. The incorporation of this haptic sensory modality in the form of a pen tablet is seen to have much potential for continuous emotional detection. In Study 1 (136 students, 84 females, M age = 23.55, SD = 3.89), the learning effectiveness in terms of students' writing performance was assessed. We further examined the relationship between discrete learning-centered emotions such as enjoyment, frustration and boredom on the one hand, and pen pressure parameters on the other. The results demonstrated a significant gain in writing. Generalized estimating equation (GEE) models showed that minimum, maximum and average pen pressures all serve as indicators of enjoyment and frustration; on the other hand, for boredom, no significant relationships were found. Compared to enjoyment, the results demonstrate that, the higher a student's frustration when learning, the more pressure they apply to the pen when writing. In Study 2 (90 students, 60 females, M age = 22.94, SD = 3.82), the findings from Study 1 were used to understand the practical implications of the results, investigating the relationship between emotion-based pen pressure, student engagement and writing performance over the course of the learning process. As in Study 1, the results, again, showed a significant gain in learning how to write Japanese letters. The partial least squares path model (PLS-PM) analyses revealed that higher pen pressure was a negative predictor of engagement, whereas higher engagement was a positive predictor of writing performance. Furthermore, the relationship between pen pressure and performance was mediated by engagement. Overall, the findings can contribute to our understanding of the interaction between students' emotions and different learning variables in an attempt to support the development of adaptive learning technologies. [ABSTRACT FROM AUTHOR]