Search

Public speaking is a vital skill for making good impressions, effectively exchanging ideas, and influencing others. Yet, public speaking anxiety (PSA) ranks as a top social phobia. Recent advancements in wearable devices and ubiquitous virtual reality (VR) interfaces can help measure and mitigate PSA. This research quantifies PSA through bio-behavioral markers related to individuals' physiological and acoustic characteristics. The effect of virtual reality (VR) training on alleviating PSA is measured through self-reported and bio-behavioral indices. Psychological (e.g., general trait anxiety, personality) and demographic (e.g., age, gender, highest education, native language) traits are examined as moderating factors between bio-behavioral indices and PSA, as well as moderating factors for measuring the VR effectiveness in mitigating PSA. These measures are also used as clustering criteria for stratifying participants in group-based models of PSA. Results indicate the significance of such traits to modeling PSA with the proposed group-based models yielding Spearman's correlation of 0:55 ( $p \lt 0:05$ ) between the actual and predicted outcome. Results further demonstrate that systematic exposure to public speaking in VR can alleviate PSA in terms of both self-reported ( $p \lt 0:05$ ) and physiological ( $p \lt 0:05$ ) indices. Findings from this study will enable researchers to better understand antecedents and causes of PSA and lay the foundation for personalized adaptive feedback for PSA interventions.

Background: The benefits of engaging in outdoor physical activity are numerous for older adults. However, previous work on outdoor monitoring of physical activities did not sufficiently identify how older adults characterize and respond to diverse elements of urban built environments, including structural characteristics, safety attributes, and aesthetics.Objective: To synthesize emerging multidisciplinary trends on the use of connected technologies to assess environmental barriers and stressors among older adults and for persons with disability.Methods: A multidisciplinary overview and literature synthesis.Results: First, we review measurement and monitoring of outdoor physical activity in community environments and during transport using wearable sensing technologies, their contextualization and using smartphone-based applications. We describe physiological responses (e.g., gait patterns, electrodermal activity, brain activity, and heart rate), stressors and physical barriers during outdoor physical activity. Second, we review the use of visual data (e.g., Google street images, Street score) and machine learning algorithms to assess physical (e.g., walkability) and emotional stressors (e.g., stress) in community environments and their impact on human perception. Third, we synthesize the challenges and limitations of using real-time smartphone-based data on driving behavior, incompatibility with software data platforms, and the potential for such data to be confounded by environmental signals in older adults. Lastly, we summarize alternative modes of transport for older adults and for persons with disability.Conclusion: Environmental design for connected technologies, interventions to promote independence and mobility, and to reduce barriers and stressors, likely requires smart connected age and disability-friendly communities and cities.

With the increasing trend of older adults living alone, an efficient and nonintrusive way to monitor these individuals’ mental health status is required for early diagnosis of mental diseas...

As hospital workers face a growing number of patients and have to meet increasingly rigorous standards of care, their ability to successfully modulate their emotional reactions and flexibly handle stress presents a significant challenge. This paper examines a multimodal signal-driven way to quantify emotion self-regulation and stress spillover through a dynamical systems model (DSM). The proposed DSM models day-to-day changes of emotional arousal, captured through speech, physiology, and daily activity measures, and its interplay with daily stress. The parameters of the DSM quantify the degree of self-regulation and stress spillover, and are associated with work performance and cognitive ability in a multimodal dataset of 130 full-time hospital workers recorded over a 10-week period. Linear regression experiments indicate the effectiveness of the proposed features to reliably estimate individuals’ work performance and cognitive ability, providing significantly higher Pearson’s correlations compared to aggregate measures of emotional arousal. Results from this study demonstrate the importance of quantifying oscillatory behaviors from longitudinal ambulatory signals and can potentially deepen our understanding of emotion self-regulation and stress spillover using signal-driven measurements, which complement self-reports and provide estimates of the psychological constructs of interest in a fine-grained time resolution.

The present study investigated whether the presence of a romantic partner in daily life is associated with attenuated sympathetic nervous system responses. Additionally, romantic attachment style was tested as a moderator. For one day, 106 heterosexual young adult dating couples wore ambulatory sensors that monitored electrodermal activity (EDA) - an index of sympathetic arousal. Couples reported whether they were together or apart for every hour of the data collection day. Men and women exhibited lower EDA during hours in which their partner was present compared to hours in which they were absent. Additionally, romantic attachment style moderated this association; those who had low anxious attachment showed a stronger attenuating effect of partner presence compared to those with higher anxious attachment. Similarly, those who had low avoidant attachment showed heightened effects of partner presence compared to those with higher avoidant attachment. Romantic partner presence may facilitate everyday health-promoting physiological processes.

Prolonged influence of negative emotions can result in clinical depression or anxiety, and while many prescribed techniques exist, music therapy approaches, coupled with psychotherapy, have shown to help lower depressive symptoms, supplementing traditional treatment approaches. Identifying the appropriate choice of music, therefore, is of utmost importance. Selecting appropriate playlists, however, are challenged by user feedback that may inadvertently select songs that amplify the negative effects. Therefore, this work uses electroencephalogram (EEG) that automatically identifies the emotional impact of music and trains a reinforcement-learning approach to identify an adaptive personalized playlist of music to lead to improved emotional states. This work uses data from 32 users, collected in the publicly available DEAP dataset, to select songs for users that guide them towards joyful emotional states. Using a domain-specific reward-shaping function, a Q-learning agent is able to correctly guide a majority of users to the target emotional states, represented in a common emotion wheel. The average angular error of all users is 57°, with a standard deviation of 2.8 and the target emotional state is achieved.Clinical relevance- Music therapy for improving clinical depression and anxiety can be supplemented by additional emotion-guided music decisions in remote and personal settings by using automated techniques to capture emotional state and identify music that best guides users to target joyful states.

Public speaking anxiety (PSA) is among the top social phobias in the world. Quantifying PSA in a reliable and unobtrusive manner can lay the foundation toward personalized and inexpensive technology-based interventions. Existing work for quantifying PSA often relies on self-reported measures and statistical aggregates of bio-behavioral indices, such as physiology and speech. Such aggregated bio-behavioral indices are not able to capture time-based trajectories of PSA variation, that can be very useful for better understanding and reliably predicting moments of anxiety. We tackle this problem by introducing temporal parametric models to quantify bio-behavioral trajectories of PSA throughout a public speaking encounter. Using data from 55 participants in a real-life public speaking task, the parameters of the proposed models are found to be significantly correlated with individuals’ trait characteristics of general and communication-based anxiety, outperforming aggregate mean bio-behavioral measures.

Background Although past longitudinal research demonstrates that romantic partners affect one another’s health outcomes, considerably less is known about how romantic experiences “get under the skin” in everyday life. Purpose The current study investigated whether young couples’ naturally occurring feelings of closeness to and annoyance with each other during waking hours were associated with their overnight cardiovascular activity. Methods Participants were 63 heterosexual young adult dating couples (Mage = 23.07). Using ecological momentary assessments, couples reported their hourly feelings of closeness to and annoyance with their partners across 1 day; subsequent overnight heart rate was captured through wearable electrocardiogram biosensors. Actor–partner interdependence models tested whether individuals’ overnight heart rate varied as a function of (a) their own daytime feelings of closeness and annoyance (actor effects) and (b) their partner’s daytime feelings of closeness and annoyance (partner effects) while controlling for daytime heart rate. Results Although young adults’ feelings of romantic closeness and annoyance were unrelated to their own overnight heart rate (i.e., no actor effects), gender-specific partner effects emerged. Young men’s nocturnal heart rate was uniquely predicted by their female partner’s daytime relationship feelings. When women felt closer to their partners during the day, men exhibited lower overnight heart rate. When women felt more annoyed with their partners during the day, men exhibited heightened overnight heart rate. Conclusions The findings illustrate gender-specific links between couple functioning and physiological arousal in the everyday lives of young dating couples, implicating physiological sensitivity to partner experiences as one potential pathway through which relationships affect health.

The physiology of compassion, typically studied in individuals, has not yet been studied dyadically and in the context of close relationships. The current study used dating partners’ discussions of personal losses to simultaneously investigate the physiological profile of compassion in an individual sharing an emotionally salient loss, the “narrator,” and his or her partner, the “listener.” In a given moment, listeners who perceived themselves as being more compassionate had a lower heart rate. This association was stronger for listeners whose partners were less anxiously attached. Narrators and listeners who were more avoidantly attached showed stronger positive associations between their average heart rate and overall compassion. Additionally, narrators’ compassion moderated linkage between partners’ heart rates such that heart rates were more strongly linked in moments with lower levels of compassion. These results support studying compassion dyadically and illustrate the importance of attachment style in the physiology of compassion.

With the increasing use of smartphone technologies and wearable biosensors, we are currently undergoing what many have termed a “data revolution,” where intensive, multichannel data are passively collected over long time frames. Such procedures are transforming the way psychologists conceptualize research and have the potential to spur important advances in the study of close relationships. This proof-of-concept study from the Couple Mobile Sensing Project, a partnership between psychologists and engineers, combines big data and ambulatory assessment methodologies to study multimodal, microprocesses in couples’ everyday lives. These data collection procedures are designed to test how characteristics of everyday behavioral, physiological, and vocal interactions are integrated within and across individuals. We present two mini-illustrations to show how these data can be synchronized across modalities and partners and can be linked to generalized relationship dimensions. Discussion highlights the potential a...

The COVID-19 pandemic caused by the novel SARS-Coronavirus-2 (n-SARS-CoV-2) has impacted people's lives in unprecedented ways. During the time of the pandemic, social vloggers have used social media to actively share their opinions or experiences in quarantine. This paper collected videos from YouTube to track emotional responses in conversational vlogs and their potential associations with events related to the pandemic. In particular, vlogs uploaded from locations in New York City were analyzed given that this was one of the first epicenters of the pandemic in the United States. We observed some common patterns in vloggers' acoustic and linguistic features across the time span of the quarantine, which is indicative of changes in emotional reactivity. Additionally, we investigated fluctuations of acoustic and linguistic patterns in relation to COVID-19 events in the New York area (e.g. the number of daily new cases, number of deaths, and extension of stay-at-home order and state of emergency). Our results indicate that acoustic features, such as zero-crossing-rate, jitter, and shimmer, can be valuable for analyzing emotional reactivity in social media videos. Our findings further indicate that some of the peaks of the acoustic and linguistic indices align with COVID-19 events, such as the peak in the number of deaths and emergency declaration., Comment: accepted in ACM Multimedia workshop on Media Analytics for Societal Trends (MAST)

Public speaking is key to effectively exchanging ideas, persuading others, and making tangible impact. Yet, public speaking anxiety ranks as a top social phobia in the U.S., especially among first generation college students, and those who belong to underrepresented or minority groups. This paper uses physiological signals captured from wearable devices to detect self-reported state-based anxiety during a real-life public speaking encounter. Individual and contextual factors moderating the association between physiological reactivity and state-based public speaking anxiety are taken into account. Such factors include indices of trait-based anxiety, level of preparation, and gender. Results highlight the significance of including such factors for detecting public speaking anxiety and indicate that the proposed approach can yield up to 0.79 (p

Public speaking is key to effectively exchanging ideas, persuading others, and making a tangible impact. Yet, public speaking anxiety (PSA) ranks as a top social phobia among many people. This paper leverages bio-behavioural indices captured from wearable devices and virtual reality (VR) interfaces to quantify PSA. The significance of individual-specific factors, such as general trait anxiety and personality, as well as contextual factors, such as age, gender, highest education, and native language, in moderating the association between bio-behavioral indices and PSA is further examined through group-based machine learning models. Results highlight the importance of including such factors for detecting PSA with the proposed group-based PSA models yielding Spearman's correlation of 0.55(p < 0.05) between the actual and predicted state-based anxiety scores. This work further analyzes whether systematic exposure to public speaking tasks in the VR environment can help alleviate PSA. Results indicate that systematic exposure to public speaking in VR can alleviate PSA in terms of both self-reported (p < 0.05) and physiological (p < 0.05) indices. Findings of this study will enable researchers to better understand antedecedents and causes of PSA contributing to behavioral interventions using VR.

Individuals exposed to aggression and who perpetrate aggression against others show differences in their physiological activation during stress; the goal of the present study is to investigate physiological stress reactivity as a factor contributing to the intergenerational transmission of aggression. To test associations between family-of-origin aggression (FOA), physiological reactivity in daily life, and dating aggression perpetration, we used ecological momentary assessment to monitor fluctuations in young adult (M(age) = 23.1 years) dating couples’ electrodermal activity (EDA) over an entire day and examined how naturally-occurring bouts of annoyance between partners relate to EDA, FOA, and dating aggression perpetration. Dating perpetration was linked to lower general levels of EDA in both men and women, while FOA was linked to lower general levels of EDA in men only. For women, multi-group, multilevel models showed that FOA and dating aggression perpetration moderated the association between feeling annoyed and EDA, such that those with greater FOA and dating aggression perpetration showed greater EDA reactivity during naturally-occurring relationship stress. Furthermore, this pattern of EDA reactivity mediated the link between FOA and dating aggression perpetration in women. These results provide evidence that FOA and dating aggression perpetration are linked to patterns of physiological responsivity in everyday life and suggest that these patterns could be important factors contributing to the intergenerational transmission of aggression.

This paper presents an introduction to the "Human-Habitat for Health (H3): Human-habitat multimodal interaction for promoting health and well-being in the Internet of Things era" workshop, which was held at the 20th ACM International Conference on Multimodal Interaction on October 16th, 2018, in Boulder, CO, USA. The main theme of the workshop focused on the effect of the physical or virtual environment on individual's behavior, well-being, and health. The H3 workshop included keynote speeches that provided an overview and future directions of the field, as well as presentations including position papers and research contributions. The workshop brought together experts from academia and industry spanning a set of multi-disciplinary fields, including computer science, speech and spoken language understanding, construction science, life-sciences, health sciences, and psychology, to discuss their respective views and identify synergistic and converging research directions and solutions.

Negative emotional arousal during conflict has been related to negative outcomes in romantic relationships and degraded quality of family life. Despite its extensive study in psychology, it is still challenging to quantify emotional arousal in a meaningful way with objective indices beyond traditionally-used self-reported scores. We examine the association of acoustic and physiological arousal between dating couples through speech prosodic patterns and Electrodermal Activity (EDA) features. We use a dynamical systems model (DSM) approach to capture the interplay of arousal indices within and between people. The DSM parameters reflect the amount of self-regulation with respect to the acoustic and physiological cues within a person, the degree of cross-regulation between the two modalities, as well as the within-couple co-regulation. Our results through statistical analysis and classification experiments indicate a significant association between the estimated system parameters and the participants' self-reported relationship satisfaction measures. This is consistent with previous findings and can help towards better understanding regulation mechanisms and escalation effects of emotional arousal during couples' discussions.

Speech signal processing is being increasingly explored in health domains given both its centrality as a behavioral cue and the promise of robust, automated analysis of data at scale. We discuss general issues in health-related speech research. We further highlight two health applications we have undertaken: addiction counseling and autism spectrum disorder. Methods range from deep supervised learning to knowledge-based signal processing of highly subjective constructs of psychological states and traits. A unique aspect of the research has been to model both health care provider and patient behaviors jointly in clinical encounters, wherein any individual behavior cannot be considered in isolation given the inherent mutual influence.

Children's interpersonal synchrony has been related to various benefits in social, mental and emotional development. We explore verbal and acoustic synchrony patterns between pairs of children playing a speech-controlled video game. Verbal features include word timing and duration patterns, while acoustic cues contain prosodic information. Synchrony is captured through a random-effects model taking into account multiple sources of variation and repeated measurements for each pair of children. Our findings indicate the presence of synchrony between participants during game play, which increases as they become more engaged in the game. These results are discussed in relation to personalized human-computer interaction and adaptive game environments.

The development of ecologically valid procedures for collecting reliable and unbiased emotional data towards computer interfaces with social and affective intelligence targeting patients with mental disorders.Following its development, presented with, the Athens Emotional States Inventory (AESI) proposes the design, recording and validation of an audiovisual database for five emotional states: anger, fear, joy, sadness and neutral. The items of the AESI consist of sentences each having content indicative of the corresponding emotion. Emotional content was assessed through a survey of 40 young participants with a questionnaire following the Latin square design. The emotional sentences that were correctly identified by 85% of the participants were recorded in a soundproof room with microphones and cameras. A preliminary validation of AESI is performed through automatic emotion recognition experiments from speech.The resulting database contains 696 recorded utterances in Greek language by 20 native speakers and has a total duration of approximately 28 min. Speech classification results yield accuracy up to 75.15% for automatically recognizing the emotions in AESI.These results indicate the usefulness of our approach for collecting emotional data with reliable content, balanced across classes and with reduced environmental variability.

Early intervention in individuals with Autism Spectrum Disorder (ASD) can improve core and associated symptoms and facilitate skills that increase social opportunities. However, determining effective intervention success in this population, and the mechanisms that produce it, is currently restricted to observable behavior. The need of therapy assessment metrics beyond traditional behavioral criteria, led to the use of physiological signals for capturing child-therapist internal dynamics during an intervention session. Internal physiological states were measured through Electrodermal Activity (EDA) and modeled in relation to observed self- and co-regulatory behaviors. A common measure of EDA, Skin Conductance Response (SCR), was the primary signal of interest and assumed to form a non-homogeneous Poisson Process whose rate function is determined by observed regulatory behaviors. Through likelihood and residual goodness of fit analysis, statistical tests and classification tasks, our results indicate that SCR changes and observable behavior in child-therapist dyads are temporally associated and the estimated model parameters can be linked to the types of regulation stimuli.

Signal-derived measures can provide effective ways towards quantifying human behavior. Verbal Response Latencies (VRLs) of children with Autism Spectrum Disorders (ASD) during conversational interactions are able to convey valuable information about their cognitive and social skills. Motivated by the inherent gap between the external behavior and inner affective state of children with ASD, we study their VRLs in relation to their explicit but also implicit behavioral cues. Explicit cues include the children's language use, while implicit cues are based on physiological signals. Using these cues, we perform classification and regression tasks to predict the duration type (short/long) and value of VRLs of children with ASD while they interacted with an Embodied Conversational Agent (ECA) and their parents. Since parents are active participants in these triadic interactions, we also take into account their linguistic and physiological behaviors. Our results suggest an association between VRLs and these externalized and internalized signal information streams, providing complementary views of the same problem.

The quality of shared enjoyment in interactions is a key aspect related to Autism Spectrum Disorders (ASD). This paper discusses two types of enjoyment: the first refers to humorous events and is associated with one's positive affective state and the second is used to facilitate social interactions between people. These types of shared enjoyment are objectively specified by their proximity to a voiced and unvoiced laughter instance, respectively. The goal of this work is to study the acoustic differences of areas surrounding the two kinds of shared enjoyment instances, called “social zones”, using data collected from children with autism, and their parents, interacting with an Embodied Conversational Agent (ECA). A classification task was performed to predict whether a “social zone” surrounds a voiced or an unvoiced laughter instance. Our results indicate that humorous events are more easily recognized than events acting as social facilitators and that related speech patterns vary more across children compared to other interlocutors.

Storytelling is a commonly used technique for rating linguistic and communicative abilities of children with Autism Spectrum Disorders (ASD). It highlights their language use beyond sentence-level production, and their ability to cohesively link events into a plot, including incorporating social context. A key scenario of interest we consider is spoken narrative creation in interactive settings, where confederates such as parents can offer scaffolding to their children’s narratives by eliciting answers with appropriate questions, shaping the structure of the resulting narrative. We analyze the structure of children’s stories narrated with the help of their parents using entity-based feature-level patterns in order to see how there are influenced by the parents’ narrative elicitation techniques. The frequency distribution and evolution of entities -meaning the co-referent people, objects and ideas- can capture the main axis of the story plot. Our results indicate that the type of questions the parents ask can be reflected in the entity-based features of a narrative, affecting its underlying structure and coherence. Index Terms: Narrative Structure, Coherence, Text Entities, Autism Spectrum Disorders