Search

Eyesweb patches for extraction of motion information from video, used in preparation of IEMP Collection.

This project collects together pieces of code used in preparation of IEMP Collection.

Interpersonal musical entrainment—temporal synchronization and coordination between individuals in musical contexts—is a ubiquitous phenomenon related to music’s social functions of promoting group bonding and cohesion. Mechanisms other than sensorimotor synchronization are rarely discussed, while little is known about cultural variability or about how and why entrainment has social effects. In order to close these gaps, we propose a new model that distinguishes between different components of interpersonal entrainment: sensorimotor synchronization—a largely automatic process manifested especially with rhythms based on periodicities in the 100–2000 ms timescale—and coordination, extending over longer timescales and more accessible to conscious control. We review the state of the art in measuring these processes, mostly from the perspective of action production, and in so doing present the first cross-cultural comparisons between interpersonal entrainment in natural musical performances, with an exploratory analysis that identifies factors that may influence interpersonal synchronization in music. Building on this analysis we advance hypotheses regarding the relationship of these features to neurophysiological, social, and cultural processes. We propose a model encompassing both synchronization and coordination processes and the relationship between them, the role of culturally shared knowledge, and of connections between entrainment and social processes.

Full-body human movement is characterized by fine-grain expressive qualities that humans are easily capableof exhibiting and recognizing in others’ movement. In sports (e.g., martial arts) and performing arts(e.g., dance), the same sequence of movements can be performed in a wide range of ways characterized bydifferent qualities, often in terms of subtle (spatial and temporal) perturbations of the movement. Even anon-expert observer can distinguish between a top-level and average performance by a dancer or martialartist. The difference is not in the performed movements–the same in both cases–but in the “quality” of theirperformance.In this article, we present a computational framework aimed at an automated approximate measure ofmovement quality in full-body physical activities. Starting from motion capture data, the framework computeslow-level (e.g., a limb velocity) and high-level (e.g., synchronization between different limbs) movementfeatures. Then, this vector of features is integrated to compute a value aimed at providing a quantitative assessmentof movement quality approximating the evaluation that an external expert observer would give ofthe same sequence of movements.Next, a system representing a concrete implementation of the framework is proposed. Karate is adoptedas a testbed.We selected two different katas (i.e., detailed choreographies of movements in karate) characterizedby different overall attitudes and expressions (aggressiveness, meditation), and we asked seven athletes,having various levels of experience and age, to perform them. Motion capture data were collected from theperformances and were analyzed with the system. The results of the automated analysis were compared withthe scores given by 14 karate experts who rated the same performances. Results show that the movementquality scores computed by the system and the ratings given by the human observers are highly correlated (Pearson’s correlations r = 0.84, p = 0.001 and r = 0.75, p = 0.005).

‘Serious games’ are becoming extremely relevant to individuals who have specific needs, such as children with an Autism Spectrum Condition (ASC). Often, individuals with an ASC have difficulties in interpreting verbal and non-verbal communication cues during social interactions. The ASC-Inclusion EU-FP7 funded project aims to provide children who have an ASC with a platform to learn emotion expression and recognition, through play in the virtual world. In particular, the ASC-Inclusion platform focuses on the expression of emotion via facial, vocal, and bodily gestures. The platform combines multiple analysis tools, using on-board microphone and web-cam capabilities. The platform utilises these capabilities via training games, text-based communication, animations, video and audio clips. This paper introduces current findings and evaluations of the ASC-Inclusion platform and provides detailed description for the different modalities.

As art influences science and technology, science and technology can in turn inspire art. Recognizing this mutually beneficial relationship, researchers at the Casa Paganini-InfoMus Research Centre work to combine scientific research in information and communications technology (ICT) with artistic and humanistic research. Here, the authors discuss some of their work, showing how their collaboration with artists informed work on analyzing nonverbal expressive and social behavior and contributed to tools, such as the EyesWeb XMI hardware and software platform, that support both artistic and scientific developments. They also sketch out how art-informed multimedia and multimodal technologies find application beyond the arts, in areas including education, cultural heritage, social inclusion, therapy, rehabilitation, and wellness.

In this paper we identify developments that have led to the current interest from computer scientists in Brain-Computer Interfacing (BCI). Nondisabled users have become a target group for BCI applications. Non-disabled users can not be treated as patients. They are free to move and use their hands during the interaction with an application. Therefore BCI should be integrated in a multimodal approach. Games are an important research area since shortcomings of BCI can be translated into challenges in multimodal cooperative, competitive, social and casual games.

Does using a brain-computer interface (BCI) influence the social interaction between people when playing a cooperative game? By measuring the amount of speech, utterances, instrumental gestures and empathic gestures during a cooperative game where two participants had to reach a certain goal, and questioning participants about their own experience afterwards this study attempts to provide answers to this question. The results showed that social interaction changed when using a BCI compared to using a mouse. There was a higher amount of utterances and empathic gestures. This indicates that the participants reacted more to the higher difficulty of the BCI selection method. Participants also reported that they felt they cooperated better during the use of the mouse.

Although psychological research indicates that bodily expressions convey important affective information, to date research in emotion recognition focused mainly on facial expression or voice analysis. In this paper we propose an approach to realtime automatic emotion recognition from body movements. A set of postural, kinematic, and geometrical features are extracted from sequences 3D skeletons and fed to a multi-class SVM classifier. The proposed method has been assessed on data acquired through two different systems: a professionalgrade optical motion capture system, and Microsoft Kinect. The system has been assessed on a "six emotions" recognition problem, and using a leave-one-subject-out cross validation strategy, reached an overall recognition rate of 61.3% which is very close to the recognition rate of 61.9% obtained by human observers. To provide further testing of the system, two games were developed, where one or two users have to interact to understand and express emotions with their body.

Building a complete virtual human application from scratch is a daunting task, and it makes sense to rely on existing platforms for behavior generation. When building such an interactive application, one needs to be able to adapt and extend the capabilities of the virtual human offered by the platform, without having to make invasive modications to the platform itself. This paper describes how Elckerlyc, a novel platform for controlling a virtual human, offers these possibilities.

We live in a world of continuous information overflow, but the quality of information and communication is suffering. Single value devices contribute to information and communication quality by focussing on one explicit, relevant piece of information. The information is decoupled from a computer and represented in an object, integrated into daily life. The contribution of this paper is on different levels: Firstly, we identify single value devices as a class, and, secondly, illustrate it through examples in a survey. Thirdly, we collect characterizations of single value devices into a taxonomy. The taxonomy also provides a collection of design choices that allow one to more easily find new combinations or alternatives, and that facilitate the design of new, meaningful, effective and working objects. Finally, when we want to step from experimental examples to commercializable products, a number of issues become relevant that are identified and discussed in the remainder of this paper.

This paper presents a novel method for interactive playground design, based on traditional children's play. This method combines the rich interaction possibilities of computer games with the physical and open-ended aspects of traditional children's games. The method is explored by the development of a prototype interactive playground, which has been implemented and evaluated over two iterations.

During face-to-face interpersonal interaction people have a tendency to mimic each other, that is, they change their own behaviors to adjust to the behavior expressed by a partner. In this paper we describe how behavioral information expressed between two interlocutors can be used to detect and identify mimicry and improve recognition of interrelationship and affect between them in a conversation. To automatically analyze how to extract and integrate this behavioral information into a mimicry detection framework for improving affective computing, this paper addresses the main challenge: mimicry representation in terms of optimal behavioral feature extraction and automatic integration.

Building a complete virtual human application from scratch is a daunting task, and it makes sense to rely on existing platforms for behavior generation. When building such an interactive application, one needs to be able to adapt and extend the capabilities of the virtual human offered by the platform, without having to make invasive modifications to the platform itself. This paper describes how Elckerlyc, a novel platform for controlling a virtual human, offers these possibilities.

In this paper, we introduce a vision called Smart Material Interfaces (SMIs), which takes advantage of the latest generation of engineered materials that has a special property defined “smart‿. They are capable of changing their physical properties, such as shape, size and color, and can be controlled by using certain stimuli (light, potential difference, temperature and so on). We describe SMIs in relation to Tangible User Interfaces (TUIs) to convey the usefulness and a better understanding of SMIs.

Description

International audience; The last decade has seen the development of standards for music notation (MusicXML), audio analysis (SDIF), and sound control (OSC), but there are no widespread standards, nor structured approaches, for handling music-related movement, action and gesture data. This panel will address the needs for such formats and standards in the computer music community, and discuss possible directions for future development.

International audience; This paper summarises a panel discussion at the 2007 International Computer Music Conference on movement and gesture data formats, presents some of the formats currently in development in the computer music community, and outlines some of the challenges involved in future development.

This paper presents the gesture desk, a new platform for a human-computer interface at a regular computer workplace. It extends classical input devices like keyboard and mouse by arm and hand gestures, without the need to use any inconvenient accessories like data gloves or markers. A central element is a "gesture box" containing two infrared cameras and a color camera which is positioned under a glass desk. Arm and hand motions are tracked in three dimensions. A synchronizer board has been developed to provide an active glare-free IR-illumination for robust body and hand tracking. As a first application, we demonstrate interactive real-time browsing and querying of auditory self-organizing maps (AuSOMs). An AuSOM is a combined visual and auditory presentation of high-dimensional data sets. Moving the hand above the desk surface allows to select neurons on the map and to manipulate how they contribute to data sonification. Each neuron is associated with a prototype vector in high-dimensional space, so that a set of 2D-topologically ordered feature maps is queried simultaneously. The level of detail is selected by hand altitude over the table surface, allowing to emphasize or deemphasize neurons on the map.