Search

International audience; Advancing knowledge and understanding about performed music is hampered by a lack of annotation data for music expressivity. To enable large-scale collection of annotations and explorations of performed music, the authors have created a workflow that is enabled by CosmoNote, a Web-based citizen science tool for annotating musical structures created by the performer and experienced by the listener during expressive piano performances. To enable annotation tasks with CosmoNote, annotators can listen to the recorded performances and view synchronized music visualization layers including the audio waveform, recorded notes, extracted audio features such as loudness and tempo, and score features such as harmonic tension. Annotators have the ability to zoom into specific parts of a performance and see visuals and listen to the audio from just that part. The annotation of performed musical structures is done by using boundaries of varying strengths, regions, comments, and note groups. By analyzing the annotations collected with CosmoNote, performance decisions will be able to be modeled and analyzed in order to aid in the understanding of expressive choices in musical performances and discover the vocabulary of performed musical structures.

International audience

International audience; Music improvisation is the ability of musical generative systems to interact with either another music agent or a human improviser. This is a challenging task, as it is not trivial to define a quantitative measure that evaluates the creativity of the musical agent. It is also not feasible to create huge paired corpora of agents interacting with each other to train a critic system. In this paper we consider the problem of controlling machine improvisation by switching between several pre-trained models by finding the best match to an external control signal. We introduce a measure SymTE that searches for the best transfer entropy between representations of the generated and control signals over multiple generative models.

International audience; Instrument sound synthesis using deep neural networks has received numerous improvements over the last couple of years. Among them, the Differentiable Digital Signal Processing (DDSP) framework has modernized the spectral modeling paradigm by including signal-based synthesizers and effects into fully differentiable architectures. The present work extends the applications of DDSP to the task of polyphonic sound synthesis, with the proposal of a differentiable piano synthesizer conditioned on MIDI inputs. The model architecture is motivated by high-level acoustic modeling knowledge of the instrument which, along with the sound structure priors inherent to the DDSP components, makes for a lightweight, interpretable and realistic sounding piano model. A subjective listening test has revealed that the proposed approach achieves better sound quality than a state-of-the-art neural-based piano synthesizer, but physical-modeling-based models still hold the best quality. Leveraging its interpretability and modularity, a qualitative analysis of the model behavior was also conducted: it highlights where additional modeling knowledge and optimization procedures could be inserted in order to improve the synthesis quality and the manipulation of sound properties. Eventually, the proposed differentiable synthesizer can be further used with other deep learning models for alternative musical tasks handling polyphonic audio and symbolic data

International audience; Flamenco is a traditional Spanish music from Andalusia. Based on oral tradition and transmission, traditional Flamenco, has implicit rules that allow musicians and dancers to interpret complex patterns without previous rehearsal. The experiments described here attempt to discover these rules by setting a musical improvisation program to play in the flamenco style and by analyzing the reactions of expert musicians listening to it. There are numerous styles or “palos” in Flamenco, however some general guidelines can be extracted. Here we focus on two general rules: i) During improvised musical developments where one partner takes the lead, musicians “resolve” together through a “remate” ii) The underlying common pattern is rhythmWe tried to determine the minimal conditions to provide “acceptable Flamenco” through the Djazz environment developed by IRCAM and EHESS. We examined typical “palos”: Tangos (binary), Fandangos (ternary) and Bulerias (mixed). The acceptability of the Djazz productions were tested by i) submitting the improvisations to musicians and listen to their advice, ii) co-improvising with musicians. In a first attempt, a “letra” (song) of Fandango was investigated. No grid was imposed. The results were unacceptable, as neither the harmony nor the rhythm and the remate were distinguishable. Then, a harmony pattern (chords) was applied. It was again not satisfactory, as the same chord in different places has not the same role. Thus, harmony and rhythm must be considered. We then focused on the rhythm aspects of Bulerias, using “palmas” (hands clapping). The acceptable grid was the one, where the “remate” is clearly identified. Additionally, we could define the statistical fingerprint of the Bulerias palo and compared the Djazz production to that of musicians. The conclusion of the study on Bulerias was directly applied to a live improvisation with musicians on Tangos, without preliminary tests, but we directly imposed a grid where the remate is clearly identified. Djazz production was satisfying for two criteria: the palo was respected and the musicians (guitar, dance) could co-improvise upon the Djazz Proposals.

Electric vehicles (EVs) are very quiet at low speed, which can be hazardous for pedestrians, especially visually impaired people. It is now mandatory (since mid-2019 in Europe) to add external warning sounds, but poor sound design can lead to noise pollution, and consequently annoyance. Moreover, it is possible that EVs are not sufficiently detectable in urban areas because of the masking effect from the background noise. In this paper, we propose a method for the design of warning sounds that takes into account both detectability and unpleasantness. The method implements a multiobjective interactive genetic algorithm (IGA) for the optimisation of the characteristics of synthesised sounds. An experiment is proposed to a first panel of participants in order to define a set of Pareto efficient sounds. At the individual level, sounds obtained with the IGA are compared to different sound design proposals. Results show that the quality of the sounds designed by the IGA method is comparable to those provided by a sound designer. From the sounds of the Pareto set, a design recommendation method based on the probability distributions of the sounds’ characteristics is proposed. An external validation with a second panel of participants shows that these recommended sounds constitute relevant trade-offs when compared to other design proposals.

International audience; Recent neural-based synthesis models have achieved impressive results for musical instrument sound generation. In particular, the Differentiable Digital Signal Processing (DDSP) framework enables the usage of spectral modeling analysis and synthesis techniques in fully differentiable architectures. Yet currently, it has only been used for modeling monophonic instruments. Leveraging the interpretability and modularity of this framework, the present work introduces a polyphonic differentiable model for piano sound synthesis, conditioned on Musical Instrument Digital Interface (MIDI) inputs. The model architecture is motivated by high-level acoustic modeling knowledge of the instrument which, in tandem with the sound structure priors inherent to the DDSP components, makes for a lightweight, interpretable and realistic sounding piano model. The proposed model has been evaluated in a listening test, demonstrating improved sound quality compared to a benchmark neural-based piano model, with significantly less parameters and even with reduced training data. The same listening test indicates that physical-modeling-based models still achieve better quality, but the differentiability of our lightened approach encourages its usage in other musical tasks dealing with polyphonic audio and symbolic data.

International audience; In this paper we survey the methods for control and cre- ative interaction with pre-trained generative models for au- dio and music. By using reduced (lossy) encoding and sym- bolization steps we are able to examine the level of informa- tion that is passing between the environment (the musician) and the agent (machine improvisation). We further use the concept of music information dynamics to find an optimal symbolization in terms of predictive information measure. Methods and strategies for generative models are surveyed in this paper and their implications for creative interaction with the machine are discussed in the musical improvisation framework.

This paper presents two experiments focusing on perception of mechanical sounds produced by expressive robot movement and blended sonifications thereof. In the first experiment, 31 participants evaluated emotions conveyed by robot sounds through free-form text descriptions. The sounds were inherently produced by the movements of a NAO robot and were not specifically designed for communicative purposes. Results suggested no strong coupling between the emotional expression of gestures and how sounds inherent to these movements were perceived by listeners; joyful gestures did not necessarily result in joyful sounds. A word that reoccurred in text descriptions of all sounds, regardless of the nature of the expressive gesture, was “stress”. In the second experiment, blended sonification was used to enhance and further clarify the emotional expression of the robot sounds evaluated in the first experiment. Analysis of quantitative ratings of 30 participants revealed that the blended sonification successfully contributed to enhancement of the emotional message for sound models designed to convey frustration and joy. Our findings suggest that blended sonification guided by perceptual research on emotion in speech and music can successfully improve communication of emotions through robot sounds in auditory-only conditions.

International audience; Somax 2 is a multi-agent interactive system performing live machine co-improvisation with musicians, based on machine-listening, machine-learning, and generative units. The actual version ([Borg 2020], [Borg 2021a], [Borg 2021b]) is a recent development and algorithms improvement from the former Somax version and previous work in RepMus team ([Bonasse-Gahot 2012], [Bonasse-Gahot 2014], [Carsault 2017], [Carsault 2019], [Carsault 2020], [Carsault 2021]).Agents provide stylistically coherent improvisations based on learned musical knowledge while continuously listening to and adapting to input from musicians or other agents in real time. The system is trained on any musical materials chosen by the user, effectively constructing a generative model (called a corpus), from which it draws its musical knowledge and improvisation skills. Corpora, inputs and outputs can be MIDI as well as audio, and inputs can be live or streamed from Midi or audio files. Somax 2 is one of the improvisation systems descending from the well-known Omax software, presented here in a totally new implementation. As such it shares with its siblings, the general loop [listen/learn/model/generate], using some form of statistical modeling that ends up in creating a highly organized memory structure from which it can navigate into new musical organizations, while keeping style coherence, rather than generating unheard sounds as other ML systems do. However Somax 2 adds a totally new versatility by being incredibly reactive to the musician decisions, and by putting its creative agents to communicate and work together in the same way, thanks to cognitively inspired interaction strategies and finely optimized concurrent architecture that make all its units smoothly cooperate together.

The way the visual system processes different scales of spatial information has been widely studied, highlighting the dominant role of global over local processing. Recent studies addressing how the auditory system deals with local–global temporal information suggest a comparable processing scheme, but little is known about how this organization is modulated by long-term musical training, in particular regarding musical sequences. Here, we investigate how non-musicians and expert musicians detect local and global pitch changes in short hierarchical tone sequences structured across temporally-segregated triplets made of musical intervals (local scale) forming a melodic contour (global scale) varying either in one direction (monotonic) or both (non-monotonic). Our data reveal a clearly distinct organization between both groups. Non-musicians show global advantage (enhanced performance to detect global over local modifications) and global-to-local interference effects (interference of global over local processing) only for monotonic sequences, while musicians exhibit the reversed pattern for non-monotonic sequences. These results suggest that the local–global processing scheme depends on the complexity of the melodic contour, and that long-term musical training induces a prominent perceptual reorganization that reshapes its initial global dominance to favour local information processing. This latter result supports the theory of “analytic” processing acquisition in musicians.

Movement-sound interactive systems are at the interface of different artistic and educational practices. Within this multiplicity of uses, we examine common denominators in terms of learning, appropriation and relationship to technological systems. While these topics have been previously reported at NIME, we wanted to investigate how practitioners, coming from different perspectives, relate to these questions. We conducted interviews with 6 artists who are engaged in movementsound interactions: 1 performer, 1 performer/composer, 1 composer, 1 teacher/composer, 1 dancer/teacher, 1 dancer. Through a thematic analysis of the transcripts we identified three main themes related to (1) the mediating role of technological tools (2) usability and normativity, and (3) learning and practice. These results provide ground for discussion about the design and study of movement-sound interactive systems., Comment: New Interfaces for Musical Expression (NIME), Jun 2022, Auckland, New Zealand

International audience

International audience

In this article, we discuss some of our research with Local Area Networks (LAN) in the context of sound installations or musical performances. Our systems, built on top of Web technologies, enable novel possibilities of collective and collaborative interaction, in particular by simplifying public access to the artwork by presenting the work through the web browser of their smartphone/tablet. Additionally, such a technical framework can be extended with so-called nano-computers, microprocessors and sensors. The infrastructure is completely agnostic as to how many clients are attached, or how they connect, which means that if the work is available in a public space, groups of friends, or even informally organised flash mobs, may engage with the work and perform the contents of the work at any time, and if available over the Internet, at any place. More than the technical details, the specific artistic directions or the supposed autonomy of the agents of our systems, this article focuses on how such ‘networks of devices’ interleave with the ‘network of humans’ composed of the people visiting the installation or participating in the concert. Indeed, we postulate that an important point in understanding and describing such proposals is to consider the relation between these two networks, the way they co-exist and entangle themselves through perception and action. To exemplify these ideas, we present a number of case studies, sound installations and concert works, very different in scope and artistic goal, and examine how this interaction is materialised from several standpoints.

International audience

International audience; This paper describes the temporal model of a scheduler geared towards show control and live music applications. This model relies on multiple interrelated temporal axes, called timescales. Timescales allow scheduling computations using abstract dates and delays, much like a score uses symbolic positions and durations (e.g. bars, beats, and note values) to describe musical time. Abstract time is ultimately mapped onto wall-clock time through the use of time transformations, specified as tempo curves, for which we provide a formalism in terms of differential equations on symbolic position. In particular, our model allows specifying tempo both as a function of time or as a function of symbolic position, and allows piecewise tempo curves to be built from parametric curves.

International audience

International audience; Does being part of an orchestra from an early age have an impact on cognitive and emotional capacities? Researchers from the University of Geneva (Unige), Switzerland, the University of Genoa (Unige-IT), and the Institute of Research and Coordination Acoustic/Music (IRCAM) in Paris investigated this question in the context of the Demos orchestras organized by the Philharmonie de Paris-Cité. This two-year study evaluated the development and evolution of the cognitive, emotional, and motor skills of 255 children engaged in two Demos orchestras, aged 7 to 13 years living in education priority zones (ZEPs) in France. The results showed that after two years of playing music as part of a group, the working memory, cognitive flexibility, and emotion understanding skills of the children improved at a higher rate than the published measurements on standard tests. Important improvements were also shown in movement synchronization.

International audience

International audience

The integration of users' perception in the design process is and important challenge for the optimization of products. This study describes how design recommendations can be drawn, from a perceptual experiment with a panel of subjects using a multi-objective interactive genetic algorithm (IGA). The application concerns the bi-objective optimization of the unpleasantness and the detectability of sounds for electric vehicles (EV). After a description of the experimental protocol for the assessment of the detectability and the unpleasantness of EV sounds (listening test), a set of optimal sounds (Pareto efficient) is defined with an IGA experiment. The analysis of these sounds, based on a probabilistic analysis of the selection process, leads to the definition of design recommendations. A second listening test, involving recommended sounds but also other design proposals, allows an evaluation of the validity of the approach. Results show that the sounds recommended obtain interesting performance, in particular to improve the detectability of EV sounds.

International audience

International audience; CosmoNote is a web-based citizen science tool for annotating musical structures, with a focus on structures created by the performer during expressive musical performance. The software interface enables the superimposition of synchronized discrete and continuous information layers which include note data representations, audio features such as loudness and tempo, and score features such as harmonic tension in a visual and audio environment. The tools provide the means for users to signal performance decisions such as segmentation and prominence using boundaries of varying strengths, regions, comments, and note groupings. User-friendly interaction features have been built in to facilitate ease of annotation; these include the ability to zoom in, listen to, and mark up specific segments of music. The data collected will be used to discover the vocabulary of performed music structures and to aid in the understanding of expressive choices and nuances.

Texte écrit par Patrick Susini (STMS Ircam-CNRS-SU). Ce texte dans sa forme intégrale devait figurer initialement dans un ouvrage collectif sur le design sonore publié chez Dunod.

Computer assisted music extensively relies on audio sample libraries and virtual instruments which provide users an ever increasing amount of contents to produce music with. However, principled methods for large-scale interactions are lacking so that browsing samples and presets with respect to a target sound idea is a tedious and arbitrary process. Indeed, library metadata can only describe coarse categories of sounds but do not meaningfully traduce the underlying acoustic contents and continuous variations in timbre which are key elements of music production and creativity. Timbre perception has been studied by carrying listening tests and organising human ratings into low dimensional spaces which reflect the perceived similarity of sounds, however these analysis spaces do not generalise to new and unrated examples, nor they allow to synthesise audio.Digital signal processing models have been applied to analysis and synthesis, so that the extracted parameters can be manipulated and inverted back to audio. However, we observe that these methods require a high number of parameters and representation dimensions to allow accurate reconstructions. Visualisation and control are thus little intuitive, moreover these invertible dimensions mainly correspond to low-level signal properties and do not represent much semantic information. The recent advances in deep generative modelling show unprecedented successes at learning large-scale unsupervised representations which invert to data as diverse as images, texts and audio. These probabilistic models could be refined to specific generative tasks such as unpaired image translation and semantic manipulations of visual features, demonstrating the ability of learning transformations and representations that are perceptually meaningful.The application of deep generative models to musical audio is at early stages and requires adapted model architectures and interactions. High quality auto-regressive waveform synthesis has been achieved for both speech and musical audio, however these models are computationally intensive, unsuited to moderate dataset sizes and offer little control over the generation for creative purposes. In this thesis, we target efficient analysis and synthesis with auto-encoders to learn low dimensional acoustic representations for timbre manipulations and intuitive interactions for music production. We adapt domain translation techniques to timbre transfer and propose alternatives to adversarial learning for many-to-many transfers. In this process, timbre is implicitly modelled by disentangling the representations of domain specific and domain invariant features. Then we develop models for explicit modelling of timbre variations and controllable audio sampling using conditioning for semantic attribute manipulations and hierarchical learning to represent both acoustic and temporal variations. We also apply discrete representation learning to decompose a target timbre into short-term acoustic features that are applied to audio conversions such as timbre transfer and voice-driven synthesis. By analysing and mapping this discrete latent representation, we show that we can directly control synthesis by acoustic descriptors. Finally, we investigate the possibility of further reducing the complexity of trained models by weight trimming for real-time inference with constrained computational resources. Because the objectives used for training the models are often disjoint from the ultimate generative application, our discussion and evaluation emphasise both aspects of learning performance and usability as a creative tool for music production.The organisation of this thesis is as follows. The first section introduces computational music processing, its different levels of expression and sets the problem domain of meaningful audio synthesis for music and with machine learning tools. The second section details the representation properties of audio and music information, it reviews classical models for audio synthesis and introduces the data-driven approach to music processing with machine learning. The third section introduces unsupervised learning and reviews the fundamentals of generative modelling with deep learning. The fourth section discusses related works in the field of neural audio synthesis with an emphasis in music applications. The fifth section presents the experiments carried during this thesis and discusses the results and evaluations, in the format of a "thesis by publications". The sixth section summarizes the results and publications along with related projects that were carried during the thesis and conclude with future works.; La musique assistée par ordinateur fait beaucoup usage de librairies d’échantillons audios et d'instruments numériques qui offrent des possibilités de composition sans précédent. Cependant, l’abondance des matériaux sonores disponibles nécessite de nouvelles méthodes d’interaction en adéquation avec ceux-ci sans quoi le parcours des échantillons et configurations audios est inefficace et arbitraire. En effet, les métadonnées qui structurent traditionnellement ces librairies ne peuvent que traduire grossièrement les caractéristiques acoustiques des différentes catégories sonores. Notamment, les variations continues du timbre musical ne sont pas exprimées alors qu’elles jouent un rôle significatif dans la production et la créativité musicale. La perception du timbre a été étudiée par des testes d’écoute et l’analyse de ces résultats a permis la construction d’espaces de timbre dont les dimensions traduisent la similarité perceptive des différents sons. Cependant, ces espaces ne permettent pas d’analyser de nouveaux échantillons sonores et ils n’offrent aucun mécanisme inverse pour la génération audio.Les modèles de traitement du signal numérique permettent l’analyse et la synthèse, de telle manière que les paramètres extraits du son peuvent être manipulés pour la production de nouveaux sons. Bien que ces techniques soient performantes, elles nécessitent souvent l’ajustement de nombreux paramètres afin d’obtenir des reconstructions précises et leur visualisation est ardue de part leurs représentations à haute dimensionnalité. Ainsi, le contrôle des techniques basées sur le traitement du signal manque d’intuitivité et les dimensions de ces espaces de synthèse sont principalement liées à des propriétés de bas niveau du signal qui ont une valeur sémantique limitée. Les progrès des modèles d’apprentissage génératif ont démontré des capacités sans précédent pour le traitement des données à grande échelle. Ces méthodes probabilistes permettent la construction d’espaces non supervisés pour la synthèse de données telles que les images, le texte ou le son et ont permis de nouvelles interactions telles que la conversion automatique d’images et la manipulation d’attributs perceptifs et stylistiques.L’application des modèles d’apprentissage profond pour la génération audio a pris un essor au cours des dernières années et ce développement requiert des architectures adaptées ainsi que la conception d’interactions spécifiquement pensées pour la synthèse sonore. La synthèse directe de forme d’onde par des processus auto-régressifs a établi l’état de l’art pour la production de la voix et des sons musicaux. Bien qu’ils atteignent une haute qualité, ces modèles requièrent des puissances de calcul prohibitives et ne sont pas efficaces sur des bases de données de tailles limitées. De plus, les mécanismes auto-régressifs ont une modélisation locale performante mais leurs représentations et interactions sur les propriétés à long terme sont limitées. Au cours de cette thèse, nous développons des techniques d’analyse/synthèse efficaces basées sur les modèles auto-encodeurs afin d’apprendre des représentations acoustiques inversibles de basse dimensionnalité pour la manipulation intuitive du timbre musical. En premier lieu, nous adaptons les techniques non supervisées de conversion d’images au transfert de propriétés de timbre. Nous proposons des objectifs alternatifs à l’entrainement par réseaux antagonistes génératifs qui permettent le transfert entre de multiples domaines, tels que des collections d’échantillons audios de différents instruments. Nous référons à cette approche comme une modélisation implicite du timbre qui est définit comme l’ensemble des propriétés qui ne sont pas partagées entre les différents domaines sonores. Ensuite, nous introduisons de nouveaux modèles pour l’apprentissage explicite de représentations du timbre musical et l’échantillonnage avec contrôle des propriétés acoustiques et sémantiques. Ces modèles s’appuient notamment sur le conditionnement du réseau génératif (décodeur) par des attributs musicaux cibles et l’apprentissage hiérarchique de représentations acoustiques locales et séquentielles à plus long terme. De plus, nous appliquons l’apprentissage de représentation discrète pour la décomposition acoustique du timbre qui permet de quantifier et convertir d’autres sources audios par reconstruction avec les propriétés de timbre apprisent dans le domaine cible. Ce faisant, nous proposons une méthode d’analyse de cette représentation discrète par descripteurs acoustiques qui permet le contrôle direct de la synthèse de variations acoustiques cibles. Enfin, nous avons conduit une étude sur la réduction des modèles d’apprentissage profond pour le traitement et la synthèse audio qui permet de réduire drastiquement la taille et le cout de calcul nécessaires à leur déploiement sur des systèmes grand-public et embarqués. Ainsi, notre discussion et évaluation ne se concentrent pas seulement sur la performance d’apprentissage mais aussi sur les qualités d’interaction et l’efficacité de ces modèles pour un usage avec des ressources de calcul contraintes.L’organisation de cette thèse s’articule de la manière suivante. La première section introduit le traitement numérique de la musique, ses différents niveaux d’expression et pose la problématique de la synthèse audio avec les techniques d’apprentissage automatique. La seconde section détaille les propriétés des représentations de l’information musicale et sonore. Pour ce faire, nous récapitulons les méthodes classiques d’analyse et de synthèse ainsi que l’introduction des approches d’apprentissage. La troisième section détaille les fondements de l’apprentissage non supervisé et les principaux modèles génératifs de la littérature. La quatrième section détaille les tâches et modèles de référence appliqués à la synthèse audio musicale. La cinquième section fait un compte rendu des expériences effectuées au cours de la thèse, les contributions et résultats sont alors présentés dans le format d’une “thèse par articles”. Enfin la sixième et dernière section conclut le manuscrit avec un résumé des travaux de recherche effectués, une discussion des projets conduits en parallèle de la thèse et les directions futures de recherche.

International audience; Dans la collection des Dialogiques d’Uzeste, le musicien de jazz Bernard Lubat dialogue avec les systèmes d'intelligence artificielle (IA) musicale créés par l’équipe Représentations musicales de l’Ircam et leurs concepteurs Gérard Assayag (Ircam-STMS) et Marc Chemillier (CAMS-EHESS).Ces cyber-improvisations musicales sont l'aboutissement de plus d'une décennie de complicité artistique, de recherche scientifique au plus haut niveau (désormais honorée par le très compétitif European Research Council Advanced Research Grant, ERC REACH), d'expérimentations tous azimuts et de développement d'une série de logiciels qui ont véritablement changé le monde de l'IA musicale (Omax, somax, ImproteK, Djazz, DYCI2).Le disque regroupe trois performances jazz enregistrées à New York, Philadelphie (lors des festivals Improtech organisés par les auteurs) et Uzeste, et six expériences in-vitro en studio. Le livre contient 168 pages bilingues d'appareil critique et philosophique sur les enjeux de la création partagée entre humains et machines par Bernard Lubat, Gérard Assayag, Marc Chemillier, et le grand pionnier du domaine, George Lewis.

International audience

Aerial hoops are circular, hanging devices for both acrobatic exercise and artistic performance that let us explore the role of interactive sonification in physical activity. We present SonicHoop, an augmented aerial hoop that generates auditory feedback via capacitive touch sensing, thus becoming a digital musical instrument that performers can play with their bodies. We compare three sonification strategies through a structured observation study with two professional aerial hoop performers. Results show that SonicHoop fundamentally changes their perception and choreographic processes: instead of translating music into movement, they search for bodily expressions that compose music. Different sound designs affect their movement differently, and auditory feedback, regardless of type of sound, improves movement quality. We discuss opportunities for using SonicHoop as an aerial hoop training tool, as a digital musical instrument, and as a creative object; as well as using interactive sonification in other acrobatic practices to explore full-body vertical interaction.

Music listening is a ubiquitous activity for most people. Music is communicated through performance; performers introduce acoustic variations to help listeners parse the music through segmentation and prominence. Such structures are widely studied from a computational point of view. Humans are able to abstract subtle, but important aspects of the structures conceived and shaped by the performer(s). CosmoNote is a citizen science tool to aid in the study of such performed structures. The interface provides representations of music along with custom visualizations of its prosodic and harmonic features. It enables users (non-musicians and musicians alike) to identify and mark music structures in performance, with training videos and examples. Attendees will learn about the CosmoNote platform, see annotation examples, and learn about performed music structures such as boundaries, transitions, groupings and tipping points., Escuchar música es una actividad omnipresente para la mayoría de la gente. La música se comunica a través de la interpretación; los intérpretes introducen variaciones acústicas para ayudar a los oyentes a analizar la música a través de segmentación y prominencia. Estas estructuras son ampliamente estudiadas desde un punto de vista computacional. El ser humano es capaz de abstraer aspectos sutiles pero importantes de estructuras concebidas y moldeadas por los intérpretes. CosmoNote es una herramienta de investigación para ayudar al estudio de estas estructuras en interpretación musical. La interfaz presenta representaciones musicales junto con visualizaciones personalizadas de rasgos prosódicos y armónicos de la música. Esta interfaz permite a los usuarios (tanto a no músicos como músicos) identificar y marcar estructuras musicales en una interpretación, utilizando un marco de Ciencia Ciudadana, con vídeos de capacitación y ejemplos. Los asistentes aprenderán sobre la plataforma CosmoNote, verán ejemplos de anotaciones y aprenderán sobre estructuras en música interpretada tales como fronteras, transiciones, agrupaciones y puntos de inflexión., L'écoute de la musique est une activité omniprésente pour la plupart des gens. La musique est communiquée par le biais de la performance; les artistes introduisent des variations acoustiques pour aider les auditeurs à analyser la musique par le biais de la segmentation et de la mise en évidence. Ces structures sont largement étudiées d'un point de vue informatique. Les humains sont capables d'abstraire des aspects subtils mais importants des structures conçues et façonnées par le ou les interprètes. CosmoNote est un outil de recherche destiné à faciliter l'étude de ces structures. L'interface présente des représentations de la musique ainsi que des visualisations personnalisées des caractéristiques prosodiques et harmoniques de la musique. Elle permet aux utilisateurs (non musiciens et musiciens) d'identifier et de marquer les structures musicales lors de l'exécution, en utilisant un cadre de Sciences Participatives, avec des vidéos de formation et des exemples. Les participants découvriront la plateforme CosmoNote, verront des exemples d'annotation et apprendront à connaître les structures musicales exécutées telles que les frontières, les transitions, les regroupements et les points de bascule.

Negative body perceptions are a major predictor of physical inactivity, a serious health concern. Sensory feedback can be used to alter such body perceptions; movement sonification, in particular, has been suggested to affect body perception and levels of physical activity (PA) in inactive people. We investigated how metaphorical sounds impact body perception and PA. We report two qualitative studies centered on performing different strengthening/flexibility exercises using SoniBand, a wearable that augments movement through different sounds. The first study involved physically active participants and served to obtain a nuanced understanding of the sonifications’ impact. The second, in the home of physically inactive participants, served to identify which effects could support PA adherence. Our findings show that movement sonification based on metaphors led to changes in body perception (e.g., feeling strong) and PA (e.g., repetitions) in both populations, but effects could differ according to the existing PA-level. We discuss principles for metaphor-based sonification design to foster PA.

International audience; Influence of loudness on sound recognition was investigated in an explicit memory experiment based on a conscious recollection-test phase-of previously encoded information-study phase. Three encoding conditions were compared: semantic (sounds were sorted in three different categories), sensory (sounds were rated in loudness), and control (participants were solely asked to listen to the sounds). Results revealed a significant study-to-test change effect: loudness change between the study and the test phases affects recognition. The effect was not specific to the encoding condition (semantic vs sensory) suggesting that loudness is an important hint for everyday sounds recognition. [Q-JF] Date

International audience; We describe the Tonnetz web environment and some of the possible applications we have developed within a pedagogical workshop on mathematics and music that has been conceived for high-school students. This web environment makes use of two geometrical representations that constitute intuitive ways of accessing some theoretical concepts underlying the equal tempered system and their possible mathematical formalizations. The environment is aimed at enhancing "mathemusical" learning processes by enabling the user to interactively manipulate these representations. Finally, we show how Tonnetz is currently being adapted in order to lead computer-based experiences in music perception and cognition that will be mainly carried at universities. These experiences will explore the way in which geometrical models could be implicitly encoded during the listening process. Their outcome may reinforce educational strategies for learning mathematics through music.

International audience; Although it is known that imitating other's displays during conversations can lead to better conversation outcomes, it is not known if these effects are mediated by participants' sharing of emotional states or by emergent social dynamics. Using real time face manipulation during a speed dating task, we found evidence for a causal and emergent effect of expressive alignment on conversation outcomes.

International audience

International audience; The aim of the SONAO project is to establish new methods based on sonification of expressive movements for achieving a robust interaction between users and humanoid robots. We want to achieve this by combining competences of the research team members in the fields of social robotics, sound and music computing, affective computing, and body motion analysis. We want to engineer sound models for implementing effective mappings between stylized body movements and sound parameters that will enable an agent to express high-level body motion qualities through sound. These mappings are paramount for supporting feedback to and understanding robot body motion. The project will result in the development of new theories, guidelines, models, and tools for the sonic representation of high-level body motion qualities in interactive applications. This work is part of the growing research field known as data sonification, in which we combine methods and knowledge from the fields of interactive sonification, embodied cognition, multisensory perception, non-verbal and gestural communication in robots.

extended version of CMMR 2019 conference article hal-02172427; International audience; DJ techniques are an important part of popular music culture. However, they are also not sufficiently investigated by researchers due to the lack of annotated datasets of DJ mixes. Thus, this paper aims at filling this gap by introducing novel methods to automatically deconstruct and annotate recorded mixes for which the constituent tracks are known. A rough alignment first estimates where in the mix each track starts, and which time-stretching factor was applied. Second, a sample-precise alignment is applied to determine the exact offset of each track in the mix. Third, we propose a new method to estimate the cue points and the fade curves which operates in the time-frequency domain to increase its robustness to interference with other tracks. The proposed methods are finally evaluated on our new publicly available DJ-mix dataset UnmixDB. This dataset contains automatically generated beat-synchronous mixes based on freely available music tracks, and the ground truth about the placement, transformations and effects of tracks in a mix.

International audience

The success of human cooperation crucially depends on mechanisms enabling individuals to detect unreliability in their conspecifics. Yet, how such epistemic vigilance is achieved from naturalistic sensory inputs remains unclear. Here we show that listeners’ perceptions of the certainty and honesty of other speakers from their speech are based on a common prosodic signature. Using a data-driven method, we separately decode the prosodic features driving listeners’ perceptions of a speaker’s certainty and honesty across pitch, duration and loudness. We find that these two kinds of judgments rely on a common prosodic signature that is perceived independently from individuals’ conceptual knowledge and native language. Finally, we show that listeners extract this prosodic signature automatically, and that this impacts the way they memorize spoken words. These findings shed light on a unique auditory adaptation that enables human listeners to quickly detect and react to unreliability during linguistic interactions., It remains unclear whether and how listeners can infer speakers’ reliability from the sound of their voice. Here, the authors show that listeners from several languages rely on a common prosodic signature to infer whether speakers are lying or doubting.

Rapid technological advances in artificial intelligence are creating opportunities for real-time algorithmic modulations of a person’s facial and vocal expressions, or ‘deep-fakes’. These developments raise unprecedented societal and ethical questions which, despite much recent public awareness, are still poorly understood from the point of view of moral psychology. We report here on an experimental ethics study conducted on a sample of N = 303 participants (predominantly young, western and educated), who evaluated the acceptability of vignettes describing potential applications of expressive voice transformation technology. We found that vocal deep-fakes were generally well accepted in the population, notably in a therapeutic context and for emotions judged otherwise difficult to control, and surprisingly, even if the user lies to their interlocutors about using them. Unlike other emerging technologies like autonomous vehicles, there was no evidence of social dilemma in which one would, for example, accept for others what they resent for themselves. The only real obstacle to the massive deployment of vocal deep-fakes appears to be situations where they are applied to a speaker without their knowing, but even the acceptability of such situations was modulated by individual differences in moral values and attitude towards science fiction. This article is part of the theme issue ‘Voice modulation: from origin and mechanism to social impact (Part II)’.

This paper is an overview of the new score objects and editors available in the OM# visual programming and computer-assisted composition environment.

International audience; Imitation is one of the core building blocks of human social cognition, supporting capacities as diverse as empathy, social learning, and knowledge acquisition 1 . Newborns’ ability to match others’ motor acts, while quite limited initially, drastically improves during the first months of development 2 . Of notable importance to human sociality is our tendency to rapidly mimic facial expressions of emotion. Facial mimicry develops around six months of age 3 , but because of its late emergence, the factors supporting its development are relatively unknown. One possibility is that the development of facial mimicry depends on seeing emotional imitative behavior in others 4 . Alternatively, the drive to imitate facial expressions of emotion may be independent of visual learning and be supported by modality-general processes. Here we report evidence for the latter, by showing that congenitally blind participants facially imitate smiles heard in speech, despite having never seen a facial expression.

International audience; This paper discusses findings from a survey on interfaces for making electronic music. We invited electronic music makers of varying experience to reflect on their practice and setup and to imagine and describe their ideal interface for music-making. We also asked them to reflect on the state of gestural controllers, machine learning, and artificial intelligence in their practice. We had 118 people respond to the survey, with 40.68% professional musicians, and 10.17% identifying as living with a disability or access requirement. Results highlight limitations of music-making setups as perceived by electronic music makers, reflections on how imagined novel interfaces could address such limitations, and positive attitudes towards ML and AI in general.

International audience; The Cosmologies project aims to situate the listener inside a virtual grand piano by enabling computer processes to learn from the spatial presence of the live instrument and performer. We propose novel techniques that leverage measurements of natural acoustic phenomena to inform spatial sound composition and synthesis. Measured radiation patterns of acoustic instruments are applied interactively in response to a live input to synthesize spatial forms in real time. We implement this with software tools for the first time connecting audio descriptor analysis and corpus-based synthesis to spatialization using Higher-Order Ambisonics and machine learning. The resulting musical work, Cosmologies for piano and 3D electronics, explodes the space inside the grand piano out to the space of the concert hall, allowing the listener to experience its secret inner life.

International audience

We describe an art–science project called “Feral Interactions—The Answer of the Humpback Whale” inspired by humpback whale songs and interactions between individuals based on mutual influences, learning process, or ranking in the dominance hierarchy. The aim was to build new sounds that can be used to initiate acoustic interactions with these whales, not in a one-way direction, as playbacks do, but in real interspecies exchanges. Thus, we investigated how the humpback whales generate sounds in order to better understand their abilities and limits. By carefully listening to their emitted vocalizations, we also describe their acoustic features and temporal structure, in a scientific way and also with a musical approach as it is done withmusique concrète, in order to specify the types and the morphologies of whale sounds. The idea is to highlight the most precise information to generate our own sounds that will be suggested to the whales. Based on the approach developed inmusique concrète, similarities with the sounds produced by bassoon were identified and then were processed to become “concrete sound elements.” This analysis also brought us to design a new music interface that allows us to create adapted musical phrases in real-time. With this approach, interactions will be possible in both directions, from and to whales.

International audience; This paper presents exploratory work on sonic and visual representations of heartbeats of a COVID-19 patient and a medical team. The aim of this work is to sonify heart signals to reflect how a medical team comes together during a COVID-19 treatment, i.e. to highlight other aspects of the COVID-19 pandemic than those usually portrayed through sonification, which often focuses on the number of cases. The proposed framework highlights synergies between sound and heart signals through mapping between timefrequency coherence (TFC) of heart signals and harmonic tension and dissonance in music. Results from a listening experiment suggested that the proposed mapping between TFC and harmonic tension was successful in terms of communicating low versus high coherence between heart signals, with an overall accuracy of 69%, which was significantly higher than chance. In the light of the performed work, we discuss how links between heart-and sound signals can be further explored through sonification to promote understanding of aspects related to cardiovascular health.

Ce volume contient une sélection révisée et étendue des contribution au colloque Rigueur organisé par le Centre de Mathématiques Laurent Schwartz (École polytechnique et CNRS) à l’École des Hautes Études en Sciences Sociales les 2 et 3 juillet 2019.; International audience; Résumé du chapitre : Si la rigueur est une nécessité pour l’informaticien qui s’adresse à une machine, peut-être un idéal pour le mathématicien ou le philosophe qui doivent convaincre leurs collègues, quelle place a-t-elle en musique pour le compositeur et l’interprète? La partition est parfois comparée à un programme et dans le cas de la musique mixte, qui combine instrumentistes humains et réponses électroniques, la partition du compositeur est effectivement augmentée d’un programme informatique chargé de synthétiser les sons électroniques ou de transformer le son des instruments. Examiner la notion de rigueur dans le champ musical à partir de ses acceptions informatiques n’est donc pas dénué d’intérêt. Nous nous y essayerons à partir de quelques outils informatiques utilisés en composition assistée par ordinateur et pour l’électronique en temps réel.A propos de l'ouvrage (4ème de couverture) :Ce volume propose d’interroger et de confronter les idées de rigueur dans les trois disciplines ayant leur propre langage que sont les mathématiques, la musique et la philosophie, afin de créer, entre autres, un axe permettant de mieux sentir leur légendaire interpénétration.À une époque où le discours public manque dangereusement de rigueur intellectuelle, au profit de toujours plus d’image et de chiffre, il a semblé utile de convier philosophes, musiciens et mathématiciens soucieux de venir témoigner de ce qui, dans leur propre démarche disciplinaire de chercheur, convoque ce qui peut s'apparenter à une démarche rigoureuse.Si la notion de rigueur semble intrinsèque aux mathématiques, le lecteur trouvera bien des surprises en rencontrant en math des rigueurs esthétiques, intuitives et un jeu subtil entre rigueur et sens. De même découvrira-t-il l’importance d’une dynamique rigoureuse nécessaire à la création musicale, qui lui semblera peut-être transverse à l’idée de sensible, tout comme la force d’une rigueur, toujours centrale et libératrice, du discours philosophique.Auteurs : G. Drouin, J-L. Giavitto, N. Giménez Comas, G. Heinzmann, S. Jaffard, J. Lassègue, G. Longo, R. Natalini, F. Patras, T. Paul, M. Schmidt.

International audience; Representation and analysis of musical structures in heart signals can benefit understanding of cardiac electrophysiology aberrations such as arrhythmias, which can in turn aid in the diagnosis and treatment of cardiac arrhythmias. The typical time-frequency analysis of electrocardiographic recordings of cardiac arrhythmias yield descriptive statistics that provide useful features for classification, but fail to capture the actual rhythms of the physiological phenomena. Here, I propose to use music notation to represent beat-to-beat and morphological feature-to-feature durations of abnormal cardiac rhythms, using articulation markings when emphasis is warranted. The rhythms and articulations captured in these representations may provide cues to differentiate between individual experiences of cardiac arrhythmia, with potential impact on personalising diagnostics and treatment decisions. Music generation is presented as an application of these rhythm transcriptions. The physiological origins ensure that the music based on heart rhythms, even abnormal ones, sound natural. The two-part music creation process draws inspiration from music collage practices, and comprises of a retrieval component followed by transformation processes, which can be applied at the melody or block levels, and complex combinations thereof. The music thus created can not only be used to identify distinct heart signatures and what they mean for different cardiac conditions, they can also provide a visceral record of the experience of an arrhythmic episode. The pounding and fluttering of arrhythmia can often be physically uncomfortable. The music created from arrhythmic traces is not easy listening; it is often provocative, but potentially instructive.

International audience; Emotions are often accompanied by vocalizations whose acoustic features provide information about the physiological state of the speaker. Here, we ask if perceiving these affective signals in one’s own voice has an impact on one’s own emotional state, and if it is necessary to identify these signals as self-originated for the emotional effect to occur. Participants had to deliberate out loud about how they would feel in various familiar emotional scenarios, while we covertly manipulated their voices in order to make them sound happy or sad. Perceiving the artificial affective signals in their own voice altered participants’ judgements about how they would feel in these situations. Crucially, this effect disappeared when participants detected the vocal manipulation, either explicitly or implicitly. The original valence of the scenarios also modulated the vocal feedback effect. These results highlight the role of the exteroception of self-attributed affective signals in the emergence of emotional feelings.