Search

International audience

International audience; In this paper, we outline a diffractive practice of machine learning (ML) in the frame of material-centered interaction design. To this aim, we review related work in ML, HCI, design, new interfaces for musical expression, and computational art, and introduce two practice-based studies of music performance and robotic art based on interactive machine learning tools, with the hope of revealing the computational materiality of ML, and the potential of embodiment to craft prototypes of ML that reconfigure conceptual or technical approaches to ML. We derive five interference conditions for such art-based ML prototypes—situational whole, small data, shallow model, learnable algorithm, and somaesthetic behaviour—and describe their widening of design and engineering practices of ML prototyping. Finally, we sketch how a process of intra-active machine learning could complement that of interactive machine learning to take materiality as an entry point for ML design within HCI.

We report on a system that enables the synchronization of electroacoustic sounds with conductor gestures. This system was specifically used in the context of the music composition 'IDEA', for instrumental ensemble and live electronics. After a series of experiments and testing, the system was used in two public performances with two different conductors. We describe in this paper the development and the implementation of the gesture following system and the data recording. In particular, the system requires the full-length recording of the conductor's right hand movements with an inertial measurement unit. The patches and the dataset are made publicly available, along with the recording of other pieces collected during the preliminary experiments. We believe this material can be used to test other following systems, to study and learn conducting gestures, as well as for more general computational analysis of movements.

International audience; Entrain is an adaptive agent designed to stimulate social interaction among users in collective music making. It registers individual user behavior and provides sonic feedback to encourage users to look up from their mobiles and interact with each other. We demonstrate a use case of Entrain in Coloop, a distributed and collective musical instrument.

International audience; The use of continuous auditory feedback for motor control and learning is still understudied and deserves more attention regarding fundamental mechanisms and applications. This paper presents the results of three experiments studying the contribution of task-, error-, and user-related sonification to visuo-manual tracking and assessing its benefits on sensorimotor learning. First results show that sonification can help decreasing the tracking error, as well as increasing the energy in participant’s movement. In the second experiment, when alternating feedback presence, the user-related sonification did not show feedback dependency effects, contrary to the error and task-related feedback. In the third experiment, a reduced exposure of 50% diminished the positive effect of sonification on performance, whereas the increase of the average energy with sound was still significant. In a retention test performed on the next day without auditory feedback, movement energy was still superior for the groups previously trained with the feedback. Although performance was not affected by sound, a learning effect was measurable in both sessions and the user-related group improved its performance also in the retention test. These results confirm that a continuous auditory feedback can be beneficial for movement training and also show an interesting effect of sonification on movement energy. User-related sonification can prevent feedback dependency and increase retention. Consequently, sonification of the user’s own motion appears as a promising solution to support movement learning with interactive feedback.

cote interne IRCAM: Caramiaux12a; None / None; National audience; This article presents a segmentation model applied to musician movements, taking into account different time structures. In particular we report on ancillary gestures that are not directly linked to sound production, whilst still being entirely part of the global instrumental gesture. Precisely, we study movements of the clarinet captured with an optical 3D motion capture system, analysing ancillary movements assuming that they can be considered as a sequence of primitive actions regarded as base shapes. A stochastic model called the Segmental Hidden Markov Model is used. It allows for the representation of a continuous trajectory as a sequence of primitive temporal profiles taken from a given dictionary. We evaluate the model using two criteria: the Euclidean norm and the log-likelihood, then show that the size of the dictionary does not influence the fitting accuracy and propose a method for building a dictionary based on the log-likelihood criterion. Finally, we show that the sequence of primitive shapes can also be considered as a sequence of symbols enabling us to interpret the data as symbolic patterns and motifs. Based on this representation, we show that circular patterns occur in all players' performances. This symbolic step produces a different layer of interpretation, linked to a larger time scale, which might not be obvious from a direct signal representation.

cote interne IRCAM: Bevilacqua12a; None / None; National audience; We present interdisciplinary research undertaken for the development of an ‘augmented’ string quartet. Hardware and software components were especially designed to enable mixed acoustic/electronic music where bow gestures drive digital sound processes. Precisely, inertial motion sensors and a bow force sensor were added to each musician's bow, and dedicated modules allowed forthe wireless data transmission to an on-line gesture analysis system. Prior to the performance, a research phase was performed to evaluate qualitatively the variability of the gesture data. Recording sessions of both gesture and audio data were carried out with a professional string quartet. The music material included a set of prototypical musical phrases containing various bowing styles and playing techniques as well as a complete music composition. The analysis of the recorded sessions allowed us to compare the consistency within and between players. While a given player was found to be generally consistent, the comparison between players revealed significant gesture idiosyncrasies. These results helped us to adapt a real-time gesture analysis system called the gesture follower. This tool was successful to automatically synchronize the live performance with electronic sound transformation in two concerts. A quantitative assessment is reported on a specific section of the piece, illustrating the accuracy and the types of errors encountered.

International audience; Communicating about sounds is a difficult task without a technical language, and naïve speakers often rely on different kinds of non-linguistic vocalizations and body gestures (Lemaitre et al. 2014). Previous work has independently studied how effectively people describe sounds with gestures or vocalizations (Caramiaux, 2014, Lemaitre and Rocchesso, 2014). However, speech communication studies suggest a more intimate link between the two processes (Kendon, 2004). Our study thus focused on the combination of manual gestures and non-speech vocalizations in the communication of sounds. We first collected a large database of vocal and gestural imitations of a variety of sounds (audio, video, and motion sensor data). Qualitative analysis of gestural strategies resulted in three hypotheses: (1) voice is more effective than gesture for communicating rhythmic information, (2) textural aspects are communicated with shaky gestures, and (3) concurrent streams of sound events can be split between gestures and voice. These hypotheses were validated in a second experiment in which 20 participants imitated 25 specifically synthesized sounds: rhythmic noise bursts, granular textures, and layered streams. Statistical analyses compared acoustics features of synthesized sounds, vocal features, and a set of novel gestural features based on a wavelet representation of the acceleration data.

In the last 4 years, we have developed a partnership between dance and neuroscience to study the relationships between body space in dance and the surrounding space, and the link between movement and audition as experienced by the dancer. The opportunity to work with a dancer/choreographer, an expert in movement, gives neuroscientists better access to the significance of the auditory-motor loop and its role in perception of the surrounding space. Given that a dancer has a very strong sense of body ownership (probably through a very accurate dynamic body schema) ( Walsh et al. 2011 ), she is an ideal subject to investigate the feeling of controlling one's own body movements, and, through them, events in the external environment ( Moore et al. 2009 , Jola et al in press). We conducted several work sessions, which brought together a choreographer/dancer, a neuroscientist, a composer, and two researchers in acoustics and audio signal processing. These sessions were held at IRCAM (Institute for Research and Coordination Acoustic/Music, Paris) in a variable-acoustics concert hall equipped with a Wave Field Synthesis (WFS) sound reproduction system and infrared cameras for motion capture. During these work sessions, we concentrated on two specific questions: 1) is it possible to extend the body space of the dancer through auditory feedback ( Maravita and Iriki 2004 )? and 2) can we alter the dancer's perception of space by altering perceptions associated with movements? We used an interactive setup in which a collection of pre-composed sound events (individual sounds or musical sentences) could be transformed and rendered in real time according to the movements and the position of the dancer, that were sensed by markers on her body and detected by a motion tracking system. The transformations applied to the different sound components through the dancer's movement and position concerned not only musical parameters such as intensity, timbre, etc. but also the spatial parameters of the sounds. The technology we used allowed us to control their trajectory in space, apparent distance and the sound reverberation ambiance. We elaborated a catalogue of interaction modes with auditory settings that changed according to the dancer's movements. An interaction mode is defined by different mappings of position, posture or gesture of the dancer to musical and spatial parameters. For instance, a sound event may be triggered if the dancer is within a certain region or if she performs a predefined gesture. More elaborated modes involved the modulation of musical parameters by continuous movements of the dancer. The pertinence at a perceptive and cognitive level of the catalogue of interactions has been tested throughout the sessions. We observed that the detachable markers could be used to create a perception of extended body space, and that the performer perceived the stage space differently according to the auditory feedback of her action. The dancer reported that each experience with the technology shed light on her need for greater awareness and exploration of her relationships with space. Real-time interactivity with sound heightened her physical awareness – as though the stage itself took on a role and became another character.

The notion of “movement qualities” is central in contemporary dance; it describes the manner in which a movement is executed. Movement qualities convey information revealing movement expressiveness; their use has strong potential for movement-based interaction with applications in arts, entertainment, education, or rehabilitation. The purpose of our research is to design and evaluate interactive reflexive visuals for movement qualities. The theoretical basis for this research is drawn from a collaboration with the members of the international dance company Emio Greco|PC to study their formalization of movement qualities. We designed a pedagogical interactive installation called Double Skin/Double Mind (DS/DM) for the analysis and visualization of movement qualities through physical model-based interactive renderings. In this article, we first evaluate dancers’ perception of the visuals as metaphors for movement qualities. This evaluation shows that, depending on the physical model parameterization, the visuals are capable of generating dynamic behaviors that the dancers associate with DS/DM movement qualities. Moreover, we evaluate dance students’ and professionals’ experience of the interactive visuals in the context of a dance pedagogical workshop and a professional dance training. The results of these evaluations show that the dancers consider the interactive visuals to be a reflexive system that encourages them to perform, improves their experience, and contributes to a better understanding of movement qualities. Our findings support research on interactive systems for real-time analysis and visualization of movement qualities, which open new perspectives in movement-based interaction design.

cote interne IRCAM: Caramiaux14a; National audience; We investigated gesture description of sound stimuli performed during a listening task. Our hypothesis is that the strategies in gestural responses depend on the level of identification of the sound source and specifically on the identification of the action causing the sound. To validate our hypothesis, we conducted two experiments. In the first experiment, we built two corpora of sounds. The first corpus contains sounds with identifiable causal actions. The second contains sounds for which no causal actions could be identified. These corpora properties were validated through a listening test. In the second experiment, participants performed arm and hand gestures synchronously while listening to sounds taken from these corpora. Afterward, we conducted interviews asking participants to verbalize their experience while watching their own video recordings. They were questioned on their perception of the listened sounds and on their gestural strategies. We showed that for the sounds where causal action can be identified, participants mainly mimic the action that has produced the sound. In the other case, when no action can be associated with the sound, participants trace contours related to sound acoustic features. We also found that the interparticipants’ gesture variability is higher for causal sounds compared to noncausal sounds. Variability demonstrates that, in the first case, participants have several ways of producing the same action, whereas in the second case, the sound features tend to make the gesture responses consistent.

cote interne IRCAM: Caramiaux14c; None / None; National audience; Gesture-to-sound mapping is generally defined as the association between gestural and sound parameters. This article describes an approach that brings forward the perception–action loop as a fundamental design principle for gesture–sound mapping in digital music instrument. Our approach considers the processes of listening as the foundation—and the first step—in the design of action–sound relationships. In this design process, the relationship between action and sound is derived from actions that can be perceived in the sound. Building on previous work on listening modes and gestural descriptions, we propose to distinguish between three mapping strategies: instantaneous, temporal, and metaphorical. Our approach makes use of machine-learning techniques for building prototypes, from digital music instruments to interactive installations. Four different examples of scenarios and prototypes are described and discussed.

cote interne IRCAM: Boyer14a; National audience; This study introduces an experiment designed to analyze the sensorimotor adaptation to a motion-based sound synthesis system. We investigated a sound-oriented learning task, namely to reproduce a tar- geted sound. The motion of a small handheld object was used to control a sound synthesizer. The object angular velocity was measured by a gy- roscope and transmitted in real time wirelessly to the sound system. The targeted sound was reached when the motion matched a given refer- ence angular velocity profile with a given accuracy. An incorrect velocity profile produced either a noisier sound or a sound with a louder high har- monic, depending on the sign of the velocity error. The results showed that the participants were generally able to learn to reproduce sounds very close to the targeted sound. A corresponding motor adaptation was also found to occur, at various degrees, in most of the participants when the profile is altered.

This SIG intends to investigate the ongoing dialogue between music technology and the field of human-computer interaction. Our specific aims are to consider major findings of musical interface research over recent years and discuss how these might best be conveyed to CHI researchers interested but not yet active in this area, as well as to consider how to stimulate future collaborations between music technology and CHI research communities. This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published in CHI EA '13 Extended Abstracts in Human Factors in Computing Systems.

Chiseling Bodies is an interactive augmented dance performance, where a dancer interacts with abstract visuals. They are massive mass-spring systems whose dynamical behaviors are echoing the dancer's movement qualities.

A novel interface for the real-time control of interactive visuals through full body dance movements is designed with a specific focus on the notion of movement qualities (the manner in which the movement is executed). The system can recognise predefined movement qualities through gesture analysis. It also allows for the control of abstract visuals based on physical models, specifically mass-springs models, displaying graphical animations with ‘qualities’ reflecting the participants’ expressions. This work has been implemented and tested in an interactive installation called ‘Double Skin/Double Mind’. In that context, we were able to test the contribution of our system to dance pedagogy and to collect participants’ feedback. These preliminary tests suggest that the dancers can embrace the interaction with physical model-based visuals. They also reveal that the movement qualities of the visuals generated by our system fairly mirror the dancers’ own qualities.

Gesture-based interaction is widespread in touch screen interfaces. The goal of this paper is to tap the richness of expressive variation in gesture to facilitate continuous interaction. We achieve this through novel techniques of adaptation and estimation of gesture characteristics. We describe two experiments. The first aims at understanding whether users can control certain gestural characteristics and if that control depends on gesture vocabulary. The second study uses a machine learning technique based on particle filtering to simultaneously recognize and measure variation in a gesture. With this technology, we create a gestural interface for a playful photo processing application. From these two studies, we show that 1) multiple characteristics can be varied independently in slower gestures (Study 1), and 2) users find gesture-only interaction less pragmatic but more stimulating than traditional menu-based systems (Study 2).

cote interne IRCAM: FdiliAlaoui12b; None / None; National audience; In this paper, we explore the use of movement qualities as interaction modality. The notion of movement qualities is widely used in dance practice and can be understood as how the movement is performed, independently of its specific trajectory in space. We implemented our approach in the context of an artistic installation called A light touch. This installation invites the participant to interact with a moving light spot reacting to the hand movement qualities. We conducted a user experiment that showed that such an interaction based on movement qualities tends to enhance the user experience favouring explorative and expressive usage.

1 Abstract. This chapter presents a general framework for gesture-controlled audio processing. The gesture parameters are assumed to be multi-dimensional temporal profiles obtained from movement or sound capture systems. The analysis is based on machine learning techniques, comparing the incoming dataflow with stored templates. The mapping procedures between the gesture and the audio processing include a specific method we called temporal mapping. In this case, the temporal evolution of the gesture input is taken into account in the mapping process. We describe an example of a possible use of the framework that we experimented with in various contexts, including music and dance performances, music pedagogy and installations.

cote interne IRCAM: Leslie10a; None / None; National audience; The interactive installation GrainStick was developed as a collaboration between the composer Pierre Jodlowski and the European project Sound And Music For Everyone Everyday Everywhere Everyway (SAME) at Ircam, Paris. The interaction design of GrainStick presents a new development in multimodal interfaces and multichannel sound by allowing users control of their auditory scene through gesture analysis performed on infrared camera motion tracking and accelerometer data. Spatial rendering was performed on a wavefield synthesis system; we discuss the advantages held by this technology in collaborative and interactive applications of this kind.

We present an ensemble of tangible objects and software modules designed for musical interaction and performance. The tangible interfaces form an ensemble of connected objects communicating wirelessly. A central concept is to let users determine the final musical function of the objects, favoring customization, assembling, repurposing. This might imply assembling the wireless interfaces with existing everyday objects or musical instruments. Moreover, gesture analysis and recognition modules allow the users to define their own action/motion for the control of sound parameters. Various sound engines and interaction scenarios were built and experimented. Some examples that were developed in a music pedagogy context are described.

cote interne IRCAM: Bloit10a; None / None; National audience; We present a method to model sound descriptor temporal profiles using Segmental Models. Unlike standard HMM, such an approach allows for the modeling of fine structures of temporal profiles with a reduced number of states. These states, we called primitives, can be chosen by the user using prior knowledge, and assembled to model symbolic musical elements. In this paper, we describe this general methodology and evaluate it on a dataset made of of violin recording containing crescendo/decrescendo, glissando and sforzando. The results show that, in this context, the segmental model can segment and recognize these different musical elements with a satisfactory level.

cote interne IRCAM: Caramiaux09d; None / None; National audience; This article reports on the exploration of a method based on canonical correlation analysis (CCA) for the analysis of the relationship between gesture and sound in the context of music performance and listening. This method is a first step in the design of an analysis tool for gesture-sound relationships. In this exploration we used motion capture data recorded from sub jects performing free hand movements while listening to short sound examples. We assume that even though the relationship between gesture and sound might be more complex, at least part of it can be revealed and quantified by linear multivariate regression applied to the motion capture data and audio descriptors extracted from the sound examples. After outlining the theoretical background, the article shows how the method allows for pertinent reasoning about the relationship between gesture and sound by analysing the data sets recorded from multiple and individual subjects.

cote interne IRCAM: Leslie09a; None / None; National audience; Wavefield synthesis (WFS), the spatialization of audio through the recreation of a virtual source’s wavefront, is uniquely suited to interactive applications where listeners move throughout the rendering space and more than one listener is involved. This paper describes the features of WFS that make it useful for interactive applications, and takes a recent project at Ircam as a case study which demonstrates these advantages. The interactive installation GrainStick was developed as a collaboration between the composer Pierre Jodlowski and the European project Sound And Music For Everyone Everyday Everywhere Everyway (SAME) at Ircam, Paris. The interaction design of GrainStick presents a new development in multimodal interfaces and multichannel sound by allowing users control of their auditory scene through gesture analysis performed on infrared camera motion tracking and accelerometer data.

cote interne IRCAM: Rasamimanana08c; National audience; Anticipatory behaviours are known to occur in music performance, notably on the control movements of instruments such as piano or drums. We studied such effects on bowed string movements, corresponding to a case where the control on sound is continuous. Movements were measured with an optical motion capture system combined with sensors on the bow. Bowing movements were analysed and compared on the basis of underlying effort costs, determined from their velocity profiles. Precisely, we used movement models that assume that jerk or impulse are minimized. These models were synthesized based on measurement data and then compared to velocity and acceleration profiles. Results on various musical cases involving separate strokes, scales, mixed bowing techniques and rhythms showed that this methodology can account, to some extent, for the different effort strategies used by the players. The presented modelling provides evidence of anticipatory behaviour during bowing movements.

cote interne IRCAM: DeLahunta07a; None / None; National audience; In this essay the authors report on the findings of the one-day workshop called Choreographic Computations: Motion Capture and Analysis for Dance held at IRCAM (Institute for music/acoustic research and coordination) in Paris in June 2006. The focus of the workshop was on new innovations combining motion capture and computer-based techniques with choreography and performance. An international group of artist programmers and dance makers who are bringing complex algorithmic procedures into alignment with choreographic creation were invited to give presentations and discuss, with each other and the invited audience, the shared understanding of movement and gesture they are developing.

cote interne IRCAM: Rasamimanana09a; National audience; The understanding of different bowing strategies can provide key concepts for the modelling of music performance. We report here an exploratory study of bowing gestures for a viola player and a violin player in the case of bow strokes performed at different frequencies. Bow and arm movements as well as bow pressure on strings were measured respectively with a 3D optical motion capture system and a custom pressure sensor. While increasing bow stroke frequency, defined as the inverse time between two strokes, players did use different bowing movements as indicated from the measurement of bow velocity and arm joint angles. First, bow velocity profiles abruptly shift from a rectangle shape to a sinus shape. Second, while bow velocity is sinusoidal, an additional change is observed: the wrist and elbow relative phase shifts from out-of-phase to in-phase at the highest frequencies, indicating a possible change in the players coordinative pattern. We finally discuss the fact that only small differences are found in the sound while significant changes occur in the velocity / acceleration profiles.

cote interne IRCAM: Bevilacqua07a; None / None; National audience; We present in this paper a complete gestural interface built to support music pedagogy. The development of this prototype concerned both hardware and software components: a small wireless sensor interface including accelerometers and gyroscopes, and an analysis system enabling gesture following and recognition. A first set of experiments was conducted with teenagers in a music theory class. The preliminary results were encouraging concerning the suitability of these developments in music education.

International audience; Bien que le monde du son ait évolué de concert avec les progrès technologiques, le monde des images semble dominer notre perception. Par exemple, la réalité virtuelle (RV) est bien souvent la métaphore inconsciente d'une réalité virtuelle visuelle, et les applications de RV, qu'elles relèvent du jeu, de l'apprentissage ou même de l'art, s'adressent principalement à la vision. On sait l'importance des notions d'espace et d'interaction pour promouvoir la sensation de présence dans des environnements virtuels. Cependant, peu de travaux s'attachent à exploiter le potentiel fourni par les stimulations virtuelles s'adressant au système auditif de l'utilisateur. Nous présentons ici le projet EarToy, dont l'objectif vise à mettre à jour les liens existant entre audition et action, afin de construire des systèmes immersifs et interactifs basés sur la perception auditive.

cote interne IRCAM: Rasamimanana06a; None / None; National audience; We developed an ”augmented violin”, i.e. an acoustic instrument with added gesture capture capabilities to control electronic processes. We report here gesture analysis we performed on three different bow strokes, detache, martele and spiccato, using this augmented violin. Different features based on velocity and acceleration were considered. A linear discriminant analysis has been performed to estimate a minimum number of pertinent features necessary to model these bow stroke classes. We found that the maximum and minimum accelerations of a given stroke were efficient to parameterize the different bow stroke types, as well as differences in dynamics playing. Recognition rates were estimated using a kNN method with various training sets. We finally discuss that bow stroke recognition allows to relate the gesture data to music notation, while a bow stroke continuous parameterization can be related to continuous sound characteristics.

cote interne IRCAM: Bevilacqua05a; None / None; National audience; In this report, we describe our development on the Max/MSP toolbox MnM dedicated to mapping between gesture and sound, and more generally to statistical and machine learning methods. This library is built on top of the FTM library, which enables the efficient use of matrices and other data structures in Max/MSP. Mapping examples are described based on various matrix manipulations such as Single Value Decomposition. The FTM and MnM libraries are freely available. 2. RELATED WORKS Mapping strategies have been reviewed in recent papers and we refer the reader to references [1] and [2] for a comprehensive overview. Van Nort et al. [9] give a mathematical formulation of mapping

cote interne IRCAM: Bevilacqua05b; None / None; National audience; Our aim is to develop a gesture follower for performing arts, which indicates inreal-time the time correspondences between an observed gesture sequence and a fixed reference gesture sequence (or in other words to time warp in real-time theobserved sequence to the learned sequence). Applications concern for example choreography or (composed) music. The possibility to continuously establishcorrespondences between observed and reference gestures has important potentials. For example, this will enable the quantitative comparison of gesture data, at similarmoments, between live and previously recorded performances. Particularly, this should facilitate the computation of parameters related to interpretation andexpression. Moreover, this will also provide new tools to control and dynamically change mapping settings between gesture and digital media.

cote interne IRCAM: Schnell05a; None / None; National audience; This article presents FTM, a shared library and a set of modules extending the Max/MSP environment. It also gives a brief description of additional sets of modules based upon FTM. The article particularly addresses the community of researchers and musicians familiar with Max or Max-like programming environments such as Pure Data. FTM extends the signal and message data flow paradigm of Max permitting the representation and processing of complex data structures such as matrices, sequences or dictionaries as well as tuples, MIDI events or score elements (notes, silences, trills etc.). The consequent integration of references to complex data structures in the Max/MSP data flow opens new possibilities to the user in terms of powerful and efficient data representations and modularization of applications. FTM is the basis of several sets of modules for Max/MSP specialized on score following, sound analysis/re-synthesis, statistical modeling and data bank access. Designed for particular applications in automatic accompaniment, advanced sound processing and gestural analysis, the libraries use a common set of basic FTM data structures. They are perfectly interoperable while smoothly integrating into the modular programming paradigm of the host environment Max/MSP.

Music is a cultural and creative practice that enables humans to express a variety of feelings and intentions through sound. Machine learning opens many prospects for designing human expression in interactive music systems. Yet, as a Computer Science discipline, machine learning remains mostly studied from an engineering sciences perspective, which often exclude humans and musical interaction from the loop of the created systems. In this dissertation, I argue in favour of designing with machine learning for interactive music systems. I claim that machine learning must be first and foremost situated in human contexts to be researched and applied to the design of interactive music systems. I present four interdisciplinary studies that support this claim, using human-centred methods and model prototypes to design and apply machine learning to four situated musical tasks: motion-sound mapping, sonic exploration, synthesis exploration, and collective musical interaction. Through these studies, I show that model prototyping helps envision designs of machine learning with human users before engaging in model engineering. I also show that the final human-centred machine learning systems not only helps humans create static musical artifacts, but supports dynamic processes of expression between humans and machines. I call co-expression these processes of musical interaction between humans - who may have an expressive and creative impetus regardless of their expertise - and machines - whose learning abilities may be perceived as expressive by humans. In addition to these studies, I present five applications of the created model prototypes to the design of interactive music systems, which I publicly demonstrated in workshops, exhibitions, installations, and performances. Using a reflexive approach, I argue that the musical contributions enabled by such design practice with machine learning may ultimately complement the scientific contributions of human-centred machine learning. I claim that music research can thus be led through dispositif design, that is, through the technical realization of aesthetically-functioning artifacts that challenge cultural norms on computer science and music.; La musique est une pratique culturelle permettant aux êtres humains d'exprimer sensiblement leurs intentions à travers le son. L'apprentissage machine définit un ensemble de modèles permettant de nouvelles formes d'expression au sein desdits systèmes interactifs musicaux. Cependant, en tant que discipline informatique, l'apprentissage machine demeure essentiellement appliquée à la musique du point de vue des sciences de l'ingénieur, qui, très souvent, conçoit les modèles d'apprentissage sans tenir compte des interactions musicales prenant place entre humains et systèmes. Dans cette thèse, j'envisage la possibilité de mener des pratiques de design avec l'apprentissage machine pour les systèmes interactifs musicaux. Je soutiens que l'apprentissage machine doit avant tout être situé au sein d'un contexte humain afin d'être conçu et appliqué au design de systèmes interactifs musicaux. Pour défendre cette thèse, je présente quatre études interdisciplinaires, dans lesquelles j'introduis des modèles intermédiaires d'apprentissage, dits modèles-prototype, au sein de méthodes de conception centrées humain, afin d'appliquer l'apprentissage machine à quatre tâches musicales situées : le mapping mouvement-son, l'exploration sonore, l'exploration de la synthèse, et l'interaction musicale collective. À travers ces études, je montre que les modèles-prototype permettent de générer des idées de design pour l'apprentissage machine en amont de la phase d'ingénierie desdits modèles, ce en lien étroit avec les utilisateurs potentiels de ces systèmes. Je montre également que les systèmes d'apprentissage machine centrés humain résultant de ce processus de conception rendent possible des processus dynamiques d'expression entre les humains et les machines, allant au-delà de la création d'artefacts musicaux statiques. Je propose de nommer co-expression ces processus d'interaction musicale entre des êtres humains - faisant preuve d'un élan expressif et créatif quelque soit leur expertise musicale - et des machines - dont les capacités d'apprentissage peuvent être perçues comme expressives du point de vue de l'humain. En outre, je présente cinq systèmes interactifs musicaux conçus avec lesdits modèles-prototypes, et relate leurs restitutions publiques au sein d'ateliers, expositions, installations et performances. Par une approche réflexive, je montre que les contributions musicales apportées par des pratiques de design avec l'apprentissage machine peuvent, à terme, complémenter les contributions scientifiques apportées par les méthodes de conception centrées humain. Ainsi, je suggère que la recherche musicale peut être menée par le design de dispositifs interactifs musicaux, c'est-à-dire, par la réalisation technique d'artefacts esthétiquement fonctionnels remettant en cause les normes culturelles régissant l'informatique et la musique.

Music is a cultural and creative practice that enables humans to express a variety of feelings and intentions through sound. Machine learning opens many prospects for designing human expression in interactive music systems. Yet, as a Computer Science discipline, machine learning remains mostly studied from an engineering sciences perspective, which often exclude humans and musical interaction from the loop of the created systems.In this dissertation, I argue in favour of designing with machine learning for interactive music systems. I claim that machine learning must be first and foremost situated in human contexts to be researched and applied to the design of interactive music systems. I present four interdisciplinary studies that support this claim, using human-centred methods and model prototypes to design and apply machine learning to four situated musical tasks: motion-sound mapping, sonic exploration, synthesis exploration, and collective musical interaction.Through these studies, I show that model prototyping helps envision designs of machine learning with human users before engaging in model engineering. I also show that the final human-centred machine learning systems not only helps humans create static musical artifacts, but supports dynamic processes of expression between humans and machines. I call co-expression these processes of musical interaction between humans—who may have an expressive and creative impetus regardless of their expertise—and machines—whose learning abilities may be perceived as expressive by humans.In addition to these studies, I present five applications of the created model prototypes to the design of interactive music systems, which I publicly demonstrated in workshops, exhibitions, installations, and performances. Using a reflexive approach, I argue that the musical contributions enabled by such design practice with machine learning may ultimately complement the scientific contributions of human-centred machine learning. I claim that music research can thus be led through dispositif design, that is, through the technical realization of aesthetically-functioning artifacts that challenge cultural norms on computer science and music.; La musique est une pratique culturelle permettant aux êtres humains d'exprimer sensiblement leurs intentions à travers le son. L'apprentissage machine définit un ensemble de modèles permettant de nouvelles formes d'expression au sein desdits systèmes interactifs musicaux. Cependant, en tant que discipline informatique, l'apprentissage machine demeure essentiellement appliquée à la musique du point de vue des sciences de l'ingénieur, qui, très souvent, conçoit les modèles d'apprentissage sans tenir compte des interactions musicales prenant place entre humains et systèmes.Dans cette thèse, j'envisage la possibilité de mener des pratiques de design avec l'apprentissage machine pour les systèmes interactifs musicaux. Je soutiens que l'apprentissage machine doit avant tout être situé au sein d'un contexte humain afin d'être conçu et appliqué au design de systèmes interactifs musicaux. Pour défendre cette thèse, je présente quatre études interdisciplinaires, dans lesquelles j'introduis des modèles intermédiaires d'apprentissage, dits modèles-prototype, au sein de méthodes de conception centrées humain, afin d'appliquer l'apprentissage machine à quatre tâches musicales situées : le mapping mouvement-son, l'exploration sonore, l'exploration de la synthèse, et l'interaction musicale collective.À travers ces études, je montre que les modèles-prototype permettent de générer des idées de design pour l'apprentissage machine en amont de la phase d'ingénierie desdits modèles, ce en lien étroit avec les utilisateurs potentiels de ces systèmes. Je montre également que les systèmes d'apprentissage machine centrés humain résultant de ce processus de conception rendent possible des processus dynamiques d'expression entre les humains et les machines, allant au-delà de la création d'artefacts musicaux statiques. Je propose de nommer co-expression ces processus d'interaction musicale entre des êtres humains—faisant preuve d'un élan expressif et créatif quelque soit leur expertise musicale—et des machines—dont les capacités d'apprentissage peuvent être perçues comme expressives du point de vue de l'humain.En outre, je présente cinq systèmes interactifs musicaux conçus avec lesdits modèles-prototypes, et relate leurs restitutions publiques au sein d'ateliers, expositions, installations et performances. Par une approche réflexive, je montre que les contributions musicales apportées par des pratiques de design avec l'apprentissage machine peuvent, à terme, complémenter les contributions scientifiques apportées par les méthodes de conception centrées humain. Ainsi, je suggère que la recherche musicale peut être menée par le design de dispositifs interactifs musicaux, c'est-à-dire, par la réalisation technique d'artefacts esthétiquement fonctionnels remettant en cause les normes culturelles régissant l'informatique et la musique.

Music is a cultural and creative practice that enables humans to express a variety of feelings and intentions through sound. Machine learning opens many prospects for designing human expression in interactive music systems. Yet, as a Computer Science discipline, machine learning remains mostly studied from an engineering sciences perspective, which often exclude humans and musical interaction from the loop of the created systems. In this dissertation, I argue in favour of designing with machine learning for interactive music systems. I claim that machine learning must be first and foremost situated in human contexts to be researched and applied to the design of interactive music systems. I present four interdisciplinary studies that support this claim, using human-centred methods and model prototypes to design and apply machine learning to four situated musical tasks: motion-sound mapping, sonic exploration, synthesis exploration, and collective musical interaction. Through these studies, I show that model prototyping helps envision designs of machine learning with human users before engaging in model engineering. I also show that the final human-centred machine learning systems not only helps humans create static musical artifacts, but supports dynamic processes of expression between humans and machines. I call co-expression these processes of musical interaction between humans - who may have an expressive and creative impetus regardless of their expertise - and machines - whose learning abilities may be perceived as expressive by humans. In addition to these studies, I present five applications of the created model prototypes to the design of interactive music systems, which I publicly demonstrated in workshops, exhibitions, installations, and performances. Using a reflexive approach, I argue that the musical contributions enabled by such design practice with machine learning may ultimately complement the scientific contributions of human-centred machine learning. I claim that music research can thus be led through dispositif design, that is, through the technical realization of aesthetically-functioning artifacts that challenge cultural norms on computer science and music.; La musique est une pratique culturelle permettant aux êtres humains d'exprimer sensiblement leurs intentions à travers le son. L'apprentissage machine définit un ensemble de modèles permettant de nouvelles formes d'expression au sein desdits systèmes interactifs musicaux. Cependant, en tant que discipline informatique, l'apprentissage machine demeure essentiellement appliquée à la musique du point de vue des sciences de l'ingénieur, qui, très souvent, conçoit les modèles d'apprentissage sans tenir compte des interactions musicales prenant place entre humains et systèmes. Dans cette thèse, j'envisage la possibilité de mener des pratiques de design avec l'apprentissage machine pour les systèmes interactifs musicaux. Je soutiens que l'apprentissage machine doit avant tout être situé au sein d'un contexte humain afin d'être conçu et appliqué au design de systèmes interactifs musicaux. Pour défendre cette thèse, je présente quatre études interdisciplinaires, dans lesquelles j'introduis des modèles intermédiaires d'apprentissage, dits modèles-prototype, au sein de méthodes de conception centrées humain, afin d'appliquer l'apprentissage machine à quatre tâches musicales situées : le mapping mouvement-son, l'exploration sonore, l'exploration de la synthèse, et l'interaction musicale collective. À travers ces études, je montre que les modèles-prototype permettent de générer des idées de design pour l'apprentissage machine en amont de la phase d'ingénierie desdits modèles, ce en lien étroit avec les utilisateurs potentiels de ces systèmes. Je montre également que les systèmes d'apprentissage machine centrés humain résultant de ce processus de conception rendent possible des processus dynamiques d'expression entre les humains et les machines, allant au-delà de la création d'artefacts musicaux statiques. Je propose de nommer co-expression ces processus d'interaction musicale entre des êtres humains - faisant preuve d'un élan expressif et créatif quelque soit leur expertise musicale - et des machines - dont les capacités d'apprentissage peuvent être perçues comme expressives du point de vue de l'humain. En outre, je présente cinq systèmes interactifs musicaux conçus avec lesdits modèles-prototypes, et relate leurs restitutions publiques au sein d'ateliers, expositions, installations et performances. Par une approche réflexive, je montre que les contributions musicales apportées par des pratiques de design avec l'apprentissage machine peuvent, à terme, complémenter les contributions scientifiques apportées par les méthodes de conception centrées humain. Ainsi, je suggère que la recherche musicale peut être menée par le design de dispositifs interactifs musicaux, c'est-à-dire, par la réalisation technique d'artefacts esthétiquement fonctionnels remettant en cause les normes culturelles régissant l'informatique et la musique.

This thesis proposes a performer-specific paradigm of embodied interaction with complex piano notation. This paradigm, which I term embodied navigation, extends and even confronts the traditional paradigm of textual interpretation. The latter assumes a linear and hierarchical process, whereby internalized understanding of the musical text is considered a prerequisite of instrumental technique towards personal interpretation. In lieu of that, I advocate for a dynamic, non-linear, embodied and external processing of music notation. At a second stage, the proposed paradigm serves as the basis for the development of methodologies and customized tools for a range of applications, including: performance analysis, embodied interactive learning, contemporary composition, free improvisation and piano pedagogy., La thèse propose un paradigme d’interaction avec la notation musicale complexe pour piano selon une perspective « incarnée » et «située » de l’interprète. Ce paradigme, que je nomme navigation incarnée, s’oppose au paradigme traditionnel d’interprétation textuelle. Le paradigme traditionnel considère un processus de lecture linéaire et hiérarchique, selon lequel la compréhension et l’internalisation du texte musical sont les conditions préalables pour l’application de la technique instrumentale, permettant par la suite une interprétation personnelle. À la place de ce paradigme, je propose de traiter la notation musicale comme un élément dynamique, non linéaire, et à la fois incarné et externalisé. Dans une deuxième phase, le paradigme proposé devient la base du développement d’outils adaptés au projet de la navigation incarnée et de diverses applications, incluant l’analyse de la performance, l’apprentissage incarné et interactif, la composition musicale et l’improvisation.

Pas de résumé disponible.; Pas de résumé disponible.