Search

International audience; Timbre provides an important cue to identify musical instruments. Many timbral attributes covary with other parameters like pitch. This study explores listeners' ability to construct categories of instrumental sound sources from sounds that vary in pitch. Nonmusicians identified 11 instruments from the woodwind, brass, percussion, and plucked and bowed string families. In experiment 1, they were trained to identify instruments playing a pitch of C4, and in experiments 2 and 3, they were trained with a five-tone sequence (F#3–F#4), exposing them to the way timbre varies with pitch. Participants were required to reach a threshold of 75% correct identification in training. In the testing phase, successful listeners heard single tones (experiments 1 and 2) or three-tone sequences from (A3–D#4) (experiment 3) across each instrument's full pitch range to test their ability to generalize identification from the learned sound(s). Identification generalization over pitch varies a great deal across instruments. No significant differences were found between single-pitch and multi-pitch training or testing conditions. Identification rates can be predicted moderately well by spectrograms or modulation spectra. These results suggest that listeners use the most relevant acoustical invariance to identify musical instrument sounds, also using previous experience with the tested instruments.

What is the function of auditory hemispheric asymmetry? We propose that the identification of sound sources relies on two complementary and perceptually relevant acoustic invariants — actions and objects — that are processed asymmetrically and sufficient to model and categorize any sound. We observed that environmental sounds are an independent combination of purely temporal and spectral acoustic modulations. Behaviorally, discrimination of actions relied on temporal modulations, while discrimination of objects relied on spectral modulations. Functional magnetic resonance imaging data showed that actions and objects are respectively decoded in the left and right hemispheres, in bilateral superior temporal and left inferior frontal regions. This asymmetry reffects a generic differential processing — through differential neural sensitivity to temporal and spectral modulations present in all environmental sounds — that supports the efficient categorization of actions and objects. These results provide an ecologically valid framework of the functional role of auditory brain asymmetry.

Within certain categories of geometric shapes, prototypical exemplars that best characterize the category have been evidenced. These geometric prototypes are classically identified through the visual and haptic perception or motor production and are usually characterized by their spatial dimension. However, whether prototypes can be recalled through the auditory channel has not been formally investigated. Here we address this question by using auditory cues issued from timbre-modulated friction sounds evoking human drawing elliptic movements. Since non-spatial auditory cues were previously found useful for discriminating distinct geometric shapes such as circles or ellipses, it is hypothesized that sound dynamics alone can evoke shapes such as an exemplary ellipse. Four experiments were conducted and altogether revealed that a common elliptic prototype emerges from auditory, visual, and motor modalities. This finding supports the hypothesis of a common coding of geometric shapes according to biological rules with a prominent role of sensory-motor contingencies in the emergence of such prototypical geometry.

We show that sleep deprivation in otherwise normal and healthy adults can be detected through machine-learning analysis of vocal recordings. Importantly, we used fully generic acoustic features, derived from auditory models, together with our own machine learning interpretation method, derived from neuroscience. Sleep deprivation impacted two broad types of acoustic features: one related to speech rhythms, the other related to the timbre of the voice. Such features plausibly reflect two independent physiological processes: one explicit, the cognitive control of speech production, and the other implicit, the inflammation of the vocal apparatus. Crucially, the relative balance of the two processes varied widely across individuals, consistent with the known but unexplained variability in responses to sleep deprivation. Overall, our results suggest that the voice may be used as a “sleep stethoscope” to characterize the individual effects of sleep deprivation. Moreover, the method we applied is fully general and could be adapted to any future investigation of vocal biomarkers using machine-learning techniques.Author summarySleep deprivation has an ever-increasing impact on individuals and societies, from accidents to chronic conditions costing billions to health systems. Yet, to date, there is no quick and objective test for sleep deprivation. We show that sleep deprivation can be detected at the individual level with voice recordings, outlining future cost-effective and non-invasive “sleep stethoscopes”. Importantly, we focused on interpretability, which identified two independent physiological effects of sleep deprivation: a change in prosody, related to cognitive control, and a change in timbre, related to inflammation. This also revealed a striking variability in individual reactions to the same deprivation. The neuroscientific framework we developed, combining auditory models and machine learning, is freely available and could be adapted to any vocal biomarker.

International audience; Auditory roughness resulting from fast temporal beatings is often studied by summing two pure tones with close frequencies. Interestingly, the tactile counterpart of auditory roughness can be provided through touch with vibrotactile actua- tors. However, whether auditory roughness could also be perceived through touch and whether it exhibits similar characteris- tics are unclear. Here, auditory roughness perception and its tactile counterpart were evaluated using pairs of pure tone stimuli. Results revealed similar roughness curves in both modalities, suggesting similar sensory processing. This study attests to the relevance of such a paradigm for investigating auditory and tactile roughness in a multisensory fashion.

Timbre perception and auditory grouping principles can provide a theoretical basis for aspects of orchestration. In Experiment 1, 36 excerpts contained two streams and 12 contained one stream as determined by music analysts. Streams—the perceptual connecting of successive events—comprised either single instruments or blended combinations of instruments from the same or different families. Musicians and nonmusicians rated the degree of segregation perceived in the excerpts. Heterogeneous instrument combinations between streams yielded greater segregation than did homogeneous ones. Experiment 2 presented the individual streams from each two-stream excerpt. Blend ratings on isolated individual streams from the two-stream excerpts did not predict global segregation between streams. In Experiment 3, Experiment 1 excerpts were reorchestrated with only string instruments to determine the relative contribution of timbre to segregation beyond other musical cues. Decreasing timbral differences reduced segregation ratings. Acoustic and score-based descriptors were extracted from the recordings and scores, respectively, to statistically quantify the factors involved in these effects. Instrument family, part crossing, consonance, spectral factors related to timbre, and onset synchrony all played a role, providing evidence of how timbral differences enhance segregation in orchestral music.

We combine perceptual research and acoustic analysis to probe the messy, pluralistic world of musical semantics, focusing on sound mass music. Composers and scholars describe sound mass with many semantic associations. We designed an experiment to evaluate to what extent these associations are experienced by other listeners. Thirty-eight participants heard 40 excerpts of sound mass music and related contemporary genres and rated them along batteries of semantic scales. Participants also described their rating strategies for some categories. A combination of qualitative stimulus analyses, Cronbach’s alpha tests, and principal component analyses suggest that cross-domain mappings between semantic categories and musical properties are statistically coherent between participants, implying non-arbitrary relations. Some aspects of participants’ descriptions of their rating strategies appear to be reflected in their numerical ratings. We sought quantitative bases for these associations in the acoustic signals. After attempts to correlate semantic ratings with classical audio descriptors failed, we pursued a neuromimetic representation called spectrotemporal modulations (STMs), which explains much more of the variance in semantic ratings. This result suggests that semantic interpretations of music may involve qualities or attributes that are objectively present in the music, since computer simulation can use sound signals to partially reconstruct human semantic ratings.

BackgroundMany scientific fields now use machine-learning tools to assist with complex classification tasks. In neuroscience, automatic classifiers may be useful to diagnose medical images, monitor electrophysiological signals, or decode perceptual and cognitive states from neural signals. However, such tools often remain black-boxes: they lack interpretability. A lack of interpretability has obvious ethical implications for clinical applications, but it also limits the usefulness of these tools to formulate new theoretical hypotheses.New methodWe propose a simple and versatile method to help characterize the information used by a classifier to perform its task. Specifically, noisy versions of training samples or, when the training set is unavailable, custom-generated noisy samples, are fed to the classifier. Multiplicative noise, so-called “bubbles”, or additive noise are applied to the input representation. Reverse correlation techniques are then adapted to extract either the discriminative information, defined as the parts of the input dataset that have the most weight in the classification decision, and represented information, which correspond to the input features most representative of each category.ResultsThe method is illustrated for the classification of written numbers by a convolutional deep neural network; for the classification of speech versus music by a support vector machine; and for the classification of sleep stages from neurophysiological recordings by a random forest classifier. In all cases, the features extracted are readily interpretable.Comparison with Existing MethodsQuantitative comparisons show that the present method can match state-of-the art interpretation methods for convolutional neural networks. Moreover, our method uses an intuitive and well-established framework in neuroscience, reverse correlation. It is also generic: it can be applied to any kind of classifier and any kind of input data.ConclusionsWe suggest that the method could provide an intuitive and versatile interface between neuroscientists and machine-learning tools.HighlightsThe heuristics of black-box classifiers can be probed with noisy inputsThe relevant features can be visualised in the input representation spaceThe method applies to any kind of data such as 2D images or 1D time seriesIt applies to any classifier such as deep neural networks, support vector machines, random forests

International audience; Humans excel at using sounds to make judgements about their immediate environment. In particular, timbre is an auditory attribute that conveys crucial information about the identity of a sound source, especially for music. While timbre has been primarily considered to occupy a multidimensional space, unravelling the acoustic correlates of timbre remains a challenge. Here we re-analyse 17 datasets from published studies between 1977 and 2016 and observe that original results are only partially replicable. We use a data-driven computational account to reveal the acoustic correlates of timbre. Human dissimilarity ratings are simulated with metrics learned on acoustic spectrotemporal modulation models inspired by cortical processing. We observe that timbre has both generic and experiment-specific acoustic correlates. These findings provide a broad overview of former studies on musical timbre and identify its relevant acoustic substrates according to biologically inspired models.

Introduction Lack of sleep drastically affects many aspects of human behavior. The early detection of sleepiness is thus a major challenge for health and security reasons. Here we investigated the effect of sleep deprivation on the acoustic properties of human speech. Methods Twenty-four participants were sleep deprived for two days (two successive nights with only 3 hours of sleep). They were recorded reading a short text aloud before and after sleep deprivation. An auditory model, based on spectro-temporal modulations, was used to analyse the acoustic properties of their speech and served as a front-end to machine-learning classifiers. Results Results showed that sleepiness could be accurately detected with individually-trained classifiers. However,we were not able to fit a generic classifier for all participants. As we relied on an auditory-inspired model,we could identify and interpret the acoustic features impacted by sleep deprivation. Again,no simple diagnostic feature could be easily identified in the group- level analyses of the speech signals. We therefore developed a novel probing method, combining signal detection theory and noise activation of the classifier, to understand what made the classifier successful for each participant. This led to a diagnostic map for each participant, specifying which frequency region and modulation rates were impacted by sleep deprivation for this particular individual Conclusion In addition to suggesting a practical machine learning algorithm to detect sleep deprivation, combining our probing method with considerations about voice production could help uncover the physiological impact of sleep deprivation at the level of each individual. Support

International audience; Natural soundscapes correspond to the acoustical patterns produced by biological and geophysical sound sources at different spatial and temporal scales for a given habitat. This pilot study aims to characterize the temporal-modulation information available to humans when perceiving variations in soundscapes within and across natural habitats. This is addressed by processing soundscapes from a previous study [Krause, Gage, and Joo. (2011). Landscape Ecol. 26, 1247] via models of human auditory processing extracting modulation at the output of cochlear filters. The soundscapes represent combinations of elevation, animal, and vegetation diversity in four habitats of the biosphere reserve in the Sequoia National Park (Sierra Nevada, USA). Bayesian statistical analysis and support vector machine classifiers indicate that: (i) amplitude-modulation (AM) and frequency-modulation (FM) spectra distinguish the soundscapes associated with each habitat; and (ii) for each habitat, diurnal and seasonal variations are associated with salient changes in AM and FM cues at rates between about 1 and 100 Hz in the low (1-3 kHz) audio-frequency range. Support vector machine classifications further indicate that soundscape variations can be classified accurately based on these perceptually inspired representations.

Modulation Power Spectra include dimensions of spectral and temporal modulation that contribute significantly to the perception of musical instrument timbres. Nevertheless, it remains unknown whether each instrument's identity is characterized by specific regions in this representation. A recognition task was applied to tuba, trombone, cello, saxophone, and clarinet sounds resynthesized with filtered spectrotemporal modulations. The most relevant parts of this representation for instrument identification were determined for each instrument. In addition, instruments that were confused with each other led to non-overlapping spectrotemporal modulation regions, suggesting that musical instrument timbres are characterized by specific spectrotemporal modulations.

Musical instrument timbre has been intensively investigated through dissimilarity rating tasks. It is now well known that audio descriptors such as attack time and spectral centroid, among others, account well for the dimensions of the timbre spaces underlying these dissimilarity ratings. Nevertheless, it remains very difficult to reproduce these perceptual judgments from distances computed on acoustical representations such as the waveform or the spectrogram. Interestingly, biologically inspired representations based on spectrotemporal modulation spectra such as spectrotemporal receptive fields (STRF) have been shown to be well-suited to reproduce human dissimilarity ratings (Patil et al., 2012). Here, we propose a meta-analysis of seven former studies on timbre spaces in light of these recently developed representations. We implemented a computational framework that optimizes the correlation between the perceptual results and distances obtained from a set of different acoustic representations, in particular through the STRF. We observed that distances computed from spectrotemporal modulation representations provide the best correlation with the perceptual results across the seven timbre spaces. Finally, we highlighted the parts of the representations contributing the most to the correlation suggesting new insights into the underlying perceptual metrics. [Supported by Canada Research Chair, NSERC (RGPIN-2015-05208, RGPAS-478121-15), (RGPIN-262808-2012), and EU MSCf (Project MIM, H2020-MSCA-IF-2014, GA no. 659).]Musical instrument timbre has been intensively investigated through dissimilarity rating tasks. It is now well known that audio descriptors such as attack time and spectral centroid, among others, account well for the dimensions of the timbre spaces underlying these dissimilarity ratings. Nevertheless, it remains very difficult to reproduce these perceptual judgments from distances computed on acoustical representations such as the waveform or the spectrogram. Interestingly, biologically inspired representations based on spectrotemporal modulation spectra such as spectrotemporal receptive fields (STRF) have been shown to be well-suited to reproduce human dissimilarity ratings (Patil et al., 2012). Here, we propose a meta-analysis of seven former studies on timbre spaces in light of these recently developed representations. We implemented a computational framework that optimizes the correlation between the perceptual results and distances obtained from a set of different acoustic representations, in partic...

In this article, we propose a control strategy for synthesized continuous-interaction sounds. The framework of our research is based on the action–object paradigm that describes the sound as the result of an action on an object and that presumes the existence of sound invariants (i.e., perceptually relevant signal morphologies that carry information about the action's or the object's attributes). Auditory cues are investigated here for the evocations of rubbing, scratching, and rolling interactions. A generic sound-synthesis model that simulates these interactions is detailed. We then suggest an intuitive control strategy that enables users to navigate continuously from one interaction to another in an “action space,” thereby offering the possibility to simulate morphed interactions—for instance, ones that morph between rubbing and rolling.

The ability of a listener to recognize sound sources, and in particular musical instruments from the sounds they produce, raises the question of determining the acoustical information used to achieve such a task. It is now well known that the shapes of the temporal and spectral envelopes are crucial to the recognition of a musical instrument. More recently, Modulation Power Spectra (MPS) have been shown to be a representation that potentially explains the perception of musical instrument sounds. Nevertheless, the question of which specific regions of this representation characterize a musical instrument is still open. An identification task was applied to two subsets of musical instruments: tuba, trombone, cello, saxophone, and clarinet on the one hand, and marimba, vibraphone, guitar, harp, and viola pizzicato on the other. The sounds were processed with filtered spectrotemporal modulations with 2D Gaussian windows. The most relevant regions of this representation for instrument identification were determined for each instrument and reveal the regions essential for their identification. The method used here is based on a “molecular approach”, the so-called bubbles method. Globally, the instruments were correctly identified and the lower values of spectrotemporal modulations are the most important regions of the MPS for recognizing instruments. Interestingly, instruments that were confused with each other led to non-overlapping regions and were confused when they were filtered in the most salient region of the other instrument. These results suggest that musical instrument timbres are characterized by specific spectrotemporal modulations, information which could contribute to music information retrieval tasks such as automatic source recognition.

International audience; In this paper, a flexible control strategy for a synthesis model dedicated to nonlinear friction phenomena is proposed. This model enables to synthesize different types of sound sources, such as creaky doors, singing glasses, squeaking wet plates or bowed strings. Based on the perceptual stance that a sound is perceived as the result of an action on an object we propose a genuine source/filter synthesis approach that enables to elude physical constraints induced by the coupling between the interacting objects. This approach makes it possible to independently control and freely combine the action and the object. Different implementations and applications related to computer animation, gesture learning for rehabilitation and expert gestures are presented at the end of this paper.

International audience; The perception and production of biological movements is characterized by the 1/3 power law, a relation linking the curvature and the velocity of an intended action. In particular, motions are perceived and reproduced distorted when their kinematics deviate from this biological law. Whereas most studies dealing with this perceptual-motor relation focused on visual or kinaesthetic modalities in a unimodal context, in this paper we show that auditory dynamics strikingly biases visuomotor processes. Biologically consistent or inconsistent circular visual motions were used in combination with circular or elliptical auditory motions. Auditory motions were synthesized friction sounds mimicking those produced by the friction of the pen on a paper when someone is drawing. Sounds were presented diotically and the auditory motion velocity was evoked through the friction sound timbre variations without any spatial cues. Remarkably, when subjects were asked to reproduce circular visual motion while listening to sounds that evoked elliptical kinematics without seeing their hand, they drew elliptical shapes. Moreover, distortion induced by inconsistent elliptical kinematics in both visual and auditory modalities added up linearly. These results bring to light the substantial role of auditory dynamics in the visuo-motor coupling in a multisensory context.

International audience; Many studies stressed that the human movement execution but also the perception of motion are constrained by specific kinematics. For instance, it has been shown that the visuo-manual tracking of a spotlight was optimal when the spotlight motion complies with biological rules such as the so-called 1/3 power law, establishing the co-variation between the velocity and the trajectory curvature of the movement. The visual or kinesthetic perception of a geometry induced by motion has also been shown to be constrained by such biological rules. In the present study, we investigated whether the geometry induced by the visuo-motor coupling of biological movements was also constrained by the 1/3 power law under visual open loop control, i.e. without visual feedback of arm displacement. We showed that when someone was asked to synchronize a drawing movement with a visual spotlight following a circular shape, the geometry of the reproduced shape was fooled by visual kinematics that did not respect the 1/3 power law. In particular, elliptical shapes were reproduced when the circle is trailed with a kinematics corresponding to an ellipse. Moreover, the distortions observed here were larger than in the perceptual tasks stressing the role of motor attractors in such a visuo-motor coupling. Finally, by investigating the direct influence of visual kinematics on the motor reproduction, our result conciliates previous knowledge on sensorimotor coupling of biological motions with external stimuli and gives evidence to the amodal encoding of biological motion.

International audience; The present study investigated the effect of handwriting sonification on graphomotor learning. Thirty-two adults, distributed in two groups, learned four new characters with their non-dominant hand. The experimental design included a pre-test, a training session, and two post-tests, one just after the training sessions and another 24h later. Two characters were learned with and two without real-time auditory feedback (FB). The first group first learned the two non-sonified characters and then the two sonified characters whereas the reverse order was adopted for the second group. Results revealed that auditory FB improved the speed and fluency of handwriting movements but reduced, in the short-term only, the spatial accuracy of the trace. Transforming kinematic variables into sounds allows the writer to perceive his/her movement in addition to the written trace and this might facilitate handwriting learning. However, there were no differential effects of auditory FB, neither long-term nor short-term for the subjects who first learned the characters with auditory FB. We hypothesize that the positive effect on the handwriting kinematics was transferred to characters learned without FB. This transfer effect of the auditory FB is discussed in light of the Theory of Event Coding.

International audience; This study investigates the human ability to perceive biological movements through friction sounds produced by drawings and, furthermore, the ability to recover drawn shapes from the friction sounds generated. In a first experiment, friction sounds, real-time synthesized and modulated by the velocity profile of the drawing gesture, revealed that subjects associated a biological movement to those sounds whose timbre variations were generated by velocity profiles following the 1/3 power law. This finding demonstrates that sounds can adequately inform about human movements if their acoustic characteristics are in accordance with the kinematic rule governing actual movements. Further investigations of our ability to recognize drawn shapes were carried out in two association tasks in which both recorded and synthesized sounds had to be associated to both distinct and similar visual shapes. Results revealed that, for both synthesized and recorded sounds, subjects made correct associations for distinct shapes, while some confusion was observed for similar shapes. The comparisons made between recorded and synthesized sounds lead to conclude that the timbre variations induced by the velocity profile enabled the shape recognition. The results are discussed in the context of the ecological and ideomotor frameworks.

International audience; This paper presents an experiment dealing with the sensorimotor relation between auditory perception and graphical movements. Subjects were asked to synchronize their gestures with synthetic friction sounds. Some geometrical and dynamical parameters of the motor productions are analyzed according to the different mappings. This experiment provides a formal framework for a wider study which aims to evaluate the relation between audition, vision and gestures.

The dynamic features of sounds make them particularly appropriate for assessing the spatiotemporal characteristics of movements. Furthermore, sounds can inform about the correctness of an ongoing movement without directly interfering with the visual and proprioceptive feedback. Finally, because of their playful characteristics, sounds are potentially effective for motivating writers in particular need of any writing assistance. By associating relevant sounds to the specific variables of handwriting movement, the present chapter aimed at reporting how supplementary auditory information allows an examiner (teacher or therapist) to assess the movement quality from his/her hearing. Furthermore, a writer could also improve his/her movement from this real-time auditory feedback. Sonification of some movement characteristics would be a relevant tool for the diagnosis and the rehabilitation of some developmental disabilities (e.g. dysgraphia) or acquired disorders (e.g. Parkinson’s disease).

International audience; Even though we generally don't pay attention to the friction sounds produced when we are writing or drawing, these sounds are recordable, and can even evoke the underlying gesture. In this paper, auditory perception of such sounds, and the internal representations they evoke when we listen to them, is considered from the sensorimotor learning point of view. The use of synthesis processes of friction sounds makes it possible to investigate the perceptual influence of each gestures parameter separately. Here, the influence of the velocity profile on the mental representation of the gesture induced by a friction sound was investigated through 3 experiments. The results reveal the perceptual relevance of this parameter, and particularly a specific morphology corresponding to biological movements, the so-called 1/3-power law. The experiments are discussed according to the sensorimotor theory and the invariant taxonomy of the ecological approach.

Si l’on excepte le leger crissement du stylo qui n’informe que tres marginalement sur la qualite du mouvement, l’ecriture est une activite silencieuse. La transformer en une activite audible peut paraitre curieux au premier abord. En y reflechissant mieux, justement parce que la modalite auditive est disponible pendant l’ecriture et parce qu’elle est particulierement pertinente pour percevoir des differences fines au plan temporel et dynamique, utiliser des sons pour renseigner sur le mouvement d’ecriture semble judicieux.Sachant que le mouvement d’ecriture d’un dysgraphique est moins « lisse », moins « melodieux » en termes biomecaniques que celui d’un normoscripteur, l’objectif de cet article est de montrer pourquoi et comment transformer des indices cinematiques en informations auditives permettrait d’ameliorer le diagnostic et la remediation de la dysgraphie.