Search

Virtual reality (VR) has recently emerged as a platform that can be employed in the context of e-health applications. Even though the majority of VR-based applications focus on visual stimuli as the main content, audio also plays a very important role. If someone has an issue with auditory processing, the comprehension of auditory information is compromised. This condition reflects negatively on one’s quality of life, given the impact of audio perception on one’s ability to communicate effectively or to differentiate and ignore “noise”. This work aims to design a VR application that can be used to: (a) optimize multimodal VR experiences based on user trials (b) extract user data continuously throughout the experiment (c) evaluate a user’s auditory processing ability. To accomplish this goal, participants are required to localise a sound source in space in the presence of multiple listening conditions with simple and complex sound stimuli configurations. Data collected from users consists of physiological and objective metrics. The results of this study highlight the relationship between user behaviour (head movement, fixation points) and performance in the sound localisation task. This information can be used to design future applications with the purpose of training one’s auditory localisation ability. In addition, the evaluation compared the impact of using two interaction methods to perform this task: using a pointer or eye gaze to indicate the location of the target source. The findings from this study show statistically significant differences in terms of physiological response when subjects are exposed to different interaction methods, with greater immersion and performance for the pointer group.

Binaural hearing is the ability of the human auditory system to integrate information received from both ears simultaneously. Binaural hearing is fundamental in understanding speech in noisy backgrounds. Any disfunction in one or both ears could cause a disruption in the processing mechanism. Auditory evoked potentials (AEPs) are electrical potentials evoked by externally presented auditory stimuli from any part of the auditory system. A non-invasive technology, electroencephalography (EEG) is used for the monitoring of AEPs. The research aims to identify the best suited electrode positions through correlation analysis and analyse the AEP signals from the selected electrodes in order to detect binaural sensitivity of the human brain. The study evaluates the time-averaged EEG responses of normal hearing subjects to auditory stimuli. The stimuli used for the study are 500 Hz Blackman windowed pure tones, presented in either homophasic (the same phase in both ears) or antiphasic (180-degree phase difference between the two ears) conditions. The study focuses on understanding the effect of phase reversal of auditory stimuli, an under interaural time difference (ITD) cue, on the middle latency response (MLR) region of the AEPs. A correlation analysis was carried out between the eight different locations and as a result, Cz and Pz electrode positions were selected as the best suited positions for further analysis. The selected electrode signals were further processed in the time domain and frequency domain analysis. In the time domain analysis, it was found that Cz electrode for eight subjects out of nine and Pz electrode for seven subjects out of nine, had the larger area under signal curve obtained in the antiphasic condition than in the homophasic signals. Frequency domain analysis showed that the frequency bands 20 to 25Hz and 25 to 30Hz had the most energy when elicited by antiphasic stimuli than by homophasic stimuli. The findings of this study can be further utilised for the detection of binaural processing in a human brain.

This paper presents a smartphone-based binaural hearing aid architecture for improving the speech intelligibility of hearing aid users. The proposed system consists of an earpiece, a smartphone and an application that performs real-time speech enhancement. The speaker’s voice, which is picked up by the microphone of the earpiece that is worn on the ear, is transmitted to the smartphone via wireless technology. After the speech intelligibility is improved in real time by the deep learning speech enhancement application, it is returned to the earpiece and generates sound. Deep learning speech enhancement algorithms can be used without performing burdensome calculations on the processors in the hearing aid. The results showed that the average usage of the central processing unit in the smartphone was approximately 26%, and the signal-to-noise ratios improve by at least 20%. The presented objective and subjective results show that the proposed method achieves comparatively more noise suppression without distorting the audio.

Virtual and augmented reality (VR/AR) systems dramatically gained in popularity with various application areas such as gaming, social media, and communication. It is therefore a crucial task to have the knowhow to efficiently utilize, store or deliver 360° videos for end-users. Towards this aim, researchers have been developing deep neural network models for 360° multimedia processing and computer vision fields. In this line of work, an important research direction is to build models that can learn and predict the observers' attention on 360° videos to obtain so-called saliency maps computationally. Although there are a few saliency models proposed for this purpose, these models generally consider only visual cues in video frames by neglecting audio cues from sound sources. In this study, an unsupervised frequency-based saliency model is presented for predicting the strength and location of saliency in spatial audio. The prediction of salient audio cues is then used as audio bias on the video saliency predictions of state-of-the-art models. Our experiments yield promising results and show that integrating the proposed spatial audio bias into the existing video saliency models consistently improves their performance., Scientific and Technological Research Council of Turkey (TÜBİTAK) 1001 Program Award; Turkish Academy of Sciences GEBIP 2018 Award; Science Academy BAGEP 2021 Award

One necessary condition for autonomous driving is an accurate and reliable representation of the environment around the vehicle. Current architectures rely on cameras, radars, and lidars to capture the visual environment and to localize and track other traffic participants. Human drivers can see but also hear and use a lot of auditory information for understanding the environment in addition to visual cues. In this paper, we present a pure sound localization and recognition system to extract an auditory representation of the environment. First, the environmental sound is classified into seven main categories of traffic objects followed by six specific kinds of sirens in an emergency case using a simple neural network layout. Second, each object is localized via a combined time-delay of arrival and amplitude-based localization algorithm. The system is evaluated on real-world data focusing on a robust detection and accurate localization of emergency vehicles.

A human is capable detecting sounds coming from all directions around him. The quest to develop cognition in a robot has opened many opportunities for researchers. In robots, sound source detection has been researched for decades and is successful in detecting in 2D as well as 3D. This paper focuses on an experimental setup that is developed to imbibe hearing cognitive in a robot. 3D sound source localization is achieved using only 2 microphones. Our experiment goes beyond the human cognition by utilizing animal cognition for sound localization that considers perceptual capabilities of humans. Barn owl ear alignment is being used as reference for robot ear alignment. The ear alignment in Barn owl is not symmetrical but one ear is at higher location than the other. To be more specific, the paper is related to directional (but not distance of sound source) sound source localization in a robot in three-dimensional space, using only two microphones; wherein the microphones are positioned asymmetrical in the robot head to form skewed alignment and one microphone is rotated to create the skew alignment at different angles between the microphones. The location of a sound source is determined by studying the TDOA computed from sound signal recorded at the different angles between the microphones.

This paper presents a novel method for the visualization of 3D spatial sounds in Virtual Reality (VR) for Deaf and Hard-of-Hearing (DHH) people. Our method enhances traditional VR devices with additional haptic and visual feedback, which aids spatial sound localization. The proposed system automatically analyses 3D sound from VR application, and it indicates the direction of sound sources to a user by two Vibro-motors and two Light-Emitting Diodes (LEDs). The benefit of automatic sound analysis is that our method can be used in any VR application without modifying the application itself. We evaluated the proposed method for 3D spatial sound visualization in a user study. Additionally, the conducted user study investigated which condition (corresponding to different senses) leads to faster performance in 3D sound localization task. For this purpose, we compared three conditions: haptic feedback only, LED feedback only, combined haptic and LED feedback. Our study results suggest that DHH participants could complete sound-related VR tasks significantly faster using LED and haptic+LED conditions in comparison to only haptic feedback. The presented method for spatial sound visualization can be directly used to enhance VR applications for use by DHH persons, and the results of our user study can serve as guidelines for the future design of accessible VR systems.

Sound Localization in robots especially in the vertical plane are either by using microphones array or using HRTF. While HRTF uses the characteristic of human ears that distort a sound frequency. We propose the multipath analysis to localize the sound source in a vertical plane. The multipath is the result of sound reflected in the pinna of the human ear before coming into the inner ear. The analysis of the sound multiple frequencies shown that the polynomial model of 1500 Hz has the best result in vertical plane sound localization. The angular error of this model is 8.407 and RMSE is 10.308. Similar results can also be seen from other frequency models that were built in this study. Those results showed that we can localize the sound in the vertical plane using multipath analysis that was recorded using only 1 microphone equipped with the artificial ear.

Intensity based soundfield reproduction methods are shown to provide impressive human perception of sound localization. However, most of the previous works in this domain either focus on a single sweet spot for the listener, or are constrained to a regular loudspeaker geometry, which is difficult to implement in real-world applications. This paper addresses both of the above challenges. We propose an intensity matching technique to optimally reproduce sound intensity at multiple sweet spots using an irregular loudspeaker array. The performance of the proposed method is evaluated by comparing it with the pressure and velocity matching method through numerical simulations and perceptual experiments. The results show that the proposed method has an improved performance.

Humans easily and routinely identify an image region that corresponds to an observed sound in their daily lives. The task is formulated as an unsupervised sound source localization without using tagged data. Recently, several methods have been proposed that utilize the activation of hidden or output layers of neural networks, such as an attention layer or feature maps in a convolutional neural network (CNN). We propose another strategy that obtains a localization map at the input side, applying the widely used input gradient method. It is computationally efficient and can be easily applied to any existing techniques because it is free from the network structure. Taking advantage of it, we propose a combination method with existing methods for higher sound localization performance. Experiments are performed using the Flickr-SoundNet data set. When a pre-trained image front-end was used, the proposed method gives better results than the attention-based method. For a completely unsupervised condition, the gradient method provides comparable performance as the conventional methods; the best results are obtained by this combination method.

Auditory display techniques require subjective evaluations of the virtual sound field to determine the accuracy of a reproduced sound with respect to a listener’s perception of spatial information. To synthesize a physically accurate sound field, local sound field synthesis limits the controlled area inside the loudspeaker array. The size and position of the controlled area affects the accuracy of the synthesis. In this study, we investigate the sound localization accuracies at different positions inside a 2Dimensional loudspeaker array by measurement of the interaural level differences in an anechoic chamber. The results show that the sound localization of the local sound field synthesis was more accurate than the conventional wave field synthesis, if a suitable parametrization of local listening area is chosen.

Localization of sound sources in 3D sound fields is an extremely challenging task, especially when the environments are reverberant and involve multiple sources. In this work, we propose a deep neural network to analyze audio signals recorded by 3D microphones and localize sound sources in a spatial sound field. In particular, we consider first-order Ambisonics microphones to capture 3D acoustic signals and represent them by spherical harmonic decomposition in the quaternion domain. Moreover, to improve the localization performance, we use quaternion input features derived from the acoustic intensity, which is strictly related to the direction of arrival (DOA) of a sound source. The proposed network architecture involves both quaternion-valued convolutional and recurrent layers. Results show that the proposed method is able to exploit both the quaternion-valued representation of ambisonic signals and to improve the localization performance with respect to existing methods.

Deep learning speech separation algorithms have achieved great success in improving the quality and intelligibility of separated speech from mixed audio. Most previous methods focused on generating a single-channel output for each of the target speakers, hence discarding the spatial cues needed for the localization of sound sources in space. However, preserving the spatial information is important in many applications that aim to accurately render the acoustic scene such as in hearing aids and augmented reality (AR). Here, we propose a speech separation algorithm that preserves the interaural cues of separated sound sources and can be implemented with low latency and high fidelity, therefore enabling a real-time modification of the acoustic scene. Based on the time-domain audio separation network (TasNet), a single-channel time-domain speech separation system that can be implemented in real-time, we propose a multi-input-multi-output (MIMO) end-to-end extension of TasNet that takes binaural mixed audio as input and simultaneously separates target speakers in both channels. Experimental results show that the proposed end-to-end MIMO system is able to significantly improve the separation performance and keep the perceived location of the modified sources intact in various acoustic scenes., To appear in ICASSP 2020

Automatic localization and classification of environmental sound events can provide great aid to many human-centric applications. However as many papers have mentioned, environmental sound events in daily life are complicated and hard to classify especially when multiple sounds happen simultaneously. Being different from many other works, we use an acoustic-sensor-network to solve this problem and decompose overlapping sound events using a sound localization model. The core of our contribution is to first find and locate the keypoints from each microphone's spectrogram and then aggregate them. With these aggregated keypoints as input, we then use 2 different classification models to further classify the type of sound sources. Compared with other classification models that only use single microphone, our experiments show that our solution is both accurate and low-cost in terms of calculation effort.

For every event occurring in the real world, most often a sound is associated with the corresponding visual scene. Humans possess an inherent ability to automatically map the audio content with visual scenes leading to an effortless and enhanced understanding of the underlying event. This triggers an interesting question: Can this natural correspondence between video and audio, which has been diminutively explored so far, be learned by a machine and modeled jointly to localize the sound source in a visual scene? In this paper, we propose a novel algorithm that addresses the problem of localizing sound source in unconstrained videos, which uses efficient fusion and attention mechanisms. Two novel blocks namely, Audio Visual Fusion Block (AVFB) and Segment-Wise Attention Block (SWAB) have been developed for this purpose. Quantitative and qualitative evaluations show that it is feasible to use the same algorithm with minor modifications to serve the purpose of sound localization using three different types of learning: supervised, weakly supervised and unsupervised. A novel Audio Visual Triplet Gram Matrix Loss (AVTGML) has been proposed as a loss function to learn the localization in an unsupervised way. Our empirical evaluations demonstrate a significant increase in performance over the existing state-of-the-art methods, serving as a testimony to the superiority of our proposed approach.

The spatial-musical association of response codes (SMARC) effect, in which a high-pitched (low-pitched) tone facilitates an upper (lower) response, is considered to reflect the spatial coding of sound pitch. However, previous studies have not excluded the directional effects of sound localization. Because a high-pitched (low-pitched) tone is automatically misperceived as originating from a spatially high (low) location, the location of a perceived sound source might artificially elicit the SMARC effect. This study challenged this issue, by demonstrating whether the SMARC effect occurs in response to sound imagery without sound input. To achieve this goal, participants were first trained to associate visual stimuli (novel contoured shapes) with sound pitches (high-pitched or low-pitched pure tones). Subsequently, they elicited the SMARC effect for trained visual stimuli (novel contoured shapes) associated with pitch height. Therefore, the SMARC effect was observed without sound localization but with the activation of sound imagery that was spatial.

In ecoacoustics, autonomous recording unit (ARU) is recognized as a powerful tool because it enables us to obtain long-term acoustic data. We believe that sound localization techniques using multiple microphone arrays further contribute to understanding of acoustic events or structure of soundscapes in fields. We aim to examine the applicability of HARKBird, an integrated framework to extract sound sources and their direction of arrivals (DOA) using robot audition techniques with microphone arrays, to the soundscape analysis of birdsongs. We consider novel uses of the information obtained from microphone arrays to characterize the dynamics of soundscape of birdsongs quantitatively in view of different spatial and temporal resolutions. Specifically, we conducted a field recording in California with a simple networked recording system. We propose the variance of DOA (VDOA) of songs as an ecoacoustic index. It could reflect the spatial property of soundscape, compared with the acoustic complexity index (ACI), and the combination of them could better describe the soundscape in more detail. Furthermore, we estimated the spatial position of birdsongs and successfully grasped the situation in which multiple species and individuals sang at several positions.

Recently, although the traditionally proposed Permutation Invariant Training (PIT) has attracted much attention, it performs poorly on datasets of unknown number of speakers. In this paper, we propose an approach based on beamforming and deep models (BDM) to solve the problem mentioned above. BDM firstly estimates the number of speakers by sound source localization algorithm and then enhances the target speech with beamforming in spatial domain. Subsequently the supervised deep models are used to extract the clean speech of the target speaker in time and frequency domain. Experimental results show that the BDM can improve the separation performance as well as speech intelligibility compared with the single channel and multi-channel PIT when tested on two-speaker, three-speaker and four-speaker mixtures, respectively.

Head direction has been identified to anticipate trajectory direction during human locomotion, independently of the visual condition. However, experiments so far have explored this phenomenon with visual instructions. To explore head and gaze anticipation without visual influence, we describe a system for instruction and measurement of auditory instructed locomotion. The objective of this work is to describe the system setup and validate its accuracy for scientific investigations in locomotion research. The system is comprised of an auralization server that plays virtual sound sources via wireless headphones. The auralization is fed with head position and orientation angle measured by a Vicon motion tracking system. Communication between the auralization server and the motion tracking host was facilitating the OSC protocol. Auditory instructed locomotion was performed in two visual condition: eyes open and eyes closed for comparison. First, ten sighted participants localized static virtual sound sources at eight positions to confirm the accuracy of the provided virtual sound cues. Afterwards, they listened and actively followed virtual sound sources that were moving along three trajectories: an eight shape, a circle in clockwise direction and a circle in anticlockwise direction. The virtual sound sources could be localized with an accuracy of 0.16 meters in the eyes open condition and 0.12 meters in the eyes closed condition. Participants also were able to follow the moving sound sources.

Polyphonic Sound Event Detection (SED) in real-world recordings is a challenging task because of the dynamic polyphony level, intensity, and duration of sound events. Current polyphonic SED systems fail to model the temporal structure of sound events explicitly and instead attempt to look at which sound events are present at each audio frame. Consequently, the event-wise detection performance is much lower than the segment-wise detection performance. In this work, we propose a joint model approach to improve the temporal localization of sound events using a multi-task learning setup. The first task predicts which sound events are present at each time frame; we call this branch 'Sound Event Detection (SED) model', while the second task predicts if a sound event is present or not at each frame; we call this branch 'Sound Activity Detection (SAD) model'. We verify the proposed joint model by comparing it with a separate implementation of both tasks aggregated together from individual task predictions. Our experiments on the URBAN-SED dataset show that the proposed joint model can alleviate False Positive (FP) and False Negative (FN) errors and improve both the segment-wise and the event-wise metrics., Accepted to WASPAA 2019

A recent approach to improving the robustness of sound localization in reverberant environments is based on pre-selection of time- frequency pixels that are dominated by direct sound. This approach is equivalent to applying a binary time-frequency mask prior to the localization stage. Although the binary mask approach was shown to be effective, it may not exploit the information available in the captured signal to its full extent. In an attempt to overcome this limitation, it is hereby proposed to employ a soft mask instead of the binary mask. The proposed weighting scheme is based directly on a metric of the direct-to-reverberant sound ratio in each individual time-frequency pixel. Evaluation using simulated reverberant speech recordings indicates substantial improvement in the localization performance when using the proposed soft mask weighting.

In this work, we explore the potential for morphological weighting of different regions of the pinna (outer ear) to improve the prediction of acoustic directivity patterns associated with head-related transfer functions. Using a large deformation diffeomorphic metric mapping framework, we apply kernel principal component analysis to model the pinna morphology. Different regions of the pinna can be weighted differently prior to the kernel principal component analysis. By varying the weights applied to the various regions of the pinna, we begin to learn the relative importance of the various regions to the acoustic directivity of the ear as a function of frequency. The pinna is divided into nine parts comprising the helix, scaphoid fossa, triangular fossa, concha rim, cymba concha, cavum concha, conchal ridge, ear lobe, and back of the ear. Results indicate that weighting the conchal region (concha rim, cavum and cymba concha) improves the predicted acoustic directivity for frequency bands centered around 3 kHz, 7 kHz, 10 kHz and 13 kHz. Similarly, weighting the triangular and scaphoid fossa improves the prediction of acoustic directivity in frequency bands centered around 7 kHz, 13 kHz and 15.5 kHz.

The aim of the work is to estimate the position of sound sources. The proposed method uses a setup of two acoustic vector sensors (AVS). The intersection of azimuth rays from each AVS should indicate the position of a source. In practice, the result of position estimation using this method is an area rather than a point. This is a result of inaccuracy of the individual sensors, but more importantly, of the influence of a source size. The proposed method was validated in experiments performed in an anechoic room, using a custom-made setup of two sensors built from digital MEMS microphones, for sound source placed at varying distance and angle from the sensors. The paper discusses the observed variations in the measured angle and distance to the source. The obtained results indicate that the proposed method allows for estimation of the source position with satisfactory accuracy. Errors in the position estimation depend on the source size, the distance to the source and the source angle relative to the sensors. Possible application of the proposed method is estimation of the position of moving sound sources, such as road vehicles.

This paper presents a recurrent neural network (RNN)-based speech enhancement method for a binaural hearing aid system. We design a binaural RNN architecture to train gains of subbands. The proposed method trains the binaural recurrent neural network from binaural signals separately before concatenating to a weighting fully connected layer. The method preserves more cues from binaural signals using different training strategies and weighting variations. The results show the proposed method achieves a balance between speech intelligence and noise suppression in different acoustics scene. We demonstrate that the proposed method can implement speech enhancement for hearing aids with a low complexity neural network. This makes it applicable on smartphone or embedded devices.

A binaural sound source localization method is proposed that uses interaural and spectral cues for localization of sound sources with any direction of arrival on the full-sphere. The method is designed to be robust to the presence of reverberation, additive noise and different types of sounds. The method uses the interaural phase difference (IPD) for lateral angle localization, then interaural and spectral cues for polar angle localization. The method applies different weighting to the interaural and spectral cues depending on the estimated lateral angle. In particular, only the spectral cues are used for sound sources near or on the median plane.

We propose a biologically inspired binaural sound localization system using a deep convolutional neural network (CNN) for reverberant environments. It utilizes a binaural Cascade of Asymmetric Resonators with Fast-Acting Compression (CAR-FAC) cochlear system to analyze binaural signals, a lateral inhibition function to sharpen temporal information of cochlear channels, and instantaneous correlation function on the two cochlear channels to encode binaural cues. The generated 2-D instantaneous correlation matrix (correlogram) encodes both interaural phase difference (IPD) cues and spectral information in a unified framework. Additionally, a sound onset detector is exploited to generate the correlograms only during sound onsets to remove interference from echoes. The onset correlograms are analyzed using a deep CNN for regression to the azimuthal angle of the sound. The proposed system was evaluated using experimental data in a reverberant environment, and displayed a root mean square localization error (RMSE) of 3.68° in the −90° to 90° range.