Search

We propose two end-to-end neural configurations for language diarization on bilingual code-switching speech. The first, a BLSTM-E2E architecture, includes a set of stacked bidirectional LSTMs to compute embeddings and incorporates the deep clustering loss to enforce grouping of languages belonging to the same class. The second, an XSA-E2E architecture, is based on an x-vector model followed by a self-attention encoder. The former encodes frame-level features into segmentlevel embeddings while the latter considers all those embeddings to generate a sequence of segment-level language labels. We evaluated the proposed methods on the dataset obtained from the shared task B in WSTCSMC 2020 and our handcrafted simulated data from the SEAME dataset. Experimental results show that our proposed XSA-E2E architecture achieved a relative improvement of 12.1% in equal error rate and a 7.4% relative improvement on accuracy compared with the baseline algorithm in the WSTCSMC 2020 dataset. Our proposed XSA-E2E architecture achieved an accuracy of 89.84% with a baseline of 85.60% on the simulated data derived from the SEAME dataset.

We address the problem of localizing and tracking alternating (moving or stationary) talkers using microphone arrays in a room environment. One of the main challenges is the frequent (and possibly abrupt) change of talker positions which requires the algorithm to capture the active talker rapidly. In addition, the presence of interference, background noise and room reverberation degrades the tracking performance. We propose a new algorithm that jointly exploits the advantages of the particle filter (PF) and particle swarm intelligence. The PF is used as a general tracking framework which incorporates a proposed alternating source-dynamic model for recursive estimation of talker position. Unlike the conventional PF where particles operate independently in the particle sampling stage, the use of swarm intelligence allows particles to interact with each other, thereby improving convergence toward the active talker location. In addition, the memory mechanism in swarm intelligence allows particles to remain at their previous best-fit state estimate when signals are corrupted by interference, noise and/or reverberation. Simulations and experiments were conducted to demonstrate the effectiveness of the proposed algorithm. Accepted version

Automatic speech emotion recognition is one of the challenging tasks in machine learning community mainly due to the significant variations across individuals while expressing the same emotion cue. The success of emotion recognition with machine learning techniques primarily depends on the feature set chosen to learn. Formulation of appropriate features that cater for all variations in emotion cues however is not a trivial task. Recent works on emotion recognition with deep learning techniques thus focus on the end-to-end learning scheme which identifies the features directly from the raw speech signal instead of relying on hand-crafted feature set. Existing methods in this scheme however did not take into account the fact that speech signals often exhibit distinct features at different time scales and frequencies than in the raw form. We propose the multi- scale convolution neural network (MCNN) to identify features at different time scales and frequencies from raw speech signals. This end-to-end model leverages on the multi-branch input layer and tunable convolution layers to learn the identified features which are subsequently employed to recognize the emotion cues accordingly. As a proof-of-concept, the MCNN method with a fixed transformation stage is evaluated using the SAVEE emotion database. Results showed that MCNN improves the emotion recognition performance when compared to existing methods, which underpins the necessity of learning features at different time scales. NRF (Natl Research Foundation, S’pore) Accepted version

We propose a TDOA-based algorithm for source localization on rigid surfaces. This allows the conversion of readily available large surfaces into touch interfaces using surface-mounted vibration sensors. To achieve this, we characterize the arrival of each sensor-received signal by the arrival times of its frequency components. To estimate the arrival time of each frequency component, we first model each component as a harmonic random process. A kurtosis sequence, which exhibits a sharp rising edge when the signal begins to deviate from the background noise, can then be obtained for each component. By accurately estimating the starting point of the rising edge, our algorithm can avoid the uncertainty due to gradual noise-to-signal transition. Experiment results show that the proposed algorithm achieves better localization performance than existing techniques on large surfaces.

Equalization of acoustic channels usually involves inversion of acoustic impulse responses (AIRs), and generally employs multichannel techniques. In this paper, we propose three equalization algorithms, one in the Karhunen-Loeve transform (KLT) domain and the other two in the frequency domain. Our proposed algorithm in the KLT domain provides a platform to achieve equalization in conjunction with denoising. Existing multiple-input/output inverse theorem (MINT)-based non-adaptive algorithms require the inversion of a matrix with dimension that is proportional to the AIR length, and is computationally expensive. To overcome this limitation, we propose the frequency-domain algorithm which is computationally very efficient and thus can be employed for the equalization of high-order AIRs in practical applications. In addition, the frequency-domain method is more robust to AIR estimation errors. To achieve further reduction in the complexity without significant performance degradation, we then propose a modified version of the frequency-domain algorithm.

Stereophonic acoustic echo cancellation remains one of the challenging areas for tele/video-conferencing applications. However, the existence of high interchannel coherence between the two input signals for such systems leads to considerable degradation in misalignment convergence of the adaptive filters. We propose a new algorithm for improving the convergence performance and steady-state misalignment by considering robustness to the source position in the transmission room. We achieve this by exploiting the inherent decorrelating properties of selective-tap adaptive filtering as well as employing a variable clipping threshold for the unselected taps. Simulation results using colored noise and speech signals show an improvement over existing algorithms both in terms of convergence rate as well as steady-state normalized misalignment.

We propose a new adaptive filtering algorithm for stereophonic acoustic echo cancellation. This algorithm uses a linear single-layer feedforward neural network to efficiently decorrelate the tap-input vectors. It achieves an improvement in the misalignment convergence by means of applying the resulted decorrelated tap-input vectors to the coefficient update of the adaptive filters. The advantage of our approach as compared with existing techniques is that our algorithm, in use with the nonlinear preprocessor, can achieve a high rate of misalignment convergence without significantly degrading the quality and stereophonic image of the transmitted signals since our neural network operates on the tap-input vectors as opposed to the transmitted audio signals. We then show that we can achieve an efficient implementation for the proposed decorrelation method by considering the structure of the joint-input covariance matrix of the stereophonic signals.

We propose a new approach to the development of a touch interface using surface-mounted sensors which allows one to convert a hard surface into a touch pad. This is achieved by using location template matching (LTM), a source localization algorithm that is robust to dispersion and multipath. In this interdisciplinary research, we employ mechanical vibration theories that model wave propagation of the flexural modes of vibration generated by an impact on the surface. We then verify that the amplitude variance across time for each propagating mode frequency is unique to each location on a surface. We show that the Zak transform allows us to faithfully track these amplitude variations and we exploit the uniqueness of this variance as a time-frequency classifier which in turn allows us to localize a finger tap in the context of a human-computer interface. The performance of the proposed algorithm is compared with existing LTM approaches on real surfaces.

The adaptive multiple input/output inverse theorem (A-MINT) algorithm has been applied to equalize acoustic channels. To further increase its convergence rate, we suppress any undesirable non-zero coefficients in the estimated Kronecker delta function at each iteration. We achieve this by introducing the concept of sparseness measure of the estimated Kronecker delta function and using this as an additional constraint to A-MINT. Simulation and experimental results illustrate that the proposed algorithm can achieve faster convergence than A-MINT.

We propose a new low complexity, low delay, and fast converging frequency-domain adaptive algorithm for network echo cancellation in VoIP exploiting MMax and sparse partial (SP) tap-selection criteria in the frequency domain. We incorporate these tap-selection techniques into the multidelay filtering (MDF) algorithm in order to mitigate the delay inherent in frequency-domain algorithms. We illustrate two such approaches and discuss their tradeoff between convergence performance and computational complexity. Simulation results show an improvement in convergence rate for the proposed algorithm over MDF and significantly reduced complexity. The proposed algorithm achieves a convergence performance close to that of the recently proposed, but substantially more complex improved proportionate MDF (IPMDF) algorithm.

Conventional adaptive beamformer that employs covariance matrix estimated from received signals is sensitive to array perturbation and signal direction-of-arrival mismatch. To ensure robust performance, only the interference-plus-noise components are used in the computation of covariance matrix. Any leakage of the desired signal will introduce distortion at the beamformer output and under-estimation of the interference and/or noise will lead to a reduction in interference and noise suppression. To identify interference-plus-noise components from the received microphone signals, we analyze the microphone outputs in the short-time Fourier transform domain where we exploit property of the Hermitian angle in complex vector space. This information is then used to determine the interference-plus-noise presence probability which is then incorporated to the updating process of covariance matrix estimation. We show that the proposed algorithm outperforms a recently proposed algorithm for both simulated and recorded data.

Due to practical considerations the microphone spacing is increased to achieve improved resolution by violating the spatial Nyquist criterion. Accompanied aliasing components adversely affect the identifiability of the source direction peaks. We investigate the effect of aliasing on the spatial spectrum of the steered minimum variance distortionless response (STMV) method and propose a novel multi-stage scheme assisted by subband decomposition for suppressing aliasing components. The performance of the proposed technique, evaluated with simulations and recorded room responses, reflects the improvement in the identifiability of accurate source directions under aliasing conditions.

This paper focuses on the problem of single-channel noise reduction in a transient noise environment for speech enhancement application. A typical speech enhancement algorithm requires an estimate of the noise statistics. However, the problem of noise estimation is challenging when the statistics of the noise vary significantly with time. By exploiting the fact that for speech signal most of the energy is concentrated on the harmonic bands in voiced frames, we propose an algorithm for the estimation of speech presence probability in the time-frequency domain. The estimated speech presence probability is then used for noise estimation for speech enhancement application. Evaluations are conducted to compare the speech enhancement performance between the proposed algorithm and the existing algorithm for various types of transient noise.

We address the problem of estimating direction-of-arrivals (DOAs) for multiple sound sources using a single acoustic vector sensor (AVS) in an enclosed room environment. It is well-known that multi-source DOA estimation in an enclosed environment is challenging due to room reverberation, environmental noise and overlapping of the source spectra. In this work, we propose a multi-source DOA estimation algorithm which exploits co-location of the sensor elements in AVS. We identify time-frequency (TF) zones of the received signals in which only one source is dominant with a high signal-to-reverberation ratio. DOA estimation is then achieved via the use of clustering of the Hermitian angle feature. Simulation results show that the proposed DOA estimation algorithm is robust to both reverberation and environmental noise.

The adaptive multiple-input/output inverse theorem (A-MINT) multichannel equalization algorithm was proposed to address the computational complexity of the MINT algorithm. However, similar to MINT, A-MINT also assumes an arbitrary modeling delay for the target response which, if inappropriately chosen, results in misconvergence or poor performance. We present a new adaptive algorithm that exploits the sparsity of a target response to estimate equalization filters. The proposed algorithm automatically selects a suitable modeling delay for the equalized response so as to achieve a minimum-norm solution.

The mixing matrix estimation in conventional underdetermined convolutive blind source separation (UCBSS) algorithms assume that the source signals are W-disjoint in the time-frequency (TF) domain. This assumption requires that each TF point of the received mixtures is a single-source point (SSP), which may not always be true. In this work, we propose a preprocessing technique to estimate the single-source confidence (SSC) of each TF point. Only those TF points with a high SSC value are then used by existing algorithms to obtain a more accurate estimate of the mixing matrix with reduced computational complexity. Simulation and experimental results show that the proposed preprocessing method can improve the performance of the existing UCBSS algorithms.

We present a user-friendly method that employs acoustic features to automatically classify songs. This is achieved by extracting low-level features and reducing the feature space using principle component analysis (PCA). The songs are then plotted on a song-space graphic user interface (GUI) for manual or automatic browsing. The similarity between songs is given by the Euclidean distance in this lower-dimension song space. Using this song space, a prototype application known as the “SmartDJ” has been implemented on the MAX/MSP platform. This prototype application enables users to visualize their music library, select songs based on their similarity or automate the song selection process using a given seed song. We also describe, in this paper, several features of the application including the smooth mix transition feature which provides an enhanced experience for the users to perform song transition seamlessly.

The performance of conventional acoustic source localization and tracking system reduces significantly when reverberation, noise, and acoustic interference are present. In this paper, a robust speaker tracking algorithm for an enclosed environment in the presence of interference and noise is proposed. We exploit the harmonic structure which is a distinctive feature in speech to enhance the robustness against acoustic interference. In order to extract the speech harmonic information, a beamformer is employed to enhance the signal from a prior estimated source location. A new particle weight update is then computed based on the steered response power function given the estimated speech harmonic information. Simulation results show that the proposed method achieves robustness in localization and tracking of a speech source in the presence of interference, noise and reverberation.

It is well-known that the performance of acoustic multichannel equalization (MCEQ) algorithms depend on the modeling delay that has to be pre-defined. In this work, we propose a MCEQ algorithm which achieves full equalization of acoustic room impulse responses in the presence of blind-system identification error. We achieve the above by modeling the inverse filters using multiple delays via columns of the identity matrix which serve as basis vectors for this time-domain sub-space approach. We further show that the proposed algorithm allows one to determine an optimal set of inverse filters corresponding to an appropriate modeling delay.

The multiple-input/output inverse theorem (MINT) algorithm for multichannel equalization is computationally demanding. Although adaptive MINT reduces the computational complexity, it suffers from slow convergence. In this letter, we propose a low-complexity fast-converging adaptive algorithm for multichannel equalization. The novelty of the approach lies in the adaptive equalization for each frequency bin and its ability to achieve fast convergence in a single step. The proposed algorithm can achieve better equalization of high-order acoustic impulse responses with significant reduction in complexity.

We propose a new source localization algorithm for the development of a touch interface using surface-mounted sensors. This algorithm allows one to convert a hard surface into a touch pad. The proposed algorithm is based on triangulation using phase transit time-differences-of-arrival (TDOA) between the received signals from the sensors. In this inter-disciplinary research, we analyze the wave propagation of flexural vibration due to an impact on a surface which in turn allows one to gain insights to the frequency spectrum diversity of vibration signals and background noise. We then show how the Kullback-Leibler discrimination information can be used to determine the phase transit time of the vibration signal. The performance of the proposed algorithm is compared with existing TDOA approaches on real surfaces.

An auto-relation aided multiple-input/output inverse filtering algorithm (A-RAM) is proposed for the inverse filtering of room acoustics in speech dereverberation. In A-RAM, we exploit the relationship between the received reverberant signals and the acoustic impulse responses as well as constraining the reconstructed signals of the two sub-systems. This results in an auto-relation which is then used as a constraint for the adaptive multiple input/output inversion theorem (A-MINT) algorithm. Simulation results, using both synthetic and recorded room impulse responses, show that the proposed A-RAM achieves a fast convergence compared to A-MINT.

Acoustic transmission for conferencing systems have progressed from the use of single channel to one that employs stereophonic channels. One of the most important challenges for such stereophonic system is the problem of stereophonic acoustic echo cancellation (SAEC) where a pair of echo cancellers are deployed to estimate the acoustic impulse responses of the receiving room. We propose, in this paper, a neural network based adaptive filtering approach for SAEC. The neural network is employed to decorrelate the input vectors for efficient filter updating, resulting in a high convergence rate of the adaptive filters for this multi-channel acoustic application. To further enhance the efficiency of the proposed algorithm, we then utilize the joint-input correlation matrix of the stereophonic signals so as to simplify the proposed neural network. Simulation results show the improvement in performance of the proposed adaptive SAEC approach over the state-of-the-art algorithms.

The use of partial-updating algorithm for reducing interchannel coherence in stereophonic acoustic echo cancellation has been proposed recently. In this work, we show that this algorithm suffers from the lack of robustness against source positions in the transmission room. To address this, we present an insight into this problem and propose a center clipping algorithm that improves the joint optimization between reducing interchannel coherence and increasing energies of selected taps for different source positions. Simulation results using both colored and speech inputs verify the robustness of the proposed algorithm against source positions.

In the context of acoustic echo cancellation (AEC), it is shown that the level of sparseness in acoustic impulse responses can vary greatly in a mobile environment. When the response is strongly sparse, convergence of conventional approaches is poor. Drawing on techniques originally developed for network echo cancellation (NEC), we propose a class of AEC algorithms that can not only work well in both sparse and dispersive circumstances, but also adapt dynamically to the level of sparseness using a new sparseness-controlled approach. Simulation results, using white Gaussian noise (WGN) and speech input signals, show improved performance over existing methods. The proposed algorithms achieve these improvement with only a modest increase in computational complexity.

The common zeros problem for blind system identification (BSI) is well known. It degrades the performance of classic BSI algorithms and therefore imposes the limit on the performance of subsequent speech dereverberation. The effect of near-common zeros has recently been studied in terms of channel diversity and the degradation in performance of BSI and multichannel equalization algorithms has been shown. We now introduce a novel approach to improve channel diversity which we refer to as Forced Spectral Diversity (FSD). The FSD concept uses a combination of spectral shaping filters and effective channel undermodelling. Simulation results show that the proposed approach achieves improved performance with reduced complexity for multichannel BSI in a room acoustics example.

The common zeros problem for Blind System Identification (BSI) has been well known to degrade the performance of classic BSI algorithms and therefore limits performance of subsequent speech dereverberation. Recently, we have shown that multichannel systems cannot be well identified if near-common zeros are present. In this work, we further study the near-common zeros problem using channel diversity measure. We then investigate the use of forced spectral diversity (FSD) based on a combination of spectral shaping filters and effective channel under modelling. Simulation results show the effectiveness of the proposed approach.

Source localization employing time-differences-of-arrival has been employed for many applications. The accuracy of source localization is limited by the errors in the time differences of arrival estimation as well as microphone position calibration errors. Because a microphone position error will affect multiple time differences of arrival, correlation between these quantities will be introduced. This work presents a new mathematical framework in which we quantify the localization performance of a microphone array in which the microphone positions are subject to such errors.

The performance in terms of misalignment of adaptive algorithms, in general, is dependent on the conditioning of the input signal covariance matrix. The performance of two-channel adaptive algorithms is further degraded by the high interchannel coherence between the two input signals. In this letter, we establish the relationship between interchannel coherence of the two input signals and condition of the corresponding covariance matrix for stereo acoustic echo cancellation application. We show how this relationship affects the misalignment of a frequency-domain adaptive algorithm. We provide simulation results for both white Gaussian noise and speech input to verify our mathematical analysis.