Search

• A novel approach for single channel speech enhancement is presented, using the colored spectrograms. • Using deep neural network for spectrogram denoising and regression neural network for translating the colors of denoised spectrograms to short time Fouier transform (STFT) magnitudes. • Output is comparable to the state-of-the-art baseline models at a much reduced computational cost. Speech enhancement concerns the processes required to remove unwanted background sounds from the target speech to improve its quality and intelligibility. In this paper, a novel approach for single-channel speech enhancement is presented using colored spectrograms. We propose the use of a deep neural network (DNN) architecture adapted from the pix2pix generative adversarial network (GAN) and train it over colored spectrograms of speech to denoise them. After denoising, the colors of spectrograms are translated to magnitudes of short-time Fourier transform (STFT) using a shallow regression neural network. These estimated STFT magnitudes are later combined with the noisy phases to obtain an enhanced speech. The results show an improvement of almost 0.84 points in the perceptual evaluation of speech quality (PESQ) and 1 % in the short-term objective intelligibility (STOI) over the unprocessed noisy data. The gain in quality and intelligibility over the unprocessed signal is almost equal to the gain achieved by the baseline methods used for comparison with the proposed model, but at a much reduced computational cost. The proposed solution offers a comparative PESQ score at almost 10 times reduced computational cost than a similar baseline model that has generated the highest PESQ score trained on grayscaled spectrograms, while it provides only a 1 % deficit in STOI at 28 times reduced computational cost when compared to another baseline system based on convolutional neural network-GAN (CNN-GAN) that produces the most intelligible speech. [ABSTRACT FROM AUTHOR]

Conventional automatic speaker verification systems can usually be decomposed into a front-end model such as time delay neural network (TDNN) for extracting speaker embeddings and a back-end model such as statistics-based probabilistic linear discriminant analysis (PLDA) or neural network-based neural PLDA (NPLDA) for similarity scoring. However, the sequential optimization of the front-end and back-end models may lead to a local minimum, which theoretically prevents the whole system from achieving the best optimization. Although some methods have been proposed for jointly optimizing the two models, such as the generalized end-to-end (GE2E) model and NPLDA E2E model, most of these methods have not fully investigated how to model the intra-relationship between multiple enrollment utterances. In this paper, we propose a new E2E joint method for speaker verification especially designed for the practical scenario of multiple enrollment utterances. To leverage the intra-relationship among multiple enrollment utterances, our model comes equipped with frame-level and utterance-level attention mechanisms. Additionally, focal loss is utilized to balance the importance of positive and negative samples within a mini-batch and focus on the difficult samples during the training process. We also utilize several data augmentation techniques, including conventional noise augmentation using MUSAN and RIRs datasets and a unique speaker embedding-level mixup strategy for better optimization. • Extend attention back-end to end-to-end for multi-enrollment speaker verification. • Introduce focal loss, suggest speaker embedding mixup for fine-tuning end-to-end model. • Analyze the effect of the number of enrollment utterances. • Analyze the embedding-level attention weights for speech genres. [ABSTRACT FROM AUTHOR]

In many speaker verification tasks, the quality of speaker embedding is an important factor in affecting speaker verification systems. Advanced speaker embedding extraction networks aim to capture richer speaker features through the multi-branch network architecture. Recently, speaker verification systems based on transformer encoders have received much attention, and many satisfactory results have been achieved because transformer encoders can efficiently extract the global features of the speaker (e.g., MFA-Conformer). However, the large number of model parameters and computational latency are common problems faced by the above approaches, which make them difficult to apply to resource-constrained edge terminals. To address this issue, this paper proposes an effective, lightweight transformer model (MCA-former) with multi-scale convolutional self-attention (MCA), which can perform multi-scale modeling and channel modeling in the temporal direction of the input with low computational cost. In addition, in the inference phase of the model, we further develop a systematic re-parameterization method to convert the multi-branch network structure into the single-path topology, effectively improving the inference speed. We investigate the performance of the MCA-former for speaker verification under the VoxCeleb1 test set. The results show that the MCA-based transformer model is more advantageous in terms of the number of parameters and inference efficiency. By applying the re-parameterization, the inference speed of the model is increased by about 30%, and the memory consumption is significantly improved. • Designing a lightweight multi-scale convolutional self-attention module. • An efficient transformer encoder for speaker verification. • Using the re-parameterization method improves the model's inference efficiency. [ABSTRACT FROM AUTHOR]

Speech difficulties are common in children and teenagers, but they can also occur in adults as a result of physical problems. A speech disorder is a situation in which an individual struggles to produce or construct the spoken sounds necessary for interpersonal communication. As a result, it could be challenging to comprehend the person's speech. Articulation abnormalities are typical speech problems. In this situation, automatic speech recognition (ASR) technology may be used to detect and further rectify such deficiencies. The first attempts to detect speech abnormalities were made in the early 1970s, and they appear to have followed the same path as those on the ASR. These early experiments did rely heavily on signal processing techniques. As time goes on, more ideas from ASR technology are being incorporated into systems that deal with speech impairments. Many traditional techniques are executed in the ASR system. In this paper, we developed an automatic speech recognition technology based on deep learning techniques. In this paper, we research alternative extraction and classification methods of electroencephalography (EEG) to help diagnose speech disorders (SD). The EEG data is prepared before degradation into numerous EEG sub-strands with a discrete wavelet transformation to eliminate unimportant errors. For sharpening signals, the Eigenvector crack curvature wavelet method was used. A hyper-similarity abnormality coder is used for feature extraction in the EEG recording and to detect synchronization between EEG channels, which may show abnormalities in communication. The recovered functions are then categorized using the Deep Residual–encoder–based VGG net CNN Classification Method. Thus, the techniques proposed to produce the most promising outcome aren't the suggested technique attained better classification accuracy when compared to the traditional methodologies. [ABSTRACT FROM AUTHOR]

Physical exertion is a stress condition that affects how we normally produce speech. It alters both the temporal and spectral pattern of speech characteristics. Therefore, speech utterances can be used as a cost-effective telehealth solution to detect whether a person is under physical exertion. This paper deals with the detection of shortness of breath (or out-of-breath) condition from speech under physical load by using an multi-task learning (MTL) framework. The primary classification targets are the neutral and the out-of-breath classes. Naturally these are binary targets and do not reflect the actual extent of exertion. This leads to the creation of a novel auxiliary target learning in collaboration with a pre-trained expert system. The targets here indicate the level of exertion under physical load. The MTL framework for both the convolutional neural network (CNN) and CNN-long short-term memory (CLSTM) network performs nearly 1.5% (F1-score) better than the single task learning (STL) framework. In addition to that, out-of-breath speech has more influence on the lower frequency spectrum. Warped spectrograms are given as input to the networks, enabling the deep networks to focus on lower spectral regions. The non-linear frequency warping is achieved by Mel-scale transformation and constant-Q-transform (CQT). CQT, being less dependent on fixed-window size, shows at least 6.57% (F1-score) improvement over Mel-based inputs. The MTL framework, combined with the warped spectrum, performs better in classifying out-of-breath speech from neutral. • Aims to detect speech-based out-of-breath (or physical load) conditions. • Efficiently determines the extent of physical exertion using speech utterances. • Lower frequency-centric approach for better detection of out-of-breath condition. • Multi-task learning further improves the performance by combing the above two steps. [ABSTRACT FROM AUTHOR]

• A novel voice conversion technique for body-conducted unvoiced speech is proposed based on Taylor-Boltzmann Machines. The proposed algorithm was the integration of Taylor Series with Boltzmann machine Deep Neural Networks. • A body conductive microphone referred as the non audible murmur (NAM) microphone is used to capture the non-audible murmur from the neck part behind the ear. • The classifier was trained with the features likes "pitch chroma, spectral centroid, and spectral skewness" that are derived from the incoming body conducted low voice verbal communication signal. • In the initial training phase, a Taylor series approximation DNN model is trained from a collection of stereo data which encompasses the NAM speech of same phonemes represented as log-power spectral features. • Moreover, in the recognition phase, a well trained Taylor-DNN model that is based on Boltzmann Machine intakes the speech features and generates the "log-power-spectra features" and it aids in recognizing the speech waveform. • The paper may gain high interest from the readers and experts of Speech processing, Voice Conversion Techniques, body-conducted unvoiced speech. Communication becomes effectual while the verbal communication signal reaches the destination with profound characteristics. An application based on human voice attracts the researcher's attention. It leads the researcher to proceed towards the speech conversion for low voice humans having murmured voices as their attitude. The researchers introduced a technique for improving the body-conducted unvoiced speech for silent speech communication using Taylor-Boltzmann machine-trained optimized Deep Neural Network (DNN). The projected model includes two main stages: the training stage as well as recognition stage. In the training stage, the following functionalities are undergone: (i) pre-processing, (ii) feature extraction, and (iii) deep learning-based speech enhancement. Initially, the collected input Non-Audible Murmur (NAM) speech signal is subjected to the pre-processing stage, wherein the improved wiener filtering is employed for suppressing the noise in the input signal. From the pre-processed speech signal, the most relevant characteristics such as "spectral skewness, spectral chroma, spectral centroid and improved Mel Frequency Cepstral Coefficients (MFCC)" are extracted, and they are fused together. Then, using these fused features, the Taylor series with optimized DNN and Boltzmann Machine (Taylor-DNN-BM) model is trained. The final enhanced speech is acquired from the Taylor- optimized DNN-BM. The recognition phase encapsulates the feature extraction and deep learning-based speech enhancement stage. The input speech data enters into the feature extraction phase, wherein the characteristics such as "spectral skewness, spectral chroma, spectral centroid and improved (MFCC)" are retrieved. After that, these features are fused and fed as input to the Taylor-optimized DNN-BM model. From Taylor- optimized DNN-BM, the enhanced speech signal is acquired. In DNN, the weight function is fine-tuned using the newly projected Self Improved Flower Pollination Algorithm (SI-FPA) algorithm. This SI-FPA model is the conceptual improvement of the Flower Pollination Algorithm (FPA). Finally, the projected model is compared with the existing models to validate its efficiency in terms of speech enhancement. [ABSTRACT FROM AUTHOR]

This paper discusses how automatic speech recognition systems are and could be designed, in order to best exploit the discriminative information encoded in human speech. This contrasts with many recent machine learning approaches that apply general recognition architectures to signals to identify, with little concern for the nature of the input. The implicit assumption has often been that training can automatically discover the useful properties that exist in signals, with minimal manual intervention. These approaches may be suitable for some tasks such as image recognition, where the diversity of visual input is vast; e.g., an image may be any (natural or synthetic) scene that a camera views. We first examine what makes speech special, i.e., a natural signal from a complex tube, driven by a source that is quasi-periodic and/or noisy, aiming to communicate a wide variety of information, using the different vocal systems of human speakers. Then, we view how pertinent features are extracted from speech via efficient means, related to the objectives of communication. We see how to reliably and efficiently identify the different units of oral language. We learn from the history of attempts to do ASR, e.g., why they succeeded and how improved methods exploited the increasing availability of data and computer power (in particular, deep neural networks). Finally, we suggest ways to render ASR both more accurate and efficient. This work is aimed at both newcomers to ASR and experts, in terms of presenting issues broadly, but without mathematical or algorithmic details, which are readily found in the references. [ABSTRACT FROM AUTHOR]

Sexism, a form of oppression based on one's sex, manifests itself in numerous ways and causes enormous suffering. In view of the growing number of experiences of sexism reported online, automatically classifying these recollections can aid in the battle against sexism by allowing gender studies researchers and government officials involved in policymaking to conduct more effective analyses. This paper investigates the 23-class fine-grained, multi-label classification of accounts (reports) of sexism. Moreover, we propose a knowledge-based cascaded multi-task framework for fine-grained multi-label sexism classification. We leverage several supporting tasks, including homogeneous and heterogeneous auxiliary tasks. Homogeneous tasks are set up without incurring any manual labeling cost and heterogeneous tasks are set up which have a high correlation with the accounts of sexism. Unlabeled accounts of sexism are utilized through unsupervised learning to help construct our multi-task setup. In addition, we incorporate a knowledge module within the framework to infuse external knowledge features into the learning process. Further, we investigate transfer learning that employs weakly labeled accounts of sexism and transfers the learning to the multi-label sexism classification. We also devise objective functions that exploit label correlations in the training data explicitly. Multiple proposed multi-task methods outperform the state-of-the-art multi-label sexism classification across five standard metrics. • Propose a knowledge-based cascaded multi-task neural framework for the multi-label sexism classification. • Explore two types of auxiliary tasks: (a) Homogeneous tasks and (b) Heterogeneous tasks • Trained the model with weakly labeled data and transfered the knowledge to the sexism classification task. • Proposed loss functions aimed at tapping label correlations in the multi-label data explicitly. [ABSTRACT FROM AUTHOR]

• In this paper, an optimal approach for text feature Selection , we work on text categorization and propose a statistical-based feature selection method (MFX) that considers all documents from the same category as one extended document, and chooses the most discriminative terms that are frequent and common across all documents of the same category, but rarely present in other categories. MFX is language independent and backed up with a mathematical formulation that finds the optimal number of features that guarantees accurate text categorization. Experimental results show the superiority of MFX over the state-of-the-art existing techniques. This work is very significant and timely given its applicability in applications such as spam filtering, opinion mining and topic spotting, among others. Traditionally, feature selection is conducted by first deriving a candidate list of features, then ranking and selecting the top features based on predefined threshold. These methods are highly dependent on the choice of the threshold, and therefore lead to sub-optimal text categorization results. In this paper, we address the selection problem by suggesting a one-step method designed to optimally select the subset of features. The selection is formulated mathematically as an optimization problem with the objective of maximizing classification accuracy while simultaneously deriving and choosing the most discriminative features. Our method, MFX, is applicable to many of the conventional methods, with two distinguishing aspects. First, it is based on considering all documents from the same category as one extended document, instead of analyzing individual documents. Second, it considers choosing the most discriminative terms that are frequent and common across all documents of the same category, and minimally present in other categories. Moreover, MFX is language-independent. It was tested on the well-known benchmark Reuters RCV1 dataset. To showcase its language independence, MFX was also tested on Arabic datasets extracted from Arabic news sources. The results indicated that MFX always performed similar to or better than other well-known feature selection methods. MFX with a Support Vector Machine (SVM) classifier was also shown to outperform recent text classification algorithms based on neural networks and word embeddings. [ABSTRACT FROM AUTHOR]

In this paper, we address the problem of speaker verification in conditions unseen or unknown during development. A standard method for speaker verification consists of extracting speaker embeddings with a deep neural network and processing them through a backend composed of probabilistic linear discriminant analysis (PLDA) and global logistic regression score calibration. This method is known to result in systems that work poorly on conditions different from those used to train the calibration model. We propose to modify the standard backend, introducing an adaptive calibrator that uses duration and other automatically extracted side-information to adapt to the conditions of the inputs. The backend is trained discriminatively to optimize binary cross-entropy. When trained on a number of diverse datasets that are labeled only with respect to speaker, the proposed backend consistently and, in some cases, dramatically improves calibration, compared to the standard PLDA approach, on a number of held-out datasets, some of which are markedly different from the training data. Discrimination performance is also consistently improved. We show that joint training of the PLDA and the adaptive calibrator is essential — the same benefits cannot be achieved when freezing PLDA and fine-tuning the calibrator. To our knowledge, the results in this paper are the first evidence in the literature that it is possible to develop a speaker verification system with robust out-of-the-box performance on a large variety of conditions. • A standard backend for speaker verification is based on PLDA followed by global calibration. • We propose a novel backend that consists of a PLDA-like formulation followed by condition-aware calibration. • The backend is trained discriminatively to optimize binary cross-entropy on a large variety of conditions. • We show that this backend results in significant gains over the standard PLDA approach. • The proposed approach results in a system that can be used out-of-the-box on data from different conditions. [ABSTRACT FROM AUTHOR]

This paper investigates the problem of measuring speech quality of received speech signal without employing the original speech signal. The problem of deterioration of the speech quality arises due to noise present in the surroundings. To this line, we propose a multiple time-instances (MTI) features-based approach for reference-free speech quality measurement model. A voice activity detector (VAD) is exploited primarily for calculating the number of active speech chunks of a speech signal. For these chunks and their successive combinations called here batches, multi-resolution auditory model (MRAM), mel-frequency cepstral coefficients (MFCC) and line spectral frequencies (LSF) features are extracted and called as MTI features. It is hypothesized that the MTI features are capable in capturing the distortions caused by time-localized effects of short-time transients, impulsive noise, and its differences from the plosive sounds. The MTI metric estimates (MTI-ME) are calculated corresponding to these MTI features employing the Gaussian mixture model (GMM) probabilistic technique. The overall objective speech quality of a speech signal is then determined as a linear combination of optimally weighted MTI-ME corresponding to distinct active speech chunks and their successive combinations, that is, batches of that speech signal. Minimum mean square error criterion or Pearson's correlation maximization criterion is employed for computing optimal weights. In addition, a deep neural network (DNN)-based speech quality model is also developed for calculating a single objective speech quality while considering all active speech chunks together. Pearson's correlation coefficient and weighted average correlation are exploited for evaluating the performance. Results demonstrate that the proposed model achieves promising improvement over the standard speech quality model (P.563) and improves correlation values by around 37%. • Addressing speech quality measuring problem by developing a reference-free speech quality model. • Developing a feature extraction technique incorporating distinct auditory features. • Developing a joint GMM training algorithm for computing objective speech quality. • Developing a deep neural network (DNN) framework for validating the proposed algorithm. • Demonstrating better performance as compared to the standard speech quality model (P.563). [ABSTRACT FROM AUTHOR]

As the volume of recorded conversations continues to surge, so does the need for their automatic processing. Plenty of information beyond words may be extracted from the speech signal that could be valuable in domains such as call-center quality assurance. In particular, describing the dynamics of turn-taking exchanges allows for a deeper understanding of the development and outcome of a dialogue. In this paper, we investigate the construction of an automatic turn-taking annotation tool based on recordings of entire conversations (in offline mode) — an unexplored topic to our knowledge. We experiment with two supervised learning approaches, using recurrent neural networks and random forests, on a corpus of Argentine Spanish task-oriented dialogues annotated with 12 turn-taking categories following standard guidelines. Our models achieve promising results, with F1 scores ranging 0.7–0.9 for the most frequent labels (e.g., smooth switches, backchannels), but much lower for the least frequent ones (various kinds of interruptions), for which further research is needed. We also evaluate our best-performing models considering their generalizability in scenarios of growing difficulty, including dialogues in two different languages (English and Slovak). Finally, to address the typical data scarcity issue, we analyze the impact of combining training data from different corpora, again including cross-linguistic data. • Machine learning (ML) models for offline annotation of turn-taking exchanges in entire dialogues. • Full taxonomy of turn-taking transitions, including smooth switches, interruptions, and backchannels. • Comparison of two ML approaches: random forests, and recurrent neural model (with a novel loss function strategy). • Generalizability assessment of our best models in scenarios of growing difficulty in Spanish. • Cross-linguistic evaluation in English and Slovak dialogues, as a way of dealing with labeled data scarcity. [ABSTRACT FROM AUTHOR]

The aim of this paper is to investigate the benefit of combining both language and acoustic modelling for speaker diarization. Although conventional systems only use acoustic features, in some scenarios linguistic data contain high discriminative speaker information, even more reliable than the acoustic ones. In this study we analyze how an appropriate fusion of both kind of features is able to obtain good results in these cases. The proposed system is based on an iterative algorithm where a LSTM network is used as a speaker classifier. The network is fed with character-level word embeddings and a GMM based acoustic score created with the output labels from previous iterations. The presented algorithm has been evaluated in a Call-Center database, which is composed of telephone interview audios. The combination of acoustic features and linguistic content shows a 84.29% improvement in terms of a word-level DER as compared to a HMM/VB baseline system. The results of this study confirms that linguistic content can be efficiently used for some speaker recognition tasks. • Linguistic content can be efficiently combined with acoustic features for the speaker diarization task. • Given a specific scenario, speaker diarization can be solved with only linguistic content. • Acoustic features lead to a better diarizatcon performance in comparison with linguistic content in large speaker segments. • Linguistic content is more discriminative than acoustic features to identify speakers in short speech segments. [ABSTRACT FROM AUTHOR]

Bidirectional long short-term memory (Bi-LSTM), as one of the effective networks for sequence labeling tasks, is widely used in named entity recognition (NER). However, the sequential nature of Bi-LSTM and the inability to recognize multiple sentences at the same time make it impossible to obtain overall information. In this paper, to make up for the shortcomings of Bi-LSTM in extracting global information, we propose a hierarchical context model embedded with sentence-level and document-level feature extraction. In sentence-level feature extraction, we use the self-attention mechanism to extract sentence-level representations considering the different contribution of each word to the sentence. For document-level feature extraction, 3D convolutional neural network (CNN), which not only can extract features within sentences, but also pays attention to the sequential relationship between sentences, is used to extract document-level representations. Furthermore, we investigate a layer-by-layer residual (LBL Residual) structure to optimize each Bi-LSTM block of our model, which can solve the degradation problem that appears as the number of model layers increases. Experiments show that our model achieves results competitive with the state-of-the-art records on the CONLL-2003 and Ontonotes5.0 English datasets respectively. [ABSTRACT FROM AUTHOR]

The deep learning methodologies in state-of-the-art speaker recognition systems are predominantly limited to the extraction of recording level embeddings. This is usually followed by generative modeling of the embeddings to output the verification score. In this paper, we explore a fully neural approach where the neural model outputs the verification score directly, given the acoustic feature inputs. This model, termed as Siamese neural network (SiamNN), combines the embedding extraction and back-end modeling into a single processing pipeline. The back-end modeling is achieved using a neural approach to PLDA modeling, called neural probabilistic linear discriminant analysis (NPLDA). In the NPLDA model, the verification score is computed as a discriminative similarity function. The development of the single neural SiamNN model allows the joint optimization of all the modules using a verification cost. Several speaker recognition experiments are performed using SITW, VOiCES, and NIST SRE datasets where the proposed SiamNN model is shown to significantly improve over the state-of-art x-vector PLDA baseline system (relative improvements of up to 35% in the primary cost metric). We also provide a detailed analysis of the influence of hyper-parameters, choice of loss functions, and data sampling strategies for training the model. In particular, we highlight that the proposed soft detection cost function based optimization improves over other loss functions considered. [ABSTRACT FROM AUTHOR]

The direction-of-arrival (DOA) of sound sources is an essential acoustic parameter used, e.g., for multi-channel speech enhancement or source tracking. Complex acoustic scenarios consisting of sources-of-interest, interfering sources, reverberation, and noise make the estimation of the DOAs corresponding to the sources-of-interest a challenging task. Recently proposed attention mechanisms allow DOA estimators to focus on the sources-of-interest and disregard interference and noise, i.e., they are signal-aware. The attention is typically obtained by a deep neural network (DNN) from a short-time Fourier transform (STFT) based representation of a single microphone signal. Subsequently, attention has been applied as binary or ratio weighting to STFT-based microphone signal representations to reduce the impact of frequency bins dominated by noise, interference, or reverberation. The impact of attention on DOA estimators and different training strategies for attention and DOA DNNs are not yet studied in depth. In this paper, we evaluate systems consisting of different DNNs and signal processing-based methods for DOA estimation when attention is applied. Additionally, we propose training strategies for attention-based DOA estimation optimized via a DOA objective, i.e., end-to-end. The evaluation of the proposed and the baseline systems is performed using data generated with simulated and measured room impulse responses of a uniform-linear microphone array under various acoustic conditions, like reverberation times, noise, and source array distances. The data contains a single source-of-interest, noise, and directional interference. The best-performing systems are also evaluated using measured data. Our experiments show that DNNs used for DOA estimation are biased to the spectral source characteristics and the spectral attention distribution used during training (e.g., spectrally flat/sparse). We also show that this bias in the DOA estimator can be avoided if signal-processing methods are used in combination with attention. Overall, DOA estimation using attention in combination with signal-processing methods exhibits a far lower computational complexity than a fully DNN-based system; however, it yields comparable results. • Attention enables source-selective direction of arrival estimation. • Combining data-driven and signal-processing methods reduces complexity. • Attention can be estimated using masking concepts from source separation. [ABSTRACT FROM AUTHOR]

Multi-source localization is an important and challenging technique for multi-talker conversation analysis. This paper proposes a novel supervised learning method using deep neural networks to estimate the direction of arrival (DOA) of all the speakers simultaneously from the audio mixture. At the heart of the proposal is a source splitting mechanism that creates source-specific intermediate representations inside the network. This allows our model to give source-specific posteriors as the output unlike the traditional multi-label classification approach. Existing deep learning methods perform a frame level prediction, whereas our approach performs an utterance level prediction by incorporating temporal selection and averaging inside the network to avoid post-processing. We also experiment with various loss functions and show that a variant of earth mover distance (EMD) is very effective in classifying DOA at a very high resolution by modeling inter-class relationships. In addition to using the prediction error as a metric for evaluating our localization model, we also establish its potency as a frontend with automatic speech recognition (ASR) as the downstream task. We convert the estimated DOAs into a feature suitable for ASR and pass it as an additional input feature to a strong multi-channel and multi-talker speech recognition baseline. This added input feature drastically improves the ASR performance and gives a word error rate (WER) of 6.3% on the evaluation data of our simulated noisy two speaker mixtures, while the baseline which does not use explicit localization input has a WER of 11.5%. We also perform ASR evaluation on real recordings with the overlapped set of the MC-WSJ-AV corpus in addition to simulated mixtures. • An end-to-end multi-source direction of arrival (DOA) estimation method is proposed. • The method tackles multi-source DOA estimation via single-source DOA estimation. • The DOA classification network is optimized to learn inter-class relationships for a fine-grained angle resolution. • DOA estimation is used as a frontend for multi-talker automatic speech recognition. [ABSTRACT FROM AUTHOR]

• A state-of-art Hindi NER system based on MuRIL language model and CRF. • Computation of token vector through various combination of encoder layers of MuRIL. • Classification of entities using variants of MuRIL and mBERT based models. • Evaluation of models on highly diverse set of Hindi named entities. Named Entity Recognition (NER) plays an important role in various Natural Language Processing (NLP) applications to extract the key information from a huge amount of unstructured text data. NER is a task of identifying and classifying the named entities into predefined categories for a given text. Recently, language models are highly appreciable in several NLP tasks as these state-of-the-art models result better even in resource scarcity. In this paper, we perform NER task on the Hindi language by incorporating the recently released multilingual language model MuRIL which stands for Multilingual Representation for Indian Languages. MuRIL is specially trained for 16 Indian languages. We develop a Hindi NER system using MuRIL with a conditional random field (CRF) layer and fine-tune the model on the ICON 2013 Hindi NER dataset. Further, in the proposed approach, we compute the addition of the last 4 layers representations of the MuRIL model instead of just using the last layer's representation and fine-tune the whole model. Several variants of this model are presented by applying different computations on token representations provided by different layers of 12-layered MuRIL architecture. The proposed model achieves state-of-the-art results as 87.89% precision, 83.74% recall and 85.77% F1-score and outperforms all other existing Hindi NER systems developed on the ICON 2013 dataset. Additionally, we develop a similar Hindi NER system by replacing the MuRIL language model with another state-of-the-art language model, called multilingual Bidirectional Encoder Representations from Transformers (mBERT) to analyze the efficiency of both language models over the Hindi NER task. [ABSTRACT FROM AUTHOR]

This paper investigates the feasibility of the task of automatic behaviour coding of spoken interactions in teamwork settings. We introduce the coding schema used to classify the behaviours of the group members and the corpus we collected to assess the coding schema reliability in real teamwork meetings. The behaviours embedded in spoken utterances are modelled using a discriminative approach based on conditional random fields, and state-of-the-art neural networks based models. Moreover, we fine-tune publicly available language models to fit our target domain and task and demonstrate how this type of knowledge transfer improves classification models' generalisation capacity. To utilise public resources, the AMI corpus was used for deploying the proposed framework. However, the models were evaluated on both AMI (matched task) and recordings of students solving an engineering challenge (mismatched task). Evaluation results reveal that neural networks are the best performing models in matched tasks, but that C R F models outperform them in mismatched tasks. Mitigating the effect of noisy data, by implementing a lightly supervised approach leads to relative improvements of 32% and 22%, in F 1 measures of C R F and B E R T , respectively. The proposed classifiers are used as a part of technological support to the training programme in collaborative skills for undergraduate students. • The viability of automatic verbal behaviour coding using speech transcriptions is investigated. • A new corpus with annotations following a precise metadiscourse coding scheme is introduced to the field of automatic behaviour analysis in teamwork meetings. The applied coding scheme is based on a well-studied psychological theory that is designed to wisely observe to provide valuable feedback. • A lightly supervised approach that produces a better version of manually coded training data is also proposed. Lexical and contextual text features are used to train sequential and neural network automatic metadiscourse tagging models. • The performance of these models is compared under different hierarchical levels of behaviour analysis. • To evaluate the proposed approach, AMI (a matched set) and a real application set taken from the ENG Challenge data (unmatched set), test sets are used. [ABSTRACT FROM AUTHOR]

This paper presents a speech intelligibility model based on automatic speech recognition (ASR), combining phoneme probabilities from deep neural networks (DNN) and a performance measure that estimates the word error rate from these probabilities. This model does not require the clean speech reference nor the word labels during testing as the ASR decoding step – which finds the most likely sequence of words given phoneme posterior probabilities – is omitted. The model is evaluated via the root-mean-squared error between the predicted and observed speech reception thresholds from eight normal-hearing listeners. The recognition task consists of identifying noisy words from a German matrix sentence test. The speech material was mixed with eight noise maskers covering different modulation types, from speech-shaped stationary noise to a single-talker masker. The prediction performance is compared to five established models and an ASR-model using word labels. Two combinations of features and networks were tested. Both include temporal information either at the feature level (amplitude modulation filterbanks and a feed-forward network) or captured by the architecture (mel-spectrograms and a time-delay deep neural network, TDNN). The TDNN model is on par with the DNN while reducing the number of parameters by a factor of 37; this optimization allows parallel streams on dedicated hearing aid hardware as a forward-pass can be computed within the 10 ms of each frame. The proposed model performs almost as well as the label-based model and produces more accurate predictions than the baseline models. [Display omitted] • New speech intelligibility model based on ASR probabilities and their estimated WER. • The model does not require clean speech nor the word labels during testing. • The model computes more accurate frame forward predictions within 10 ms. [ABSTRACT FROM AUTHOR]

According to the World Health Organization, the number of people suffering from dementia worldwide will grow to 150 million by mid-century, and Alzheimer's disease is the most common form of dementia contributing to 60%–70% of cases. The problem is compounded by the fact that current pharmacologic treatments are only symptomatic, and therapies are ineffective in slow down or cure the degenerative process. An automatic and standardize classifier for Alzheimer's disease is thereby extremely important to rapidly respond and deliver as preventive as possible interventions. Speech alterations might be one of the earliest signs of cognitive defect and, recently, the researchers showed that they can be observable well in advance other cognitive deficits become manifest. In this paper, we propose a full automated method able to classify the spontaneous spoken production of the subjects. In particular, we trained an artificial neural network using the spectrogram of the audio signal, which is the visual representation of the speech of the subject. Moreover, to overcome the problem of the large amount of annotated data usually required for training deep learning models, we used a specific data augmentation approach that avoids distorting the original samples. We evaluated the proposed method using the English Pitt Corpus from DementiaBank. The used dataset consists of 180 subjects: 43 healthy controls and 137 Alzheimer's disease patients. The proposed method outperformed the other approaches in the literature based on manual and semi-automatic transcription and annotation of speech, improving the classification capability by 5.93%, and obtained good classification results compared to the state-of-the-art neuropsychological screening tests (i.e., the Mini-Mental State Examination and the Activities of Daily Living portion of the Blessed Dementia Rating Scale) exhibiting an accuracy of 93.30% and an F1 score of 88.50%. [ABSTRACT FROM AUTHOR]

Cross-language article linking (CLAL), the task of generating links between articles in different languages from different encyclopedias, is critical for facilitating sharing among online knowledge bases. Some previous CLAL research has been done on creating links among Wikipedia wikis, but much of this work depends heavily on simple language patterns and encyclopedia format or metadata. In this paper, we propose a new CLAL method based on deep learning paragraph embeddings to link English Wikipedia articles with articles in Baidu Baike, the most popular online encyclopedia in mainland China. To measure article similarity for link prediction, we employ several neural networks with attention mechanisms, such as CNN and LSTM, to train paragraph encoders that create vector representations of the articles' semantics based only on article text, rather than link structure, as input data. Using our "Deep CLAL" method, we compile a data set consisting of Baidu Baike entries and corresponding English Wikipedia entries. Our approach does not rely on linguistic or structural features and can be easily applied to other language pairs by using pre-trained word embeddings, regardless of whether the two languages are on the same encyclopedia platform. • Cross-language article linking helps create a multilingual unified knowledge base. • Using attention-based neural network that learns to attend to the vital part of articles. • The novel method that does not rely on feature engineering and is scalable to large data. [ABSTRACT FROM AUTHOR]

In the scope of voice biometrics, the term replay attack , (RA) refers to the dishonest attempt made by an impostor to spoof someone else's identity by replaying the subject's previously recorded speech close to the Automatic Speaker Verification (ASV) system under attack. State-of-the-art strategies for RA detection, such as the Enhanced Teager Energy Cepstral Coefficients (ETECC), have shown promising results due to their precision in measuring energy from high frequency components of speech, as a function of two recently defined concepts: signal mass and Enhanced Teager Energy Operator (ETEO). Nevertheless, since the replay mechanism prominently deteriorates the speech signal spectrum just in those spectral zones, we propose the association of ETEO with different strategies to further improve the previous results in getting effective countermeasures against RAs. Specifically, comprehensive evaluations which include a detailed mathematical analysis, a simulation on amplitude and frequency modulated (AM–FM) signals, and a spectrographic inspection involving different filterbank structures, along with their experimental results, are provided in this paper. In addition, ETEO-derived features are contrasted to existing feature sets by using Paraconsistent Feature Engineering (PFE) for feature ranking, expanding our previously published results. Lastly, experiments are performed with ASVSpoof-2017 version 2.0 dataset, Realistic Replay Attack Microphone Array Speech Corpus (ReMASC), BTAS-2016, dataset, ASVSpoof-2019 challenge dataset, and ASVSpoof-2015 challenge dataset, considering Gaussian Mixture Models (GMMs), Convolutional Neural Networks (CNNs) and Light-CNN architectures as being the classifiers. The standalone ETECC-GMM system showed the best performance by producing equal error rates (EERs) of 5.55% and 10.75% on development and evaluation sets, respectively. • Improved handcrafted feature extraction technique based on ETEO operator is presented, which provides a precise estimate of signal energy by means of the concept of signal mass, to leverage the performance of TEO. • We describe an innovative assessment, based on Paraconsistent Feature Engineering (PFE), which measures the efficacy of the proposed ETECC-based feature sets, along with the existing feature sets, for the intended classification task. • We comment on the results from the application of state-of-the-art filterbanks, namely Gammatone and Cochlear, in SSD tasks. The experiments were also performed with the Ricker wavelet-based filterbank, i.e., the negative normalized second derivative of the Gaussian, and Gabor filterbank. • We performed experiments on environment-dependent scenario on ASVspoof-2017 version 2.0 dataset, Realistic Replay Attack Microphone Array Speech Corpus (ReMASC), BTAS-2016, ASVspoof-2019 challenge dataset, and ASVspoof-2015 challenge dataset. • Extended the experiments by considering deep learning architectures, namely, Convolutional Neural Networks (CNNs) and Light-CNN architectures, to work as classifiers in conjunction with the extracted features, and reported the results over ASVspoof-2017 version-2 dataset. [ABSTRACT FROM AUTHOR]

Recent advances in automatic speech recognition (ASR) have achieved accuracy levels comparable to human transcribers, which led researchers to debate if the machine has reached human performance. Previous work focused on the English language and modular hidden Markov model-deep neural network (HMM–DNN) systems. In this paper, we perform a comprehensive benchmarking for end-to-end transformer ASR, modular HMM–DNN ASR, and human speech recognition (HSR) on the Arabic language and its dialects. For the HSR, we evaluate linguist performance and lay-native speaker performance on a new dataset collected as a part of this study. For ASR the end-to-end work led to 12.5%, 27.5% , 33.8% WER; a new performance milestone for the MGB2, MGB3, and MGB5 challenges respectively. Our results suggest that human performance in the Arabic language is still considerably better than the machine with an absolute WER gap of 3.5% on average. • Comprehensive benchmarking for end-to-end, modular, and human on the Arabic language. • New milestone in Arabic speech recognition with E2E transformer-based ASR. • The machine arguably outperforms the performance of the native speaker. • Machine mistakes showed high similarity with the expert linguist transcription. • Expert linguist performance in Arabic language is still considerably better than the machine. [ABSTRACT FROM AUTHOR]

• Capsule network only structures can successfully map sequences to sequences. • Mappings are refined by initializing routing iteration based on the previous output. • Sequence-wise routing iteration allows for non-iterative inference. • Structures of capsule network are more important than the number of parameters. • Top layer capsules become similar to the capsule corresponding to a sequence label. Capsule networks (CapsNets) have recently gotten attention as a novel neural architecture. This paper presents the sequential routing framework which we believe is the first method to adapt a CapsNet-only structure to sequence-to-sequence recognition. Input sequences are capsulized then sliced by a window size. Each slice is classified to a label at the corresponding time through iterative routing mechanisms. Afterwards, losses are computed by connectionist temporal classification (CTC). During routing, the required number of parameters can be controlled by the window size regardless of the length of sequences by sharing learnable weights across the slices. We additionally propose a sequential dynamic routing algorithm to replace traditional dynamic routing. The proposed technique can minimize decoding speed degradation caused by the routing iterations since it can operate in a non-iterative manner without dropping accuracy. The method achieves a 1.1% lower word error rate at 16.9% on the Wall Street Journal corpus compared to bidirectional long short-term memory-based CTC networks. On the TIMIT corpus, it attains a 0.7% lower phone error rate at 17.5% compared to convolutional neural network-based CTC networks (Zhang et al., 2016). [ABSTRACT FROM AUTHOR]

While human listeners can, to some extent, understand the information conveyed by the speech signal when it is mixed with noise, traditional objective intelligibility measures usually fail to operate without a priori knowledge of the clean speech signal. This hence limits the usability of those measures in situations where the clean speech signal is inaccessible. In this paper a glimpse-based method is extended to make speech intelligibility predictions directly from speech-plus-noise mixtures. Using a neural network, the proposed method estimates the time-frequency regions with a local speech-to-noise ratio above a given threshold – known as glimpses – from the mixture signal, instead of separately comparing the speech signal against the noise signal. The number and locations of the glimpses can then be used to produce an intelligibility score. In Experiment I where listener intelligibilities were measured in one stationary and nine fluctuating noise maskers, the predictions produced by the proposed method were highly correlated with the subjective data, with correlation coefficients above 0.90. In Experiment II, with the same neural network trained on normal natural speech as in Experiment I, the proposed method was used to predict the intelligibility of speech signals modified by intelligibility-enhancement algorithms and synthetic speech. The method can still maintain its predictive power by demonstrating a similar performance to its intrusive counterpart with an overall correlation coefficient of 0.81, which is superior to many modern traditional measures evaluated under the same conditions. Therefore, the proposed method can be used to estimate speech intelligibility in place of traditional measures in conditions where their capacity falls short. [ABSTRACT FROM AUTHOR]