Search

• We propose novel training algorithms for vocoder-free text-to-speech synthesis using STFT spectra based on generative adversarial networks (GANs). • We demonstrate that using GANs with the original-frequency-resolution amplitude spectra degrades the synthetic speech quality. • We show that the proposed low-frequency-resolution GANs improves the synthetic speech quality. • We also show that using the inverse mel scale for the proposed algorithm further improves the synthetic speech quality. This paper proposes novel training algorithms for vocoder-free text-to-speech (TTS) synthesis based on generative adversarial networks (GANs) that compensate for short-term Fourier transform (STFT) amplitude spectra in low/multi frequency resolution. Vocoder-free TTS using STFT amplitude spectra can avoid degradation of synthetic speech quality caused by the vocoder-based parameterization used in conventional TTS. Our previous work for the vocoder-based TTS proposed a method for incorporating the GAN-based distribution compensation into acoustic model training to improve synthetic speech quality. This paper extends the algorithm to the vocoder-free TTS and propose a GAN-based training algorithm using low-frequency-resolution amplitude spectra to overcome the difficulty in modeling complicated distribution of the high-dimensional spectra. In the proposed algorithm, amplitude spectra are transformed into low-frequency-resolution amplitude spectra by applying an average pooling function along with a frequency axis; then the GAN-based distribution compensation is performed in the low-frequency-resolution domain. Because the low-frequency-resolution amplitude spectra approximately emulate filter banks, the proposed algorithm is expected to improve synthetic speech quality by reducing differences in spectral envelopes of natural and synthetic speech. Furthermore, various frequency scales that are related to human speech perception (e.g., mel and inverse mel frequency scales) can be introduced to the proposed training algorithm by applying an frequency warping function to amplitude spectra. This paper also proposes a GAN-based training algorithm using multi-frequency-resolution amplitude spectra that uses both low- and original-frequency-resolution amplitude spectra to reduce the differences in not only spectral envelopes but also fine structures. Experimental results demonstrate that (1) GANs using low-frequency-resolution amplitude spectra improve speech quality and work robustly against the settings of the frequency resolution and hyperparameters, (2) in comparison among low-, original-, and multi-frequency-resolution amplitude spectra, the use of low-frequency-resolution ones work best improve the synthetic speech quality, and (3) the use of the inverse mel frequency scale for obtaining low-frequency-resolution amplitude spectra further improves synthetic speech quality. [ABSTRACT FROM AUTHOR]

Highlights • DSTC6: Dialog Challenge to improve performance of end-to-end dialog systems using Neural Network models and dialog breakdown detection. • Track 1, End-to-End Goal Oriented Dialog Learning: selection of the best system response. Hybrid Code Network and Memory Network were the best models. • Track 2, End-to-End Conversation Modeling: system response generation. 78.5% of the automatically generated sentences were rated as acceptable responses by humans. • Track 3, Dialogue Breakdown Detection. The submitted systems performed as well as humans in detecting dialog breakdown, for both English and Japanese data-sets. This paper describes the experimental setups and the evaluation results of the sixth Dialog System Technology Challenges (DSTC6) aiming to develop end-to-end dialogue systems. Neural network models have become a recent focus of investigation in dialogue technologies. Previous models required training data to be manually annotated with word meanings and dialogue states, but end-to-end neural network dialogue systems learn to directly output natural-language system responses without needing training data to be manually annotated. Thus, this approach allows us to scale up the size of training data and cover more dialog domains. In addition, dialogue systems require a meta-function to avoid deploying inappropriate responses generated by themselves. To challenge such issues, the DSTC6 consists of three tracks, (1). End-to-End Goal Oriented dialogue Learning to select system responses, (2). End-to-End Conversation Modeling to generate system responses using Natural Language Generation (NLG) and (3). Dialogue Breakdown Detection. Since each domain has different issues to be addressed to develop dialogue systems, we targeted restaurant retrieval dialogues to fill slot-value in Track 1, customer services on Twitter by combining goal-oriented dialogues and ChitChat in Track 2 and human-machine dialogue data for ChitChat in Track 3. DSTC6 had 141 people declaring their interests and 23 teams submitted their final results. 18 scientific papers were presented in the wrap-up workshop. We find the blending end-to-end trainable models associated to meaningful prior knowledge performs the best for the restaurant retrieval for Track 1. Indeed, Hybrid Code Network and Memory Network have been the best models for this task. In Track 2, 78.5% of the system responses automatically generated by the best system were rated better than acceptable by humans and this achieves 89% of the number of the human responses rated in the same class. In Track3, the dialogue breakdown detection technologies performed as well as human agreements, in both data-sets of English and Japanese. [ABSTRACT FROM AUTHOR]

Highlights • Different from previous studies that use a single label as the gold-standard to decide dialogue breakdown, we focused on inherent subjectivity when people regard dialogue breakdowns and variationality due to the design of conversational agents. • We propose methods for dialogue breakdown detection that first cluster annotators based on their label distributions, and then ensemble detectors trained by different sets of annotators. • Two kinds of neural networks are used as detectors. We assume that breakdowns occur locally (word and phrase levels) and/or globally (sentence and context levels). The convolutional neural network (CNN) is applied to detect former type of breakdowns, and long short-term memory (LSTM) detects the latter type of breakdowns. • Our best model achieves 63.6% F-score on breakdown detection, which is 5.6% higher than the baseline using the conditional random fields. • This is an extended version of our workshop paper submitted to the Dialogue Breakdown Detection Challange 3 (DBDC3). We newly conducted additional experiments that confirm effectiveness of clustering annotators based on their annotation distributions. In addition, detailed error analysis revealed characteristics of the CNN and LSTM on the dialogue breakdown detection task. Abstract Dialogue breakdown is a significant problem in conversational agents. Timely breakdown detection helps the agents quickly recover from mistakes, minimizing the impact on user experience. In this paper, we focus on two problems: variations in determining a response that breakdowns a conversation i.e. , subjectivity, and variations in breakdown types due to designs of conversational agents, i.e. , variationality. To address the subjectivity, which decreases the agreement rate among annotators, our methods detect a dialogue breakdown by ensembling detectors trained by different sets of annotators that are grouped using a clustering algorithm. To address the variationality, our methods apply two types of detector architectures to capture global and local breakdowns. The long short-term memory detector considers the global context and the convolutional neural networks detector is sensitive to the local characteristics. The ensemble of all detectors makes a final judgment. The results of the Japanese task in the Dialogue Breakdown Detection Challenge 3 (DBDC3) confirm that our approach significantly outperforms the baseline, which uses the conventional conditional random field. Detailed error analysis reveals that our encoders based on a convolutional neural network and a long short-term memory have different characteristics. It also confirms the effects of annotator clustering. [ABSTRACT FROM AUTHOR]

Highlights • An advanced end to end conversation system for the 6-th edition of Dialog System Technology Challenge (DSTC6). • Applying sequence adversarial training with extension of the objective function to improve both objective and subjective evaluation metrics. • Minimum Bayes risk (MBR) based system combination of multiple neural conversation models. • Example-based response selection using an embedding-based context similarity. • Thorough evaluation of three different neural conversation models, training techniques, and decoding strategies using a help-desk dialog task in DSTC6. Abstract This paper presents adversarial training and decoding methods for neural conversation models that can generate natural responses given dialog contexts. In our prior work, we built several end-to-end conversation systems for the 6th Dialog System Technology Challenges (DSTC6) Twitter help-desk dialog task. These systems included novel extensions of sequence adversarial training, example-based response extraction, and Minimum Bayes-Risk based system combination. In DSTC6, our systems achieved the best performance in most objective measures such as BLEU and METEOR scores and decent performance in a subjective measure based on human rating. In this paper, we provide a complete set of our experiments for DSTC6 and further extend the training and decoding strategies more focusing on improving the subjective measure, where we combine responses of three adversarial models. Experimental results demonstrate that the extended methods improve the human rating score and outperform the best score in DSTC6. [ABSTRACT FROM AUTHOR]

• A novel approach for single channel speech enhancement is presented, using the colored spectrograms. • Using deep neural network for spectrogram denoising and regression neural network for translating the colors of denoised spectrograms to short time Fouier transform (STFT) magnitudes. • Output is comparable to the state-of-the-art baseline models at a much reduced computational cost. Speech enhancement concerns the processes required to remove unwanted background sounds from the target speech to improve its quality and intelligibility. In this paper, a novel approach for single-channel speech enhancement is presented using colored spectrograms. We propose the use of a deep neural network (DNN) architecture adapted from the pix2pix generative adversarial network (GAN) and train it over colored spectrograms of speech to denoise them. After denoising, the colors of spectrograms are translated to magnitudes of short-time Fourier transform (STFT) using a shallow regression neural network. These estimated STFT magnitudes are later combined with the noisy phases to obtain an enhanced speech. The results show an improvement of almost 0.84 points in the perceptual evaluation of speech quality (PESQ) and 1 % in the short-term objective intelligibility (STOI) over the unprocessed noisy data. The gain in quality and intelligibility over the unprocessed signal is almost equal to the gain achieved by the baseline methods used for comparison with the proposed model, but at a much reduced computational cost. The proposed solution offers a comparative PESQ score at almost 10 times reduced computational cost than a similar baseline model that has generated the highest PESQ score trained on grayscaled spectrograms, while it provides only a 1 % deficit in STOI at 28 times reduced computational cost when compared to another baseline system based on convolutional neural network-GAN (CNN-GAN) that produces the most intelligible speech. [ABSTRACT FROM AUTHOR]

Conventional automatic speaker verification systems can usually be decomposed into a front-end model such as time delay neural network (TDNN) for extracting speaker embeddings and a back-end model such as statistics-based probabilistic linear discriminant analysis (PLDA) or neural network-based neural PLDA (NPLDA) for similarity scoring. However, the sequential optimization of the front-end and back-end models may lead to a local minimum, which theoretically prevents the whole system from achieving the best optimization. Although some methods have been proposed for jointly optimizing the two models, such as the generalized end-to-end (GE2E) model and NPLDA E2E model, most of these methods have not fully investigated how to model the intra-relationship between multiple enrollment utterances. In this paper, we propose a new E2E joint method for speaker verification especially designed for the practical scenario of multiple enrollment utterances. To leverage the intra-relationship among multiple enrollment utterances, our model comes equipped with frame-level and utterance-level attention mechanisms. Additionally, focal loss is utilized to balance the importance of positive and negative samples within a mini-batch and focus on the difficult samples during the training process. We also utilize several data augmentation techniques, including conventional noise augmentation using MUSAN and RIRs datasets and a unique speaker embedding-level mixup strategy for better optimization. • Extend attention back-end to end-to-end for multi-enrollment speaker verification. • Introduce focal loss, suggest speaker embedding mixup for fine-tuning end-to-end model. • Analyze the effect of the number of enrollment utterances. • Analyze the embedding-level attention weights for speech genres. [ABSTRACT FROM AUTHOR]

In many speaker verification tasks, the quality of speaker embedding is an important factor in affecting speaker verification systems. Advanced speaker embedding extraction networks aim to capture richer speaker features through the multi-branch network architecture. Recently, speaker verification systems based on transformer encoders have received much attention, and many satisfactory results have been achieved because transformer encoders can efficiently extract the global features of the speaker (e.g., MFA-Conformer). However, the large number of model parameters and computational latency are common problems faced by the above approaches, which make them difficult to apply to resource-constrained edge terminals. To address this issue, this paper proposes an effective, lightweight transformer model (MCA-former) with multi-scale convolutional self-attention (MCA), which can perform multi-scale modeling and channel modeling in the temporal direction of the input with low computational cost. In addition, in the inference phase of the model, we further develop a systematic re-parameterization method to convert the multi-branch network structure into the single-path topology, effectively improving the inference speed. We investigate the performance of the MCA-former for speaker verification under the VoxCeleb1 test set. The results show that the MCA-based transformer model is more advantageous in terms of the number of parameters and inference efficiency. By applying the re-parameterization, the inference speed of the model is increased by about 30%, and the memory consumption is significantly improved. • Designing a lightweight multi-scale convolutional self-attention module. • An efficient transformer encoder for speaker verification. • Using the re-parameterization method improves the model's inference efficiency. [ABSTRACT FROM AUTHOR]

In this paper an unsupervised clustering algorithm is developed for acoustic data in the context of speech recognition tasks. One of the key features of the algorithm is scalability to large data sets. Specifically, given the unlabeled training and test sets, the class-labels of the utterances are obtained in an automatic manner. The extracted labels may correspond to the speakers in the speech corpus if the data is relatively clean. The proposed scheme is attractive from an industrial perspective as it alleviates the need to store the speaker-labels manually, saving considerable amount of human efforts and expenses. The core of the algorithm comprises a three-stage architecture that processes the input data one after the other, while each stage is designed to perform a well-defined and specific task. In more detail, the first-pass involves a bottom-up clustering mechanism, the second-pass comprises a cluster splitting operation and the third-pass consists of a cluster refining process. Each of the stages allows for data parallelization using multiple CPUs that leads to faster computation. Two alternative forms of the algorithm are presented – the first considers Gaussian distributions and the other i-Vectors – to facilitate the clustering. Although the algorithm may find applications in various realms of speech recognition, in this paper, the effectiveness of the schemes are evaluated by means of speaker adaptive training (SAT) and speaker-aware training of DNN-HMM acoustic models. In particular, experiments are conducted on the Switchboard task to extract the speaker-labels for the utterances in the training and test sets. It is shown that the SAT DNN-HMM trained using the Gaussian based scheme yields a 7.2% relative improvement in the ASR accuracy over the speaker independent DNN-HMM, whereas the i-Vector approach provides an additional improvement, amounting to a 10.8% relative gain overall. The standard SAT DNN-HMM developed using the ground-truth speaker-labels is found to be only 2.7% relative better than the proposed scheme. Similar observation is made as with speaker-aware training. The analysis of computational complexity, conducted stage by stage, demonstrates the scalability of the proposed algorithms. [ABSTRACT FROM AUTHOR]

Peer-Led Team Learning (PLTL) is a learning methodology where a peer-leader co-ordinate a small-group of students to collaboratively solve technical problems. PLTL have been adopted for various science, engineering, technology and maths courses in several US universities. This paper proposed and evaluated a speech system for behavioral analysis of PLTL groups. It could help in identifying the best practices for PLTL. The CRSS-PLTL corpus was used for evaluation of developed algorithms. In this paper, we developed a robust speech activity detection (SAD) by fusing the outputs of a DNN-based pitch extractor and an unsupervised SAD based on voicing measures. Robust speaker diarization system consisted of bottleneck features (from stacked autoencoder) and informed HMM-based joint segmentation and clustering system. Behavioral characteristics such as participation, dominance, emphasis, curiosity and engagement were extracted by acoustic analyses of speech segments belonging to all students. We proposed a novel method for detecting question inflection and performed equal error rate analysis on PLTL corpus. In addition, a robust approach for detecting emphasized speech regions was also proposed. Further, we performed exploratory data analysis for understanding the distortion present in CRSS-PLTL corpus as it was collected in naturalistic scenario. The ground-truth Likert scale ratings were used for capturing the team dynamics in terms of student’s responses to a variety of evaluation questions. Results suggested the applicability of proposed system for behavioral analysis of small-group conversations such as PLTL, work-place meetings etc. . [ABSTRACT FROM AUTHOR]

Speech difficulties are common in children and teenagers, but they can also occur in adults as a result of physical problems. A speech disorder is a situation in which an individual struggles to produce or construct the spoken sounds necessary for interpersonal communication. As a result, it could be challenging to comprehend the person's speech. Articulation abnormalities are typical speech problems. In this situation, automatic speech recognition (ASR) technology may be used to detect and further rectify such deficiencies. The first attempts to detect speech abnormalities were made in the early 1970s, and they appear to have followed the same path as those on the ASR. These early experiments did rely heavily on signal processing techniques. As time goes on, more ideas from ASR technology are being incorporated into systems that deal with speech impairments. Many traditional techniques are executed in the ASR system. In this paper, we developed an automatic speech recognition technology based on deep learning techniques. In this paper, we research alternative extraction and classification methods of electroencephalography (EEG) to help diagnose speech disorders (SD). The EEG data is prepared before degradation into numerous EEG sub-strands with a discrete wavelet transformation to eliminate unimportant errors. For sharpening signals, the Eigenvector crack curvature wavelet method was used. A hyper-similarity abnormality coder is used for feature extraction in the EEG recording and to detect synchronization between EEG channels, which may show abnormalities in communication. The recovered functions are then categorized using the Deep Residual–encoder–based VGG net CNN Classification Method. Thus, the techniques proposed to produce the most promising outcome aren't the suggested technique attained better classification accuracy when compared to the traditional methodologies. [ABSTRACT FROM AUTHOR]

We introduce a set of benchmark corpora of conversational English speech derived from the Switchboard-I and Fisher datasets. Traditional automatic speech recognition (ASR) research requires considerable computational resources and has slow experimental turnaround times. Our goal is to introduce these new datasets to researchers in the ASR and machine learning communities in order to facilitate the development of novel speech recognition techniques on smaller but still acoustically rich, diverse, and hence interesting corpora. We select these corpora to maximize an acoustic quality criterion while limiting the vocabulary size (from 10 words up to 10,000 words), where both “acoustic quality” and vocabulary size are adeptly measured via various submodular functions. We also survey numerous submodular functions that could be useful to measure both “acoustic quality” and “corpus complexity” and offer guidelines on when and why a scientist may wish use to one vs. another. The corpora selection process itself is naturally performed using various state-of-the-art submodular function optimization procedures, including submodular level-set constrained submodular optimization (SCSC/SCSK), difference-of-submodular (DS) optimization, and unconstrained submodular minimization (SFM), all of which are fully defined herein. While the focus of this paper is on the resultant speech corpora, and the survey of possible objectives, a consequence of the paper is a thorough empirical comparison of the relative merits of these modern submodular optimization procedures. We provide baseline word recognition results on all of the resultant speech corpora for both Gaussian mixture model (GMM) and deep neural network (DNN)-based systems, and we have released all of the corpora definitions and Kaldi training recipes for free in the public domain. [ABSTRACT FROM AUTHOR]

Physical exertion is a stress condition that affects how we normally produce speech. It alters both the temporal and spectral pattern of speech characteristics. Therefore, speech utterances can be used as a cost-effective telehealth solution to detect whether a person is under physical exertion. This paper deals with the detection of shortness of breath (or out-of-breath) condition from speech under physical load by using an multi-task learning (MTL) framework. The primary classification targets are the neutral and the out-of-breath classes. Naturally these are binary targets and do not reflect the actual extent of exertion. This leads to the creation of a novel auxiliary target learning in collaboration with a pre-trained expert system. The targets here indicate the level of exertion under physical load. The MTL framework for both the convolutional neural network (CNN) and CNN-long short-term memory (CLSTM) network performs nearly 1.5% (F1-score) better than the single task learning (STL) framework. In addition to that, out-of-breath speech has more influence on the lower frequency spectrum. Warped spectrograms are given as input to the networks, enabling the deep networks to focus on lower spectral regions. The non-linear frequency warping is achieved by Mel-scale transformation and constant-Q-transform (CQT). CQT, being less dependent on fixed-window size, shows at least 6.57% (F1-score) improvement over Mel-based inputs. The MTL framework, combined with the warped spectrum, performs better in classifying out-of-breath speech from neutral. • Aims to detect speech-based out-of-breath (or physical load) conditions. • Efficiently determines the extent of physical exertion using speech utterances. • Lower frequency-centric approach for better detection of out-of-breath condition. • Multi-task learning further improves the performance by combing the above two steps. [ABSTRACT FROM AUTHOR]

• A novel voice conversion technique for body-conducted unvoiced speech is proposed based on Taylor-Boltzmann Machines. The proposed algorithm was the integration of Taylor Series with Boltzmann machine Deep Neural Networks. • A body conductive microphone referred as the non audible murmur (NAM) microphone is used to capture the non-audible murmur from the neck part behind the ear. • The classifier was trained with the features likes "pitch chroma, spectral centroid, and spectral skewness" that are derived from the incoming body conducted low voice verbal communication signal. • In the initial training phase, a Taylor series approximation DNN model is trained from a collection of stereo data which encompasses the NAM speech of same phonemes represented as log-power spectral features. • Moreover, in the recognition phase, a well trained Taylor-DNN model that is based on Boltzmann Machine intakes the speech features and generates the "log-power-spectra features" and it aids in recognizing the speech waveform. • The paper may gain high interest from the readers and experts of Speech processing, Voice Conversion Techniques, body-conducted unvoiced speech. Communication becomes effectual while the verbal communication signal reaches the destination with profound characteristics. An application based on human voice attracts the researcher's attention. It leads the researcher to proceed towards the speech conversion for low voice humans having murmured voices as their attitude. The researchers introduced a technique for improving the body-conducted unvoiced speech for silent speech communication using Taylor-Boltzmann machine-trained optimized Deep Neural Network (DNN). The projected model includes two main stages: the training stage as well as recognition stage. In the training stage, the following functionalities are undergone: (i) pre-processing, (ii) feature extraction, and (iii) deep learning-based speech enhancement. Initially, the collected input Non-Audible Murmur (NAM) speech signal is subjected to the pre-processing stage, wherein the improved wiener filtering is employed for suppressing the noise in the input signal. From the pre-processed speech signal, the most relevant characteristics such as "spectral skewness, spectral chroma, spectral centroid and improved Mel Frequency Cepstral Coefficients (MFCC)" are extracted, and they are fused together. Then, using these fused features, the Taylor series with optimized DNN and Boltzmann Machine (Taylor-DNN-BM) model is trained. The final enhanced speech is acquired from the Taylor- optimized DNN-BM. The recognition phase encapsulates the feature extraction and deep learning-based speech enhancement stage. The input speech data enters into the feature extraction phase, wherein the characteristics such as "spectral skewness, spectral chroma, spectral centroid and improved (MFCC)" are retrieved. After that, these features are fused and fed as input to the Taylor-optimized DNN-BM model. From Taylor- optimized DNN-BM, the enhanced speech signal is acquired. In DNN, the weight function is fine-tuned using the newly projected Self Improved Flower Pollination Algorithm (SI-FPA) algorithm. This SI-FPA model is the conceptual improvement of the Flower Pollination Algorithm (FPA). Finally, the projected model is compared with the existing models to validate its efficiency in terms of speech enhancement. [ABSTRACT FROM AUTHOR]

This paper discusses how automatic speech recognition systems are and could be designed, in order to best exploit the discriminative information encoded in human speech. This contrasts with many recent machine learning approaches that apply general recognition architectures to signals to identify, with little concern for the nature of the input. The implicit assumption has often been that training can automatically discover the useful properties that exist in signals, with minimal manual intervention. These approaches may be suitable for some tasks such as image recognition, where the diversity of visual input is vast; e.g., an image may be any (natural or synthetic) scene that a camera views. We first examine what makes speech special, i.e., a natural signal from a complex tube, driven by a source that is quasi-periodic and/or noisy, aiming to communicate a wide variety of information, using the different vocal systems of human speakers. Then, we view how pertinent features are extracted from speech via efficient means, related to the objectives of communication. We see how to reliably and efficiently identify the different units of oral language. We learn from the history of attempts to do ASR, e.g., why they succeeded and how improved methods exploited the increasing availability of data and computer power (in particular, deep neural networks). Finally, we suggest ways to render ASR both more accurate and efficient. This work is aimed at both newcomers to ASR and experts, in terms of presenting issues broadly, but without mathematical or algorithmic details, which are readily found in the references. [ABSTRACT FROM AUTHOR]

Sexism, a form of oppression based on one's sex, manifests itself in numerous ways and causes enormous suffering. In view of the growing number of experiences of sexism reported online, automatically classifying these recollections can aid in the battle against sexism by allowing gender studies researchers and government officials involved in policymaking to conduct more effective analyses. This paper investigates the 23-class fine-grained, multi-label classification of accounts (reports) of sexism. Moreover, we propose a knowledge-based cascaded multi-task framework for fine-grained multi-label sexism classification. We leverage several supporting tasks, including homogeneous and heterogeneous auxiliary tasks. Homogeneous tasks are set up without incurring any manual labeling cost and heterogeneous tasks are set up which have a high correlation with the accounts of sexism. Unlabeled accounts of sexism are utilized through unsupervised learning to help construct our multi-task setup. In addition, we incorporate a knowledge module within the framework to infuse external knowledge features into the learning process. Further, we investigate transfer learning that employs weakly labeled accounts of sexism and transfers the learning to the multi-label sexism classification. We also devise objective functions that exploit label correlations in the training data explicitly. Multiple proposed multi-task methods outperform the state-of-the-art multi-label sexism classification across five standard metrics. • Propose a knowledge-based cascaded multi-task neural framework for the multi-label sexism classification. • Explore two types of auxiliary tasks: (a) Homogeneous tasks and (b) Heterogeneous tasks • Trained the model with weakly labeled data and transfered the knowledge to the sexism classification task. • Proposed loss functions aimed at tapping label correlations in the multi-label data explicitly. [ABSTRACT FROM AUTHOR]

• In this paper, an optimal approach for text feature Selection , we work on text categorization and propose a statistical-based feature selection method (MFX) that considers all documents from the same category as one extended document, and chooses the most discriminative terms that are frequent and common across all documents of the same category, but rarely present in other categories. MFX is language independent and backed up with a mathematical formulation that finds the optimal number of features that guarantees accurate text categorization. Experimental results show the superiority of MFX over the state-of-the-art existing techniques. This work is very significant and timely given its applicability in applications such as spam filtering, opinion mining and topic spotting, among others. Traditionally, feature selection is conducted by first deriving a candidate list of features, then ranking and selecting the top features based on predefined threshold. These methods are highly dependent on the choice of the threshold, and therefore lead to sub-optimal text categorization results. In this paper, we address the selection problem by suggesting a one-step method designed to optimally select the subset of features. The selection is formulated mathematically as an optimization problem with the objective of maximizing classification accuracy while simultaneously deriving and choosing the most discriminative features. Our method, MFX, is applicable to many of the conventional methods, with two distinguishing aspects. First, it is based on considering all documents from the same category as one extended document, instead of analyzing individual documents. Second, it considers choosing the most discriminative terms that are frequent and common across all documents of the same category, and minimally present in other categories. Moreover, MFX is language-independent. It was tested on the well-known benchmark Reuters RCV1 dataset. To showcase its language independence, MFX was also tested on Arabic datasets extracted from Arabic news sources. The results indicated that MFX always performed similar to or better than other well-known feature selection methods. MFX with a Support Vector Machine (SVM) classifier was also shown to outperform recent text classification algorithms based on neural networks and word embeddings. [ABSTRACT FROM AUTHOR]

This paper addresses the issue of language model adaptation for Recurrent Neural Network Language Models ( rnnlm s), which have recently emerged as a state-of-the-art method for language modeling in the area of speech recognition. Curriculum learning is an established machine learning approach that achieves better models by applying a curriculum, i.e., a well-planned ordering of the training data, during the learning process. Our contribution is to demonstrate the importance of curriculum learning methods for adapting rnnlm s and to provide key insights on how they should be applied. rnnlm s model language in a continuous space and can theoretically exploit word-dependency information over arbitrarily long distances. These characteristics give rnnlm s the ability to learn patterns robustly with relatively little training data, implying that they are well suited for adaptation. In this paper, we focus on two related challenges facing language models: within-domain adaptation and limited-data within-domain adaptation . We propose three types of curricula that start with general data, i.e., characterizing the domain as a whole, and move towards specific data, i.e., characterizing the sub-domain targeted for adaptation. Effectively, these curricula result in a model that can be considered to represent an implicit interpolation between general data and sub-domain-specific data. We carry out an extensive set of experiments that investigates how adapting rnnlm s using curriculum learning can improve their performance. Our first set of experiments addresses the within-domain adaptation challenge, i.e., creating models that are adapted to specific sub-domains that are part of a larger, heterogeneous domain of speech data. Under this challenge, all training data is available to the system at the time when the language model is trained. First, we demonstrate that curriculum learning can be used to create effective sub-domain-adapted rnnlm s. Second, we show that a combination of sub-domain-adapted rnnlm s can be used if the sub-domain of the target data is unknown at test time. Third, we explore the potential of applying combinations of sub-domain-adapted rnnlm s to data for which sub-domain information is unknown at training time and must be inferred. Our second set of experiments addresses limited-data within-domain adaptation, i.e., adapting an existing model trained on a large set of data using a smaller amount of data from the target sub-domain. Under this challenge, data from the target sub-domain is not available at the time when the language model is trained, but rather becomes available little by little over time. We demonstrate that the implicit interpolation carried out by applying curriculum learning methods to rnnlm s outperforms conventional interpolation and has the potential to make more of less adaptation data. [ABSTRACT FROM AUTHOR]

This paper examines the individual and combined impacts of various front-end approaches on the performance of deep neural network (DNN) based speech recognition systems in distant talking situations, where acoustic environmental distortion degrades the recognition performance. Training of a DNN-based acoustic model consists of generation of state alignments followed by learning the network parameters. This paper first shows that the network parameters are more sensitive to the speech quality than the alignments and thus this stage requires improvement. Then, various front-end robustness approaches to addressing this problem are categorised based on functionality. The degree to which each class of approaches impacts the performance of DNN-based acoustic models is examined experimentally. Based on the results, a front-end processing pipeline is proposed for efficiently combining different classes of approaches. Using this front-end, the combined effects of different classes of approaches are further evaluated in a single distant microphone-based meeting transcription task with both speaker independent (SI) and speaker adaptive training (SAT) set-ups. By combining multiple speech enhancement results, multiple types of features, and feature transformation, the front-end shows relative performance gains of 7.24% and 9.83% in the SI and SAT scenarios, respectively, over competitive DNN-based systems using log mel-filter bank features. [ABSTRACT FROM AUTHOR]

In this paper, we present a survey on the application of recurrent neural networks to the task of statistical language modeling. Although it has been shown that these models obtain good performance on this task, often superior to other state-of-the-art techniques, they suffer from some important drawbacks, including a very long training time and limitations on the number of context words that can be taken into account in practice. Recent extensions to recurrent neural network models have been developed in an attempt to address these drawbacks. This paper gives an overview of the most important extensions. Each technique is described and its performance on statistical language modeling, as described in the existing literature, is discussed. Our structured overview makes it possible to detect the most promising techniques in the field of recurrent neural networks, applied to language modeling, but it also highlights the techniques for which further research is required. [ABSTRACT FROM AUTHOR]

• A series of neural adversarial learning methods are proposed for speaker recognition. • Deep learning is implemented for PLDA speaker recognition with i -vectors. • Adversarial manifold learning is proposed to build a subspace model with neighbor embeddings in latent variables. • Multiobjective learning is performed for minimax optimization. • Adversarial augmentation learning is proposed to compensate the imbalanced data problem in speaker recognition. This paper presents the adversarial learning approaches to deal with various tasks in speaker recognition based on probabilistic discriminant analysis (PLDA) which is seen as a latent variable model for reconstruction of i -vectors. The first task aims to reduce the dimension of i -vectors based on an adversarial manifold learning where the adversarial neural networks of generator and discriminator are merged to preserve neighbor embedding of i -vectors in a low-dimensional space. The generator is trained to fool the discriminator with the generated samples in latent space. A PLDA subspace model is constructed by jointly minimizing a PLDA reconstruction error, a manifold loss for neighbor embedding and an adversarial loss caused by the generator and discriminator. The second task of adversarial learning is developed to tackle the imbalanced data problem. A PLDA based generative adversarial network is trained to generate new i -vectors to balance the size of training utterances across different speakers. An adversarial augmentation learning is proposed for robust speaker recognition. In particular, the minimax optimization is performed to estimate a generator and a discriminator where the class conditional i -vectors produced by generator could not be distinguished from real i -vectors via discriminator. A multiobjective learning is realized for a specialized neural model with the cosine similarity between real and fake i -vectors as well as the regularization for Gaussianity. Experiments are conducted to show the merit of adversarial learning in subspace construction and data augmentation for PLDA-based speaker recognition. [ABSTRACT FROM AUTHOR]

• Novel deep domain architectures based on CNN for spoofing detection are proposed. • A comparison of features using a CNN as a back-end classifier. • An extensive cross-database evaluation on ASVspoof and BTAS databases. • Shows that deep domain adaptation in both supervised and unsupervised settings improve the spoofing detection performance. With the increasing use of voice as a biometric, it has become imperative to develop countermeasures to thwart malicious spoofing attacks on speaker recognition systems. Even though there has been a significant research effort over the last few years dedicated to the development of countermeasures to detect and deflect spoofing attacks, the problem is far from being solved. While a deep learning technique has been successfully applied in anti-spoofing research, it suffers from a data scarcity issue where large amounts of labeled training data are required to build a robust model. In this paper, we investigate a domain adaptation approach of deep architectures in both a supervised setting where we use labeled data, and in an unsupervised setting where we assume unlabeled data when transferring knowledge from the source to target domain. Specifically, we employ convolutional neural networks (CNNs) as back-end classifiers for spoofed speech detection. For supervised domain adaptation, we propose joint neural network training while allowing the weights to be shared between the source and target streams, and an additional domain regularizer. In the unsupervised domain adaptation scenario, the weights are not shared in order to explicitly model the domain shift. However, the weights are related by weight regularizers to take into account the difference between the two domains. We conduct extensive cross-database (domain mismatch) experiments using ASVspoof 2015 and BTAS 2016 datasets to demonstrate the generalization capability of the proposed deep domain architectures for spoofing detection. Experimental results reveal that the proposed architectures can generalize across databases for both supervised and unsupervised adaptation scenarios. [ABSTRACT FROM AUTHOR]

Keyphrases of a given document represent its main topic and they are used as a simple method to represent the document. Statistical and graph-based models as unsupervised approaches have been mainly studied. The statistical models have some difficulty in extracting keyphrases from a single document because most statistical ones generally require statistical information from a large corpus. On the other hand, graph-based models can extract keyphrases by only using the information from a single document; nevertheless, they have some drawbacks. The scores of the edges can be biased because a single document does not contain sufficient information to score the edges of a graph and this influences the score of the nodes. In this paper, we propose an effective combination method of a statistical model, C-value method, and a graph-based model to overcome the drawbacks of each model. A new scoring method for keyphrase candidates is developed by the graph-based model and the scores calculated by the new method are applied to the modified C-value method to estimate the final importance scores of the keyphrase candidates. Subsequently, the proposed model is evaluated using two datasets, SemEval 2010 and Inspec, and its results outperformed the state-of-the-art model among unsupervised models and the existing graph-based ranking models. [ABSTRACT FROM AUTHOR]

• Examine an LSTM neural tagging model for Named Entity Recognition (NER). • Demonstrate the capability of the proposed model with both word and character levels and how it can handle out-of-vocabulary words. • Convolutional Neural Network can be one of the promising technologies that can successfully extract character-level representations. • Compare different word vector models for the task of Arabic NER. • Conduct experimental evaluation that shows that our model performance outperforms state-of-the-art Arabic NER systems on many standard benchmarks. Named Entity Recognition (NER) is a significant information extraction task since it is an important component of many natural language processing applications, such as Information Retrieval, Question Answering and Speech Recognition. The complexity and morphological richness of the Arabic language is the main reason why most existing Arabic NER systems rely strongly on hand-crafted feature engineering. In this paper, we propose to augment the existing LSTM neural tagging model for Arabic NER with a Convolutional Neural Network (CNN) for the extraction of relevant character-level features. By operating on the character-level, the proposed model is able to handle out-of-vocabulary words. Our results show that character CNN is able to outperform the previously used character-level Bi-directional Long Short-Term Memory Networks (BiLSTM) in many settings. Moreover, our observations indicate that CNNs tend to perform better than BiLSTM on relatively longer tokens. In addition, we conduct a comparison of four different pre-trained word vector models for Arabic NER and results show that a Skip-Gram Word2vec model, pre-trained on a subset of the Arabic Gigaword corpus, is generally sufficient to obtain acceptable Arabic NER performance. [ABSTRACT FROM AUTHOR]

In this paper, we address the problem of speaker verification in conditions unseen or unknown during development. A standard method for speaker verification consists of extracting speaker embeddings with a deep neural network and processing them through a backend composed of probabilistic linear discriminant analysis (PLDA) and global logistic regression score calibration. This method is known to result in systems that work poorly on conditions different from those used to train the calibration model. We propose to modify the standard backend, introducing an adaptive calibrator that uses duration and other automatically extracted side-information to adapt to the conditions of the inputs. The backend is trained discriminatively to optimize binary cross-entropy. When trained on a number of diverse datasets that are labeled only with respect to speaker, the proposed backend consistently and, in some cases, dramatically improves calibration, compared to the standard PLDA approach, on a number of held-out datasets, some of which are markedly different from the training data. Discrimination performance is also consistently improved. We show that joint training of the PLDA and the adaptive calibrator is essential — the same benefits cannot be achieved when freezing PLDA and fine-tuning the calibrator. To our knowledge, the results in this paper are the first evidence in the literature that it is possible to develop a speaker verification system with robust out-of-the-box performance on a large variety of conditions. • A standard backend for speaker verification is based on PLDA followed by global calibration. • We propose a novel backend that consists of a PLDA-like formulation followed by condition-aware calibration. • The backend is trained discriminatively to optimize binary cross-entropy on a large variety of conditions. • We show that this backend results in significant gains over the standard PLDA approach. • The proposed approach results in a system that can be used out-of-the-box on data from different conditions. [ABSTRACT FROM AUTHOR]

Highlights • Automatic metric for evaluating natural language generated sentences for dialog systems. • Integration of adequacy and fluency information to jointly evaluate semantic and syntactic information. • Higher correlation (up to 12.8%) with human evaluations than the best current objective metrics at a system and sentence level. End-to-end dialog systems are gaining interest due to the recent advances of deep neural networks and the availability of large human–human dialog corpora. However, in spite of being of fundamental importance to systematically improve the performance of this kind of systems, automatic evaluation of the generated dialog utterances is still an unsolved problem. Indeed, most of the proposed objective metrics shown low correlation with human evaluations. In this paper, we evaluate a two-dimensional evaluation metric that is designed to operate at sentence level, which considers the syntactic and semantic information carried along the answers generated by an end-to-end dialog system with respect to a set of references. The proposed metric, when applied to outputs generated by the systems participating in track 2 of the DSTC-6 challenge, shows a higher correlation with human evaluations (up to 12.8% relative improvement at the system level) than the best of the alternative state-of-the-art automatic metrics currently available. [ABSTRACT FROM AUTHOR]

Deep neural networks (DNNs) have recently been successful in many applications and have become a popular approach for speech recognition. Training a DNN model for speech recognition is computationally expensive due to the model large number of parameters. Pre-training improves DNN modeling. However, DNN learning is challenging if pre-training is inefficient. This paper introduces a new framework for pre-training that utilizes label information in lower layers (layers near input) for better recognition. The proposed pre-training method dynamically inserts discriminative information not only in the last layer but also in other layers. In this algorithm, the lower layers achieve more generative information while the higher layers achieve more discriminative information. In addition, this method uses speaker information by employing the Subspace Gaussian Mixture Model (SGMM), which improves recognition accuracy. Experimental results on TIMIT, MNIST, Switchboard, and English Broadcast News datasets show that this approach significantly outperforms current state-of-the-art methods such as the Deep Belief Network and the Deep Boltzmann Machine. Moreover, the proposed algorithm has minimal memory requirements. [ABSTRACT FROM AUTHOR]

Abstract In this paper, we introduce the Task Dependent Recurrent Entity Network (TDREN) to solve Dialogue System Technology Challenges 6 (DSTC 6) track 1. Traditionally, there have been methods such as collecting the intent of the user in a conversation directly using rules. We design an end-to-end structure that properly models the restaurant pre-related user preferences that appear in the dialogue and gives appropriate responses. We perform experiments on the TDREN and achieved 97.7% at precision 1. We propose a new artificial neural network structure and recurrent cell for modeling user preference information. Then, we show that task-oriented dialogue modeling experiment results using the structure and the recurrent cell. [ABSTRACT FROM AUTHOR]

Former comparisons of human speech recognition (HSR) and automatic speech recognition (ASR) have shown that humans outperform ASR systems in nearly all speech recognition tasks. However, recent progress in ASR has led to substantial improvements of recognition accuracy, and it is therefore unclear how large the task-dependent human-machine gap still remains. This paper investigates this gap between HSR and ASR based on deep neural networks (DNNs) in different acoustic conditions, with the aim of comparing differences and identifying processing strategies that should be considered in ASR. We find that DNN-based ASR reaches human performance for single-channel, small-vocabulary tasks in the presence of speech-shaped noise and in multi-talker babble noise, which is an important difference to previous human-machine comparisons: The speech reception threshold, i.e., the signal-to-noise ratio with 50% word recognition rate is at about −7 to −8 dB both for HSR and ASR. However, in more complex spatial scenes with diffuse noise and moving talkers, the SRT gap amounts to approximately 12 dB. Based on cross comparisons that use oracle knowledge (e.g., the speakers’ true position), incorrect responses are attributed to localization errors or missing pitch information to distinguish between speakers with different gender. In terms of the SRT, localization errors and missing spectral information amount to 2.1 and 3.2 dB, respectively. The comparison hence identifies specific components in ASR that can profit from learning from auditory signal processing. [ABSTRACT FROM AUTHOR]

Abstract: The basic goal of the voice conversion system is to modify the speaker-specific characteristics, keeping the message and the environmental information contained in the speech signal intact. Speaker characteristics reflect in speech at different levels, such as, the shape of the glottal pulse (excitation source characteristics), the shape of the vocal tract (vocal tract system characteristics) and the long-term features (suprasegmental or prosodic characteristics). In this paper, we are proposing neural network models for developing mapping functions at each level. The features used for developing the mapping functions are extracted using pitch synchronous analysis. Pitch synchronous analysis provides the estimation of accurate vocal tract parameters, by analyzing the speech signal independently in each pitch period without influenced by the adjacent pitch cycles. In this work, the instants of significant excitation are used as pitch markers to perform the pitch synchronous analysis. The instants of significant excitation correspond to the instants of glottal closure (epochs) in the case of voiced speech, and to some random excitations like onset of burst in the case of nonvoiced speech. Instants of significant excitation are computed from the linear prediction (LP) residual of speech signals by using the property of average group-delay of minimum phase signals. In this paper, line spectral frequencies (LSFs) are used for representing the vocal tract characteristics, and for developing its associated mapping function. LP residual of the speech signal is viewed as excitation source, and the residual samples around the instant of glottal closure are used for mapping. Prosodic parameters at syllable and phrase levels are used for deriving the mapping function. Source and system level mapping functions are derived pitch synchronously, and the incorporation of target prosodic parameters is performed pitch synchronously using instants of significant excitation. The performance of the voice conversion system is evaluated using listening tests. The prediction accuracy of the mapping functions (neural network models) used at different levels in the proposed voice conversion system is further evaluated using objective measures such as deviation (, root mean square error () and correlation coefficient (). The proposed approach (i.e., mapping and modification of parameters using pitch synchronous approach) used for voice conversion is shown to be performed better compared to the earlier method (mapping the vocal tract parameters using block processing) proposed by the author. [Copyright &y& Elsevier]

• Expository study on adversarial attacks and possible countermeasures for deep speaker recognition systems. • White box attacks: FGSM, PGD, Carlini and Wagner. • Defensive countermeasures: Adversarial training, adversarial Lipschitz regularization. • Several ablation studies e.g., varying strength of the attack, measuring signal-to-noise ratio and perceptibility, effect of noise augmentation, transferability analysis. • Strongest attacks: PGD, Carlini & Wagner; most imperceptible adversarial samples: Carlini & Wagner; best defense: PGD-based adversarial training. Robust speaker recognition, including in the presence of malicious attacks, is becoming increasingly important and essential, especially due to the proliferation of smart speakers and personal agents that interact with an individual's voice commands to perform diverse and even sensitive tasks. Adversarial attack is a recently revived domain which is shown to be effective in breaking deep neural network-based classifiers, specifically, by forcing them to change their posterior distribution by only perturbing the input samples by a very small amount. Although, significant progress in this realm has been made in the computer vision domain, advances within speaker recognition is still limited. We present an expository paper that considers several adversarial attacks to a deep speaker recognition system, employs strong defense methods as countermeasures, and reports a comprehensive set of ablation studies to better understand the problem. The experiments show that the speaker recognition systems are vulnerable to adversarial attacks, and the strongest attacks can reduce the accuracy of the system from 94% to even 0%. The study also compares the performances of the employed defense methods in detail, and finds adversarial training based on Projected Gradient Descent (PGD) to be the best defense method in our setting. We hope that the experiments presented in this paper provide baselines that can be useful for the research community interested in further studying adversarial robustness of speaker recognition systems. [ABSTRACT FROM AUTHOR]

• We compare sequence-to-sequence text-to-speech synthesis systems with pipeline text-to-speech synthesis systems. • We investigate sequence-to-sequence text-to-speech from three aspects: a) model architecture, b) model parameter size, and c) language (Japanese/English). • We describe sequence-to-sequence and pipeline text-to-speech synthesis in detail, and configure fairer experimental conditions. • Our Japanese experiment shows that increasing model parameter size enables to compensate for a lack of rich linguistic features. • Our English experiment shows that a suitable encoder structure enables to disambiguate pronunciations. Neural sequence-to-sequence text-to-speech synthesis (TTS) can produce high-quality speech directly from text or simple linguistic features such as phonemes. Unlike traditional pipeline TTS, the neural sequence-to-sequence TTS does not require manually annotated and complicated linguistic features such as part-of-speech tags and syntactic structures for system training. However, it must be carefully designed and well optimized so that it can implicitly extract useful linguistic features from the input features. In this paper we investigate under what conditions the neural sequence-to-sequence TTS can work well in Japanese and English along with comparisons with deep neural network (DNN) based pipeline TTS systems. Unlike past comparative studies, the pipeline systems also use neural autoregressive (AR) probabilistic modeling and a neural vocoder in the same way as the sequence-to-sequence systems do for a fair and deep analysis in this paper. We investigated systems from three aspects: a) model architecture, b) model parameter size, and c) language. For the model architecture aspect, we adopt modified Tacotron systems that we previously proposed and their variants using an encoder from Tacotron or Tacotron2. For the model parameter size aspect, we investigate two model parameter sizes. For the language aspect, we conduct listening tests in both Japanese and English to see if our findings can be generalized across languages. Our experiments on Japanese demonstrated that the Tacotron TTS systems with increased parameter size and input of phonemes and accentual type labels outperformed the DNN-based pipeline systems using the complicated linguistic features and that its encoder could learn to compensate for a lack of rich linguistic features. Our experiments on English demonstrated that, when using a suitable encoder, the Tacotron TTS system with characters as input can disambiguate pronunciations and produce natural speech as good as those of the systems using phonemes. However, we also found that the encoder could not learn English stressed syllables from characters perfectly and hence resulted in flatter fundamental frequency. In summary, these experimental results suggest that a) a neural sequence-to-sequence TTS system should have a sufficient number of model parameters to produce high quality speech, b) it should also use a powerful encoder when it takes characters as inputs, and c) the encoder still has a room for improvement and needs to have an improved architecture to learn supra-segmental features more appropriately. [ABSTRACT FROM AUTHOR]

Viewing dialogue management as a reinforcement learning task enables a system to learn to act optimally by maximising a reward function. This reward function is designed to induce the system behaviour required for the target application and for goal-oriented applications, this usually means fulfilling the user’s goal as efficiently as possible. However, in real-world spoken dialogue system applications, the reward is hard to measure because the user’s goal is frequently known only to the user. Of course, the system can ask the user if the goal has been satisfied but this can be intrusive. Furthermore, in practice, the accuracy of the user’s response has been found to be highly variable. This paper presents two approaches to tackling this problem. Firstly, a recurrent neural network is utilised as a task success predictor which is pre-trained from off-line data to estimate task success during subsequent on-line dialogue policy learning. Secondly, an on-line learning framework is described whereby a dialogue policy is jointly trained alongside a reward function modelled as a Gaussian process with active learning. This Gaussian process operates on a fixed dimension embedding which encodes each varying length dialogue. This dialogue embedding is generated in both a supervised and unsupervised fashion using different variants of a recurrent neural network. The experimental results demonstrate the effectiveness of both off-line and on-line methods. These methods enable practical on-line training of dialogue policies in real-world applications. [ABSTRACT FROM AUTHOR]

In the field of human speech capturing systems, a fundamental role is played by the source localization algorithms. In this paper a Speaker Localization algorithm (SLOC) based on Deep Neural Networks (DNN) is evaluated and compared with state-of-the art approaches. The speaker position in the room under analysis is directly determined by the DNN, leading the proposed algorithm to be fully data-driven. Two different neural network architectures are investigated: the Multi Layer Perceptron (MLP) and Convolutional Neural Networks (CNN). GCC-PHAT (Generalized Cross Correlation-PHAse Transform) Patterns, computed from the audio signals captured by the microphone are used as input features for the DNN. In particular, a multi-room case study is dealt with, where the acoustic scene of each room is influenced by sounds emitted in the other rooms. The algorithm is tested by means of the home recorded DIRHA dataset, characterized by multiple wall and ceiling microphone signals for each room. In detail, the focus goes to speaker localization task in two distinct neighboring rooms. As term of comparison, two algorithms proposed in literature for the addressed applicative context are evaluated, the Crosspower Spectrum Phase Speaker Localization (CSP-SLOC) and the Steered Response Power using the Phase Transform speaker localization (SRP-SLOC). Besides providing an extensive analysis of the proposed method, the article shows how DNN-based algorithm significantly outperforms the state-of-the-art approaches evaluated on the DIRHA dataset, providing an average localization error, expressed in terms of Root Mean Square Error (RMSE), equal to 324 mm and 367 mm, respectively, for the Simulated and the Real subsets. [ABSTRACT FROM AUTHOR]

The performance of state-of-the-art i-vector speaker verification systems relies on a large amount of training data for probabilistic linear discriminant analysis (PLDA) modeling. During the evaluation, it is also crucial that the target condition data is matched well with the development data used for PLDA training. However, in many practical scenarios, these systems have to be developed, and trained, using data that is often outside the domain of the intended application, since the collection of a significant amount of in-domain data is often difficult. Experimental studies have found that PLDA speaker verification performance degrades significantly due to this development/evaluation mismatch. This paper introduces a domain-invariant linear discriminant analysis (DI-LDA) technique for out-domain PLDA speaker verification that compensates domain mismatch in the LDA subspace. We also propose a domain-invariant probabilistic linear discriminant analysis (DI-PLDA) technique for domain mismatch modeling in the PLDA subspace, using only a small amount of in-domain data. In addition, we propose the sequential and score-level combination of DI-LDA, and DI-PLDA to further improve out-domain speaker verification performance. Experimental results show the proposed domain mismatch compensation techniques yield at least 27% and 14.5% improvement in equal error rate (EER) over a pooled PLDA system for telephone-telephone and interview-interview conditions, respectively. Finally, we show that the improvement over the baseline pooled system can be attained even when significantly reducing the number of in-domain speakers, down to 30 in most of the evaluation conditions. [ABSTRACT FROM AUTHOR]

In this paper an uncertainty weighting scheme for DNN–HMM-based speech recognition is proposed to increase discriminability in the decoding process. To this end, the DNN pseudo-log-likelihoods are weighted according to the uncertainty variance assigned to the acoustic observation. The results presented here suggest that substantial reduction in WER is achieved with clean training. Moreover, modelling the uncertainty propagation through the DNN is not required and no approximations for non-linear activation functions are made. The presented method can be applied to any network topology that delivers log-likelihood-like scores. It can be combined with any noise removal technique and adds a minimal computational cost. This technique was exhaustively evaluated and combined with uncertainty-propagation-based schemes for computing the pseudo-log-likelihoods and uncertainty variance at the DNN output. Two proposed methods optimized the parameters of the weighting function by leveraging the grid search either on a development database representing the given task or on each utterance based on discrimination metrics. Experiments with Aurora-4 task showed that, with clean training, the proposed weighting scheme can reduce WER by a maximum of 21% compared with a baseline system with spectral subtraction and uncertainty propagation using the unscented transform. The uncertainty weighting method reduced the gap between clean and multi-noise/multi-condition training. This can be useful when it is not easy to train a DNN–HMM system in conditions that are similar to the testing ones. Finally, the presented results on the use of uncertainty are very competitive with those published elsewhere using the same database as the one employed here. [ABSTRACT FROM AUTHOR]

In this paper we propose to exploit the automatic Quality Estimation (QE) of ASR hypotheses to perform the unsupervised adaptation of a deep neural network modeling acoustic probabilities. Our hypothesis is that significant improvements can be achieved by: i) automatically transcribing the evaluation data we are currently trying to recognise, and ii) selecting from it a subset of “good quality” instances based on the word error rate (WER) scores predicted by a QE component. To validate this hypothesis, we run several experiments on the evaluation data sets released for the CHiME-3 challenge. First, we operate in oracle conditions in which manual transcriptions of the evaluation data are available, thus allowing us to compute the true sentence WER. In this scenario, we perform the adaptation with variable amounts of data, which are characterised by different levels of quality. Then, we move to realistic conditions in which the manual transcriptions of the evaluation data are not available. In this case, the adaptation is performed on data selected according to the WER scores predicted by a QE component. Our results indicate that: i) QE predictions allow us to closely approximate the adaptation results obtained in oracle conditions, and ii) the overall ASR performance based on the proposed QE-driven adaptation method is significantly better than the strong, most recent, CHiME-3 baseline. [ABSTRACT FROM AUTHOR]

Multilingual training of neural networks has proven to be simple yet effective way to deal with multilingual training corpora. It allows to use several resources to jointly train a language independent representation of features, which can be encoded into low-dimensional feature set by embedding narrow bottleneck layer to the network. In this paper, we analyze such features on the task of spoken language recognition (SLR), focusing on practical aspects of training bottleneck networks and analyzing their integration in SLR. By comparing properties of mono and multilingual features we show the suitability of multilingual training for SLR. The state-of-the-art performance of these features is demonstrated on the NIST LRE09 database. [ABSTRACT FROM AUTHOR]

Natural Language Inference is an important task for Natural Language Understanding. It is concerned with classifying the logical relation between two sentences. In this paper, we propose several text generative neural networks for generating text hypothesis, which allows construction of new Natural Language Inference datasets. To evaluate the models, we propose a new metric—the accuracy of the classifier trained on the generated dataset. The accuracy obtained by our best generative model is only 2.7% lower than the accuracy of the classifier trained on the original, human crafted dataset. Furthermore, the best generated dataset combined with the original dataset achieves the highest accuracy. The best model learns a mapping embedding for each training example. By comparing various metrics we show that datasets that obtain higher ROUGE or METEOR scores do not necessarily yield higher classification accuracies. We also provide analysis of what are the characteristics of a good dataset including the distinguishability of the generated datasets from the original one. [ABSTRACT FROM AUTHOR]

The paper describes a system for automatic speech recognition (ASR) that is benchmarked with data of the 3rd CHiME challenge, a dataset comprising distant microphone recordings of noisy acoustic scenes in public environments. The proposed ASR system employs various methods to increase recognition accuracy and noise robustness. Two different multi-channel speech enhancement techniques are used to eliminate interfering sounds in the audio stream. One speech enhancement method aims at separating the target speaker's voice from background sources based on non-negative matrix factorization (NMF) using variational Bayesian (VB) inference to estimate NMF parameters. The second technique is based on a time-varying minimum variance distortionless response (MVDR) beamformer that uses spatial information to suppress sound signals not arriving from a desired direction. Prior to speech enhancement, a microphone channel failure detector is applied that is based on cross-comparing channels using a modulation-spectral representation of the speech signal. ASR feature extraction employs the amplitude modulation filter bank (AMFB) that implicates prior information of speech to analyze its temporal dynamics. AMFBs outperform the commonly used frame splicing technique of filter bank features in conjunction with a deep neural network (DNN) based ASR system, which denotes an equivalent data-driven approach to extract modulation-spectral information. In addition, features are speaker adapted, a recurrent neural network (RNN) is employed for language modeling, and hypotheses of different ASR systems are combined to further enhance the recognition accuracy. The proposed ASR system achieves an absolute word error rate (WER) of 5.67% on the real evaluation test data, which is 0.16% lower compared to the best score reported within the 3rd CHiME challenge. [ABSTRACT FROM AUTHOR]

Inspired by the success of Deep Neural Networks (DNN) in text-independent speaker recognition, we have recently demonstrated that similar ideas can also be applied to the text-dependent speaker verification task. In this paper, we describe new advances with our state-of-the-art i-vector based approach to text-dependent speaker verification, which also makes use of different DNN techniques. In order to collect sufficient statistics for i-vector extraction, different frame alignment models are compared such as GMMs, phonemic HMMs or DNNs trained for senone classification. We also experiment with DNN based bottleneck features and their combinations with standard MFCC features. We experiment with few different DNN configurations and investigate the importance of training DNNs on 16 kHz speech. The results are reported on RSR2015 dataset, where training material is available for all possible enrollment and test phrases. Additionally, we report results also on more challenging RedDots dataset, where the system is built in truly phrase-independent way. [ABSTRACT FROM AUTHOR]

This paper investigates the problem of measuring speech quality of received speech signal without employing the original speech signal. The problem of deterioration of the speech quality arises due to noise present in the surroundings. To this line, we propose a multiple time-instances (MTI) features-based approach for reference-free speech quality measurement model. A voice activity detector (VAD) is exploited primarily for calculating the number of active speech chunks of a speech signal. For these chunks and their successive combinations called here batches, multi-resolution auditory model (MRAM), mel-frequency cepstral coefficients (MFCC) and line spectral frequencies (LSF) features are extracted and called as MTI features. It is hypothesized that the MTI features are capable in capturing the distortions caused by time-localized effects of short-time transients, impulsive noise, and its differences from the plosive sounds. The MTI metric estimates (MTI-ME) are calculated corresponding to these MTI features employing the Gaussian mixture model (GMM) probabilistic technique. The overall objective speech quality of a speech signal is then determined as a linear combination of optimally weighted MTI-ME corresponding to distinct active speech chunks and their successive combinations, that is, batches of that speech signal. Minimum mean square error criterion or Pearson's correlation maximization criterion is employed for computing optimal weights. In addition, a deep neural network (DNN)-based speech quality model is also developed for calculating a single objective speech quality while considering all active speech chunks together. Pearson's correlation coefficient and weighted average correlation are exploited for evaluating the performance. Results demonstrate that the proposed model achieves promising improvement over the standard speech quality model (P.563) and improves correlation values by around 37%. • Addressing speech quality measuring problem by developing a reference-free speech quality model. • Developing a feature extraction technique incorporating distinct auditory features. • Developing a joint GMM training algorithm for computing objective speech quality. • Developing a deep neural network (DNN) framework for validating the proposed algorithm. • Demonstrating better performance as compared to the standard speech quality model (P.563). [ABSTRACT FROM AUTHOR]

We first observe a potential weakness of continuous vector representations of symbols in neural machine translation. That is, the continuous vector representation, or a word embedding vector, of a symbol encodes multiple dimensions of similarity, equivalent to encoding more than one meaning of the word. This has the consequence that the encoder and decoder recurrent networks in neural machine translation need to spend substantial amount of their capacity in disambiguating source and target words based on the context which is defined by a source sentence. Based on this observation, in this paper we propose to contextualize the word embedding vectors using a nonlinear bag-of-words representation of the source sentence. Additionally, we propose to represent special tokens (such as numbers, proper nouns and acronyms) with typed symbols to facilitate translating those words that are not well-suited to be translated via continuous vectors. The experiments on En–Fr and En–De reveal that the proposed approaches of contextualization and symbolization improves the translation quality of neural machine translation systems significantly. [ABSTRACT FROM AUTHOR]

This paper investigates the use of multi-distribution deep neural networks (MD-DNNs) for automatic intonation classification in second-language (L2) English speech. If a classified intonation is different from the target one, we consider that mispronunciation is detected and appropriate diagnostic feedback can be provided thereafter. To transcribe speech data for intonation classification, we propose the RULF labels which are used to transcribe an intonation as rising, upper, lower or falling . These four types of labels can be further merged into two groups – rising and falling . Based on the annotated data from 100 Mandarin and 100 Cantonese learners, we develop an intonation classifier, which considers only 8 frames (i.e., 80 ms) of pitch value prior to the end of the pitch contour over an intonational phrase (IP). This classifier determines the intonation of L2 English speech as either rising or falling with an accuracy of 93.0%. [ABSTRACT FROM AUTHOR]

This paper presents a new domain compensation framework by using phonetically discriminative features which are extracted from domain-dependent deep neural networks (DNNs). The domain compensation can be applied in both unsupervised and supervised manner, depending on whether the domain information of the development data is provided or not in advance. In supervised manner, the DNNs are trained on the development speech recordings of each given domain separately. While in the unsupervised manner, the development datasets are first automatically clustered into different domains, by using the Gaussian Mixture Model mean supervectors which are generated from each of the speech recordings, DNNs are then trained on the resulting clusters. Finally, we compensate the domain variabilities during the target speaker modeling step using support vector machines, by feeding in statistical vectors which are derived from the discriminative features extracted from the domain-dependent DNNs. The main strength of our proposed framework is that it does not need any speaker labels in the development dataset, which makes the proposed framework of great advantage over the state-of-the-art techniques that need speaker labels to train inter-speaker and/or intra-speaker variability models or channel compensation. Three speaker verification systems are investigated to examine the effectiveness of this new framework. Experimental results on the NIST SRE 2010 task demonstrate competitive performances to the state-of-the-art techniques in an initial implementation of the proposed framework. [ABSTRACT FROM AUTHOR]

As the volume of recorded conversations continues to surge, so does the need for their automatic processing. Plenty of information beyond words may be extracted from the speech signal that could be valuable in domains such as call-center quality assurance. In particular, describing the dynamics of turn-taking exchanges allows for a deeper understanding of the development and outcome of a dialogue. In this paper, we investigate the construction of an automatic turn-taking annotation tool based on recordings of entire conversations (in offline mode) — an unexplored topic to our knowledge. We experiment with two supervised learning approaches, using recurrent neural networks and random forests, on a corpus of Argentine Spanish task-oriented dialogues annotated with 12 turn-taking categories following standard guidelines. Our models achieve promising results, with F1 scores ranging 0.7–0.9 for the most frequent labels (e.g., smooth switches, backchannels), but much lower for the least frequent ones (various kinds of interruptions), for which further research is needed. We also evaluate our best-performing models considering their generalizability in scenarios of growing difficulty, including dialogues in two different languages (English and Slovak). Finally, to address the typical data scarcity issue, we analyze the impact of combining training data from different corpora, again including cross-linguistic data. • Machine learning (ML) models for offline annotation of turn-taking exchanges in entire dialogues. • Full taxonomy of turn-taking transitions, including smooth switches, interruptions, and backchannels. • Comparison of two ML approaches: random forests, and recurrent neural model (with a novel loss function strategy). • Generalizability assessment of our best models in scenarios of growing difficulty in Spanish. • Cross-linguistic evaluation in English and Slovak dialogues, as a way of dealing with labeled data scarcity. [ABSTRACT FROM AUTHOR]

The aim of this paper is to investigate the benefit of combining both language and acoustic modelling for speaker diarization. Although conventional systems only use acoustic features, in some scenarios linguistic data contain high discriminative speaker information, even more reliable than the acoustic ones. In this study we analyze how an appropriate fusion of both kind of features is able to obtain good results in these cases. The proposed system is based on an iterative algorithm where a LSTM network is used as a speaker classifier. The network is fed with character-level word embeddings and a GMM based acoustic score created with the output labels from previous iterations. The presented algorithm has been evaluated in a Call-Center database, which is composed of telephone interview audios. The combination of acoustic features and linguistic content shows a 84.29% improvement in terms of a word-level DER as compared to a HMM/VB baseline system. The results of this study confirms that linguistic content can be efficiently used for some speaker recognition tasks. • Linguistic content can be efficiently combined with acoustic features for the speaker diarization task. • Given a specific scenario, speaker diarization can be solved with only linguistic content. • Acoustic features lead to a better diarizatcon performance in comparison with linguistic content in large speaker segments. • Linguistic content is more discriminative than acoustic features to identify speakers in short speech segments. [ABSTRACT FROM AUTHOR]

Bidirectional long short-term memory (Bi-LSTM), as one of the effective networks for sequence labeling tasks, is widely used in named entity recognition (NER). However, the sequential nature of Bi-LSTM and the inability to recognize multiple sentences at the same time make it impossible to obtain overall information. In this paper, to make up for the shortcomings of Bi-LSTM in extracting global information, we propose a hierarchical context model embedded with sentence-level and document-level feature extraction. In sentence-level feature extraction, we use the self-attention mechanism to extract sentence-level representations considering the different contribution of each word to the sentence. For document-level feature extraction, 3D convolutional neural network (CNN), which not only can extract features within sentences, but also pays attention to the sequential relationship between sentences, is used to extract document-level representations. Furthermore, we investigate a layer-by-layer residual (LBL Residual) structure to optimize each Bi-LSTM block of our model, which can solve the degradation problem that appears as the number of model layers increases. Experiments show that our model achieves results competitive with the state-of-the-art records on the CONLL-2003 and Ontonotes5.0 English datasets respectively. [ABSTRACT FROM AUTHOR]

• An ESIM model based system ranked top 1 on both datasets of DSTC7 noetic end-to-end response selection track. • An accurate and efficient two-step approach for response selection from a large amount of candidates. • Heuristic data augmentation and optimized sampling approaches improved the ESIM model performance. • Training task-specific word embeddings incorporating external domain knowledge for the ESIM model. • An effective and efficient approach for ensembling models trained with different parameter initializations and structures. • Extensive ablation analyses for various factors contributing to the ESIM model performance. • Systematic comparisons between ESIM and BERT models for response selection and ablation analyses for the BERT model. The noetic end-to-end response selection challenge as one track in the 7th Dialog System Technology Challenges (DSTC7) aims to push the state of the art of utterance classification for real world goal-oriented dialog systems, for which participants need to select the correct next utterances from a set of candidates for the multi-turn context. This paper presents our systems that are ranked top 1 on both datasets under this challenge, one focused and small (Advising) and the other more diverse and large (Ubuntu). Previous state-of-the-art models use hierarchy-based (utterance-level and token-level) neural networks to explicitly model the interactions among different turns' utterances for context modeling. In this paper, we investigate a sequential matching model based only on chain sequence for multi-turn response selection. Our results demonstrate that the potentials of sequential matching approaches have not yet been fully exploited in the past for multi-turn response selection. In addition to ranking top 1 in the challenge, the proposed model outperforms all previous models, including state-of-the-art hierarchy-based models, on two large-scale public multi-turn response selection benchmark datasets. [ABSTRACT FROM AUTHOR]

The deep learning methodologies in state-of-the-art speaker recognition systems are predominantly limited to the extraction of recording level embeddings. This is usually followed by generative modeling of the embeddings to output the verification score. In this paper, we explore a fully neural approach where the neural model outputs the verification score directly, given the acoustic feature inputs. This model, termed as Siamese neural network (SiamNN), combines the embedding extraction and back-end modeling into a single processing pipeline. The back-end modeling is achieved using a neural approach to PLDA modeling, called neural probabilistic linear discriminant analysis (NPLDA). In the NPLDA model, the verification score is computed as a discriminative similarity function. The development of the single neural SiamNN model allows the joint optimization of all the modules using a verification cost. Several speaker recognition experiments are performed using SITW, VOiCES, and NIST SRE datasets where the proposed SiamNN model is shown to significantly improve over the state-of-art x-vector PLDA baseline system (relative improvements of up to 35% in the primary cost metric). We also provide a detailed analysis of the influence of hyper-parameters, choice of loss functions, and data sampling strategies for training the model. In particular, we highlight that the proposed soft detection cost function based optimization improves over other loss functions considered. [ABSTRACT FROM AUTHOR]

In this paper, we investigate the ensemble of deep neural networks (DNNs) by using an acoustic environment classification (AEC) technique for the statistical model-based voice activity detection (VAD). From an investigation of the statistical model-based VAD, it is known that the traditional decision rule is based on the geometric mean of the likelihood ratio or the support vector machine (SVM), which is a shallow model with zero or one hidden layer. Since the shallow models cannot take an advantage of the diversity of the space distribution of features, in the training step, we basically build the multiple DNNs according the different noise types by employing the parameters of the statistical model-based VAD algorithm. In addition, the separate DNN is designed for the AEC algorithm in order to choose the best DNN for each noise. In the on-line noise-aware VAD step, the AEC is first performed on a frame-by-frame basis using the separate DNN so the a posteriori probabilities to identify noise are obtained. Once the probabilities are achieved for each noise, the environmental knowledge is contributed to allow us to combine the speech presence probabilities which are derived from the ensemble of the DNNs trained for the individual noise. Our approach for VAD was evaluated in terms of objective measures and showed significant improvement compared to the conventional algorithm. [ABSTRACT FROM AUTHOR]