Search

Word Error Rate (WER) has been the predominant metric used to evaluate the performance of automatic speech recognition (ASR) systems. However, WER is sometimes not a good indicator for downstream Natural Language Understanding (NLU) tasks, such as intent recognition, slot filling, and semantic parsing in task-oriented dialog systems. This is because WER takes into consideration only literal correctness instead of semantic correctness, the latter of which is typically more important for these downstream tasks. In this study, we propose a novel Semantic Distance (SemDist) measure as an alternative evaluation metric for ASR systems to address this issue. We define SemDist as the distance between a reference and hypothesis pair in a sentence-level embedding space. To represent the reference and hypothesis as a sentence embedding, we exploit RoBERTa, a state-of-the-art pre-trained deep contextualized language model based on the transformer architecture. We demonstrate the effectiveness of our proposed metric on various downstream tasks, including intent recognition, semantic parsing, and named entity recognition., Comment: submitted to Interspeech 2021

How to leverage dynamic contextual information in end-to-end speech recognition has remained an active research area. Previous solutions to this problem were either designed for specialized use cases that did not generalize well to open-domain scenarios, did not scale to large biasing lists, or underperformed on rare long-tail words. We address these limitations by proposing a novel solution that combines shallow fusion, trie-based deep biasing, and neural network language model contextualization. These techniques result in significant 19.5% relative Word Error Rate improvement over existing contextual biasing approaches and 5.4%-9.3% improvement compared to a strong hybrid baseline on both open-domain and constrained contextualization tasks, where the targets consist of mostly rare long-tail words. Our final system remains lightweight and modular, allowing for quick modification without model re-training., Comment: Accepted for presentation at INTERSPEECH 2021

End-to-end automatic speech recognition (ASR) models with a single neural network have recently demonstrated state-of-the-art results compared to conventional hybrid speech recognizers. Specifically, recurrent neural network transducer (RNN-T) has shown competitive ASR performance on various benchmarks. In this work, we examine ways in which RNN-T can achieve better ASR accuracy via performing auxiliary tasks. We propose (i) using the same auxiliary task as primary RNN-T ASR task, and (ii) performing context-dependent graphemic state prediction as in conventional hybrid modeling. In transcribing social media videos with varying training data size, we first evaluate the streaming ASR performance on three languages: Romanian, Turkish and German. We find that both proposed methods provide consistent improvements. Next, we observe that both auxiliary tasks demonstrate efficacy in learning deep transformer encoders for RNN-T criterion, thus achieving competitive results - 2.0%/4.2% WER on LibriSpeech test-clean/other - as compared to prior top performing models., Accepted for publication at IEEE Spoken Language Technology Workshop (SLT), 2021

Recurrent Neural Network Transducer (RNN-T), like most end-to-end speech recognition model architectures, has an implicit neural network language model (NNLM) and cannot easily leverage unpaired text data during training. Previous work has proposed various fusion methods to incorporate external NNLMs into end-to-end ASR to address this weakness. In this paper, we propose extensions to these techniques that allow RNN-T to exploit external NNLMs during both training and inference time, resulting in 13-18% relative Word Error Rate improvement on Librispeech compared to strong baselines. Furthermore, our methods do not incur extra algorithmic latency and allow for flexible plug-and-play of different NNLMs without re-training. We also share in-depth analysis to better understand the benefits of the different NNLM fusion methods. Our work provides a reliable technique for leveraging unpaired text data to significantly improve RNN-T while keeping the system streamable, flexible, and lightweight., submitted to ICASSP 2021

OBJECTIVE This study aimed to examine the degree of clinical and functional improvement after paliperidone long-acting injectable (LAI) administration according to the duration of illness. METHODS Patients with schizophrenia diagnosed by ICD-10 criteria who were planned to start once-monthly paliperidone LAI were recruited from 2010 to 2017. Clinical and functional changes were measured every 4 weeks using the Clinical Global Impressions-Severity of Illness scale (CGI-S) and Personal and Social Performance scale (PSP), respectively, for 6 months after paliperidone LAI initiation. Improvements after starting paliperidone LAI were compared among patients with duration of illness < 3 years, ≥ 3 and < 10 years, and ≥ 10 years. RESULTS A total of 1,166 participants (duration of illness < 3 years, n = 240; 3 ≤ duration of illness < 10 years, n = 442; duration of illness ≥ 10 years, n = 484) were enrolled. The total olanzapine-equivalent doses of antipsychotics and the LAI monotherapy proportion at the final visit were significantly different among the 3 duration of illness groups (dose: F₂,₁₁₆₃ = 18.41, P < .001; monotherapy: χ²₂ = 11.73, P = .003). The changes in CGI-S score were significantly different according to the duration of illness, and those with duration of illness < 3 years showed the best improvement (group × week: χ²₁₂ = 25.33, P = .013). All 3 groups showed significantly improved PSP scores (week: χ²₆ = 294.2, P < .001). CONCLUSIONS Starting paliperidone LAI significantly improved clinical and functional outcomes in patients with schizophrenia, especially those with shorter duration of illness. These findings suggest that LAI antipsychotic administration may be considered in early-stage schizophrenia for improved outcomes.

Processing of conversations is a core technique in conversational AI. However, current speech recognition solutions, even the state-of-the-art systems, model a single, isolated utterance, not an entire conversation. These systems are therefore unable to use potentially important contextual information that spans across multiple utterances or speakers in a conversation. This thesis focuseson designing an End-to-End speech recognition system that processes entire conversations. To achieve this goal, I propose three novel techniques: 1) an efficient way to preserve long conversational contexts by creating a context encoder that maps spoken utterance histories to a singlecontext vector; 2) an effective way to integrate conversational contexts into End-to-End modelsusing a gating mechanism; and 3) various methods to encode conversational contexts by using previously spoken utterances and augmenting with world knowledge using external linguistic resources (e.g. BERT, fastText). I show accuracy improvements with three different large corpora,Switchboard (300 hours), Fisher (2,000 hours), and Medical conversation (1,700 hours), and share the analysis to demonstrate the effectiveness of my approach. This thesis will provide insight into designing conversational speech recognition systems and spoken language understandingsystems, which are becoming increasingly important as voice-driven device interfaces become mainstream.

Conventional automatic speech recognition (ASR) based on a hidden Markov model (HMM)/deep neural network (DNN) is a very complicated system consisting of various modules such as acoustic, lexicon, and language models. It also requires linguistic resources, such as a pronunciation dictionary, tokenization, and phonetic context-dependency trees. On the other hand, end-to-end ASR has become a popular alternative to greatly simplify the model-building process of conventional ASR systems by representing complicated modules with a single deep network architecture, and by replacing the use of linguistic resources with a data-driven learning method. There are two major types of end-to-end architectures for ASR; attention-based methods use an attention mechanism to perform alignment between acoustic frames and recognized symbols, and connectionist temporal classification (CTC) uses Markov assumptions to efficiently solve sequential problems by dynamic programming. This paper proposes hybrid CTC/attention end-to-end ASR, which effectively utilizes the advantages of both architectures in training and decoding. During training, we employ the multiobjective learning framework to improve robustness and achieve fast convergence. During decoding, we perform joint decoding by combining both attention-based and CTC scores in a one-pass beam search algorithm to further eliminate irregular alignments. Experiments with English (WSJ and CHiME-4) tasks demonstrate the effectiveness of the proposed multiobjective learning over both the CTC and attention-based encoder–decoder baselines. Moreover, the proposed method is applied to two large-scale ASR benchmarks (spontaneous Japanese and Mandarin Chinese), and exhibits performance that is comparable to conventional DNN/HMM ASR systems based on the advantages of both multiobjective learning and joint decoding without linguistic resources.

Objective:The present study analysed and compared the nutritional qualities of private brand (PB) and national brand (NB) ready meal (RM) products in South Korea.Design:Data on RM products were collected via a food information database (EatSight) between February and May of 2017. A total of ninety-one PB products from a leading discount store chain in South Korea and 415 NB products were selected and divided into four main categories of ‘refrigerated soups/stews’, ‘refrigerated noodles’, ‘frozen rice/noodles’ and ‘frozen dumplings’. Nutritional information was extracted for each product and analyses were performed to compare nutritional differences.Setting:South Korea.Participants:A total of 506 RM products.Results:PB products explained the highest percentage of refrigerated RM products while NB products occupied the highest percentage of frozen ones. Notably, main PB and NB products showed evident nutritional problems. Among PB products, refrigerated soups/stews were significantly higher in energy, fat, Na, saturated fat and trans-fat. Likewise, in the case of NB products, frozen rice/noodles were significantly higher in Na and saturated fat. Particularly, overall Na contents were high. When the Korean guidance on traffic-light labelling was applied, a mere 0·2 and 5·9 % of total RM products showed all four and three green lights, respectively. Meanwhile, those with no green lights accounted for 42·3 %.Conclusions:Although it is difficult to draw a definitive conclusion on the comparative nutritional qualities due to the lack of consistency in the results, the study suggests that there is room to improve the nutritional qualities of overall RM products.

We present a novel conversational-context aware end-to-end speech recognizer based on a gated neural network that incorporates conversational-context/word/speech embeddings. Unlike conventional speech recognition models, our model learns longer conversational-context information that spans across sentences and is consequently better at recognizing long conversations. Specifically, we propose to use the text-based external word and/or sentence embeddings (i.e., fastText, BERT) within an end-to-end framework, yielding a significant improvement in word error rate with better conversational-context representation. We evaluated the models on the Switchboard conversational speech corpus and show that our model outperforms standard end-to-end speech recognition models., ACL 2019

Existing speech recognition systems are typically built at the sentence level, although it is known that dialog context, e.g. higher-level knowledge that spans across sentences or speakers, can help the processing of long conversations. The recent progress in end-to-end speech recognition systems promises to integrate all available information (e.g. acoustic, language resources) into a single model, which is then jointly optimized. It seems natural that such dialog context information should thus also be integrated into the end-to-end models to improve further recognition accuracy. In this work, we present a dialog-context aware speech recognition model, which explicitly uses context information beyond sentence-level information, in an end-to-end fashion. Our dialog-context model captures a history of sentence-level context so that the whole system can be trained with dialog-context information in an end-to-end manner. We evaluate our proposed approach on the Switchboard conversational speech corpus and show that our system outperforms a comparable sentence-level end-to-end speech recognition system., Comment: submitted to SLT