Search

In recent years, the research pertaining to speech recognition technology in the Kazakh language has gained significant importance. This is due to the increasing demand for natural language processing applications in the region where Kazakh is predominantly spoken. Thus, there exists an urgent requirement for precise and dependable speech recognition systems. The research study examines the application of sophisticated deep learning methodologies, such as Natural Language Processing (NLP) and Hidden Markov Model (HMM), in facilitating speech recognition for the Kazakh language. Additionally, the investigation delves into how various techniques, including data preprocessing, acoustic modeling, and language modeling, can aid in devising effective speech recognition systems. The article deliberates on the feasible uses of speech recognition technology in the geographic area where Kazakh language is spoken and outlines its future research prospects. The investigation underscores the significance of persistent inquiry in this realm to confront distinctive obstacles encountered in creating speech recognition systems for languages with restricted resources. [ABSTRACT FROM AUTHOR]

With increasing multimedia content and social activities, moderation problems increase. There are different approaches to moderation and automation. However, they have limitations in terms of usage in real-time. The analysis of scientific papers revealed that most of the more common approaches solve the task of detection instead of prediction by considering the final utterance. For this reason, calls are unprotected in toxic languages, and online broadcasts can be unpredictable. In this work, a new way for automatic speech moderation in terms of dynamic word prediction was suggested. The considered task involves the analysis of the auditory and textual channels of speech. Words can have different meanings depending on the context, so in solving the problem it is planned to consider profanity, which is socially unacceptable regardless of the context. In this paper approaches for working with speech stream in the task of profanity prediction were proposed. It can be possible to have smaller latency with usage of audio features. We also suggest the pipeline for real-time (with the ability to predict the sequence with a higher duration than the latency of the processing) prediction for multimodal prediction, which compensates the latency of ASR systems. As a result, in this paper, we compared different solutions for the next color prediction task for English speech and reached the F1 score of 86.6 for 3 class prediction. [ABSTRACT FROM AUTHOR]

Human-Machine Interfaces (HMIs) are getting more and more important in a hyper-connected society. Traditional HMIs are built considering cognitive features while emotional ones are often neglected, bringing sometimes such interfaces to misuse. As a part of a long run research, oriented to the definition of an HMI engineering approach, this paper concretely proposes a method to build an emotional-aware explicit model of the user starting from the behaviour of the human with a virtual agent. The paper also proposes an instance of this model inference process in voice assistants in an automatic depression context, which can constitute the core phase to realize a Human Digital Twin of a patient. The case study generated a model composed of Fluid Stochastic Petri Net sub-models, achieved after the data analysis by a Support Vector Machine. [ABSTRACT FROM AUTHOR]

This paper introduces a sign language translation system designed to bridge the communication gap between Hard of Hearing (HOH) or mute individuals, and address challenges in job opportunities and mental health. The system offers bi-directional translation of sign language and speech/text. For sign language to speech/text translation, the system utilizes either camera-based gestures, or data-glove inputs. The reverse translation is facilitated through a 3D avatar demonstrating the corresponding sign language for the input speech or text. Our findings demonstrate the system's translation accuracy and its potential to empower HOH and mute individuals in various communication scenarios. [ABSTRACT FROM AUTHOR]

Hate speech pertains to the expression of negative emotions by people. The spread of hate speech contributes to societal inequity by imposing adverse consequences on individuals' emotional and physical well-being, financial circumstances, and overall quality of life, thereby hindering their daily existence. Hence, the development of a hate detection and diffusion prediction system is of utmost importance. The literature study reveals two primary findings regarding the development of a hate speech model. Firstly, it is crucial to understand the potential impact of a hate speech message, specifically the extent to which it influences the users exposed to it. Secondly, numerous researchers have focused on studying information propagation, with a particular emphasis on analysing network topology and structure. This paper introduces psycho-sociological analysis as another approach to understanding hate speech spread on social networks. We develop a hate speech diffusion model considering many psycho-sociological dimensions of social media users. This paper aims to answer a few basic questions: (a) Who starts hateful tweets on social media? (b) Who are the users who share, respond to, or like such posts? (c) Can we build a better hate-speech diffusion model by understanding each user's psycho-sociological traits towards hateful information? To answer the above questions, we analyse user psycho-sociological traits by studying three different dimensions of user behaviour, i.e., inner personality, societal personality, and dark emotion using the Big5 personality model, Schwartz value model, and the Dark triad behaviour model. We develop a collection of classifiers to detect users' personality traits. In addition, we proposed a system to predict the spread of hate content on social networks. Several empirical analyses are presented in this paper in order to understand the correlation between human personality and online hateful behaviour. From our empirical analysis of hostile behaviour, several key observations are reported in this paper, a few of which include the following: (i) People with dark triad orientations are mostly the originators of online hate speech, (ii) Those who post sexist tweets are often Machiavellian, (iii) Narcissistic users are more likely to make racist tweets, and (iv) Those who post sexist and racist tweets on social networks are often psychopathic. Using these insights, we have mapped the user behaviour dimension into a vector space called behaviour embedding. This space is used for developing the hate-speech diffusion model. We show that the proposed models also perform better than the state-of-the-art with a significant F 1 -score margin. • Hate spreaders can be identified by DL model using physiological features. • Dark triad traits are linked to the hate user and hate post spreader. • Deep CNN network to detect hate users on social networks. • Diffusion system predicts who will spread the hate post on social media. [ABSTRACT FROM AUTHOR]

Speech Emotion Recognition (SER) is a very interesting task that allows the machine to identify and recognize the different emotional states from human speech using new technologies. The SER can be represented by two main steps, namely feature extraction and emotion classification. Our contribution to the SER field will focus on these two phases. This paper seeks to investigate the influence of embedded features in Wav2vec2 and HuBERT models on SER, two variants per module are implemented, including Wav2vec2 base, Wav2vec2 large, HuBERT large and HuBERT X-large. In addition, we adopt a linear Support Vector Machine (SVM) as a downstream model to recognize emotions. The proposed approach relying on the combination of HuBERT X-large features with the SVM model led to the highest recognition rate of 82.6% on the RAVDESS database. Moreover, the results obtained are promising and in compliance with the current SER state-of-the-art. Hence, the embedded features of HuBERT X-large model have shown significant results for SER. [ABSTRACT FROM AUTHOR]

This paper conducts comparative analysis of speaker identification techniques, namely Gaussian Mixture Models (GMM), Bidirectional Long Short-Term Memory (BiLSTM) neural networks, and Recurrent Neural Networks (RNN), applied to the Kazakh Speech Corpus. The study's central objective is to explore the influence of speech duration on the efficacy of speaker identification. Our results indicate that all models exhibit enhanced accuracy with increased speech duration. Furthermore, the incorporation of delta and fundamental frequency (F0) as supplementary features bolsters the performance of these models. This research contributes to the understanding of how speech duration and additional features can optimize speaker identification techniques, providing valuable insights for future developments in the field. [ABSTRACT FROM AUTHOR]

Expert testimony is only admissible in common-law systems if it will potentially assist the trier of fact. In order for a forensic-voice-comparison expert's testimony to assist a trier of fact, the expert's forensic voice comparison should be more accurate than the trier of fact's speaker identification. "Speaker identification in courtroom contexts – Part I" addressed the question of whether speaker identification by an individual lay listener (such as a judge) would be more or less accurate than the output of a forensic-voice-comparison system that is based on state-of-the-art automatic-speaker-recognition technology. The present paper addresses the question of whether speaker identification by a group of collaborating lay listeners (such as a jury) would be more or less accurate than the output of such a forensic-voice-comparison system. As members of collaborating groups, participants listen to pairs of recordings reflecting the conditions of the questioned- and known-speaker recordings in an actual case, confer, and make a probabilistic consensus judgement on each pair of recordings. The present paper also compares group-consensus responses with "wisdom of the crowd" which uses the average of the responses from multiple independent individual listeners. • Forensic voice comparison based on automatic-speaker-recognition technology. • Speaker identification by groups of collaborating lay listeners. • "Wisdom of the crowd" from independent individual-listener responses. • The forensic-voice-comparison system outperformed most groups of listeners. [ABSTRACT FROM AUTHOR]

Hate Speech Detection (HSD) aims to identify whether a text contains hate speech content, which often refers to discrimination and is even associated with a hate crime. The mainstream methods jointly train the HSD problem with relevant auxiliary problems, e.g., emotion detection and sentiment analysis, under the paradigm of Multi-Task Learning (MTL). In this paper, we improve HSD by integrating it with emotion detection, since we take inspiration from the potential correlations between hate speech and certain negative emotion states, which have been studied theoretically and empirically. To be specific, we can concatenate their hateful labels and predicted emotion states as pseudo-multiple labels for hate speech samples, formulating a pseudo-Multi-Label Learning (MLL) problem. Beyond the existing MTL-HSD methods, we further incorporate this pseudo-MLL problem and solve it by capturing the correlations between hate speech and negative emotion states, so as to improve the performance of HSD. Based on these ideas, we propose a novel HSD method named the E motion-correlated H ate S peech Detect OR (EHS or). We conduct extensive experiments to evaluate EHS or , and the results show that it can consistently outperform the existing HSD methods across benchmark datasets. • Validating correlations between hate speech and specific negative emotions. • Proposing a new MTL-based HSD method EHSOR from the perspective of MLL. • EHSOR empirically outperforms baseline methods over three HSD benchmarks. [ABSTRACT FROM AUTHOR]

Speech Emotion Recognition (SER) is an important area of research in speech processing that aims to identify and classify emotional states conveyed through speech signals. Recent studies have shown considerable performance in SER by exploiting deep contextualized speech representations from self-supervised learning (SSL) models. However, SSL models pre-trained on clean speech data may not perform well on emotional speech data due to the domain shift problem. To address this problem, this paper proposes a novel approach that simultaneously exploits an SSL model and a domain-agnostic spectral feature (SF) through the Mixture of Experts (MoE) technique. The proposed approach achieves the state-of-the-art performance on weighted accuracy compared to other methods in the IEMOCAP dataset. Moreover, this paper demonstrates the existence of the domain shift problem of SSL models in the SER task. [ABSTRACT FROM AUTHOR]

Hypokinetic dysarthria, a motor speech disorder characterized by reduced movement and control in the speech-related muscles, is mostly associated with Parkinson's disease. Acoustic speech features thus offer the potential for early digital biomarkers to diagnose and monitor the progression of this disease. However, the influence of language on the successful classification of healthy and dysarthric speech remains crucial. This paper explores the analysis of acoustic speech features, both established and newly proposed, in a multilingual context to support the diagnosis of PD. The study aims to identify language-independent and highly discriminative digital speech biomarkers using statistical analysis and machine learning techniques. The study analyzes thirty-three acoustic features extracted from Czech, American, Israeli, Columbian, and Italian PD patients, as well as healthy controls. The analysis employs correlation and statistical tests, descriptive statistics, and the XGBoost classifier. Feature importances and Shapley values are used to provide explanations for the classification results. The study reveals that the most discriminative features, with reduced language dependence, are those measuring the prominence of the second formant, monopitch, and the frequency of pauses during text reading. Classification accuracies range from 67% to 85%, depending on the language. This paper introduces the concept of language robustness as a desirable quality in digital speech biomarkers, ensuring consistent behaviour across languages. By leveraging this concept and employing additional metrics, the study proposes several language-independent digital speech biomarkers with high discrimination power for diagnosing PD. • Digital speech biomarkers enable objective assessment of hypokinetic dysarthria (HD). • Language affects speaker classification accuracy. • Introducing concept of features' language robustness. • Promising biomarkes: lower second formant prominence, monopitch, fewer pauses. • Text reading essential in HD assessment. [ABSTRACT FROM AUTHOR]

Deep learning has been vastly applied in classification systems of speech signals. However, adversarial examples crafted by adding small perturbations to input speech signals can make those classification systems produce incorrect results. Beyond detecting and discarding adversarial examples when securing the classification systems, we consider to purify the input adversarial examples to normal ones in this paper. Speech enhancement is a technique known as removing background noises to enhance voice quality. By treating adversarial perturbations as noises, we hereby propose a unified speech enhancement approach to mitigate both background noises and adversarial perturbations. The approach is named by Speech Enhancement and Adversarial Defense Network (SEADNet), which performs model-level fusion of conventional speech enhancement and adversarial defenses. In SEADNet, a learnable noise-adding module is firstly created to adaptively putting noises to the input speech to destroy the functions of adversarial perturbations. Subsequently, a speech enhancement network is deployed to reconstruct clean speech from noisy inputs. Finally, an adversarial example detection module is designed as a discriminator to guide the reconstructed speech in the previous speech enhancement network to mitigate adversarial perturbations. The overall network is optimized by minimizing a joint loss function. Experiments are conducted on the tasks of speech enhancement and automatic speaker verification with benchmark datasets. The proposed SEADNet performs well on both tasks of conventional noise mitigation and adversarial defenses compared to the related competitive baselines. • The first work to unify speech enhancement and adversarial defense in one model. • A learnable noise-adding module to mitigate adversarial attack adaptively. • Adversarial example detection to regularize speech enhancement networks. [ABSTRACT FROM AUTHOR]

Automatic Speech Recognition (ASR) systems are quickly becoming a crucial element in supporting healthcare providers, improving the flow of information among medical teams, and enhancing the patient's experience. However, to be fully supportive, these ASR systems must meet certain requirements dictated by market realities: high accuracy of speech recognition and low rate of errors, the possibility of additional training the model, and the possibility of on-premise system installation. Therefore, the aim of this paper is to perform a comparative analysis of leading ASR systems available on the Polish market for the needs of conducting medical interviews. We selected three systems, Google ASR, Microsoft ASR, and Techmo ASR, and we compared their performance on a prepared data set of medical-related expressions spoken in Polish. The results of our analysis indicated that there are minor discrepancies in the accuracy of speech recognition between all three evaluated ASR systems, whereas only two ASR systems met the raised requirements, in both cases partially. Still, they all exhibited specific problems in recognising word endings or word boundaries. We were able to categorise such problems into: Misrecognitions, Quality Problems, and Word Boundaries, varying in their level of influence on the further speech recognition process. Our research findings are expected to provide valuable insights to a wide range of stakeholders facilitating the development of tailored speech recognition solutions that meet the specific needs of medical sector. [ABSTRACT FROM AUTHOR]

In this paper, a low-complex chip to extract the Mel Frequency Cepstral Coefficient for a speech recognition system is presented. The architecture can operate in a continuous-flow manner to process streaming or the stored speech signal at high speed. The frame-overlap Hamming window, DFT and Mel-filter bank computations are deeply integrated to share memory buffers and avoid bit-reversal circuit to reduce area and latency. Moreover, normalised energy consumption and area delay product are reduced by 32%, and speed is increased by 5.2 times compared to prior works. Further, the fixed-point word-length is optimised to minimise the area without affecting the accuracy. • The paper focuses on the design of ASIC chip for the extraction of Mel Frequency Cepstral Coefficient from the speech input. • The MFCC chip process streaming or stored speech signal at high speed in continuous flow manner with a little area overhead. • The area and latency are reduced by integrating frame-overlap Hamming window, DFT and Mel Filter banks to share hardware. • The proposed architecture process a single frame with almost 5.2 times lesser clock cycles compared to other prior systems. • The reduction of area and computation cycles results in 32% reduction of area-delay product and normalized energy consumption. [ABSTRACT FROM AUTHOR]

In this paper, we propose a joint training framework that efficiently combines time-domain speech enhancement (SE) with an end-to-end (E2E) automatic speech recognition (ASR) system utilizing attention-based latent features. Using the latent feature to train E2E ASR implies that various time-domain SE models can be applied for noise-robust ASR and our modified framework is the first approach. We implement a fully E2E scheme pipelined from SE to ASR without domain knowledge and short-time Fourier transform (STFT) consistency constraints by applying a time-domain SE model. Therefore, using the latent feature of time-domain SE as appropriate features for ASR inputs is the main approach in our framework. Furthermore, we apply an attention algorithm to the time-domain SE model to selectively concentrate on certain latent features to achieve the better relevant feature for the task. Detailed experiments are conducted on the hybrid CTC/attention architecture for E2E ASR, and we demonstrate the superiority of our approach compared to baseline ASR systems trained with Mel filter bank coefficients features as input. Compared to the baseline ASR model trained only on clean data, the proposed joint training method achieves 63.6% and 86.8% relative error reductions on the TIMIT and WSJ "matched" test set, respectively. [ABSTRACT FROM AUTHOR]

Hate speech is a public statement that demonstrates resentment or provokes disturbance towards a person or group often based upon race, age, religion, sexual orientation, minority group, psychical disability, political belief, etc. Such an act is a leading cause of mental degeneration in individuals observed throughout the world. We have witnessed an upsurge in the spreading of hateful speech through videos in recent times due to increased social media usage. Researchers are working on this issue because it has become more frequent on several social media platforms, and it leads to low self-esteem and has significant negative impacts on human life. In this work, we focus on collecting data from such videos as nowadays people are sharing numerous videos of this negative nature on platforms like Facebook and YouTube. The audio data of these videos were then converted into text to build the dataset, and we applied some classifier models to our dataset. In this paper, we utilized a transfer learning Bidirectional Encoder Representations from Transformers (BERT) model that gives state-of-the-art outcomes. More precisely, we fine-tuned our model based on transfer learning to evaluate BERT's capacity to capture hostile contexts inside YouTube videos. We examined Fine–Tuning BERT; with different learning rates and listed the outcomes. We train the BERT by freezing all the hyperparameters but with various random seeds to evaluate our suggested Fine-tuning approach. Compared to previous methodologies that used our dataset, our proposition fared better in terms of accuracy and execution time. [ABSTRACT FROM AUTHOR]

This paper proposes a novel multimodal self-supervised architecture for energy-efficient audio-visual (AV) speech enhancement that integrates Graph Neural Networks with canonical correlation analysis (CCA-GNN). The proposed approach lays its foundations on a state-of-the-art CCA-GNN that learns representative embeddings by maximizing the correlation between pairs of augmented views of the same input while decorrelating disconnected features. The key idea of the conventional CCA-GNN involves discarding augmentation-variant information and preserving augmentation-invariant information while preventing capturing of redundant information. Our proposed AV CCA-GNN model deals with multimodal representation learning context. Specifically, our model improves contextual AV speech processing by maximizing canonical correlation from augmented views of the same channel and canonical correlation from audio and visual embeddings. In addition, it proposes a positional node encoding that considers a prior-frame sequence distance instead of a feature-space representation when computing the node's nearest neighbors, introducing temporal information in the embeddings through the neighborhood's connectivity. Experiments conducted on the benchmark ChiME3 dataset show that our proposed prior frame-based AV CCA-GNN ensures a better feature learning in the temporal context, leading to more energy-efficient speech reconstruction than state-of-the-art CCA-GNN and multilayer perceptron. • Demonstrates a GNN-CCA application for energy-efficient AV speech enhancement. • Proposes a novel GNN-CCA model for self-supervised fusion of correlated features. • Proposes the prior-frame positional encoding to model temporal information in graphs. • Outperforms baselines considering both audio reconstruction and energy efficiency. [ABSTRACT FROM AUTHOR]

This paper addresses the development of an Automatic Speech Recognition (ASR) system for the Moroccan Dialect. Dialectal Arabic (DA) refers to the day-to-day vernaculars spoken in the Arab world. In fact, Moroccan Dialect is very different from the Modern Standard Arabic (MSA) because it is highly influenced by the French Language. It is observed throughout all Arab countries that standard Arabic widely written and used for official speech, news papers, public administration and school but not used in everyday conversation and dialect is widely spoken in everyday life but almost never written. we propose to use the Mel Frequency Cepstral Coefficient (MFCC) features to specify the best speaker identification system. The extracted speech features are quantized to a number of centroids using vector quantization algorithm. These centroids constitute the codebook of that speaker. MFCC's are calculated in training phase and again in testing phase. Speakers uttered same words once in a training session and once in a testing session later. The Euclidean distance between the MFCC's of each speaker in training phase to the centroids of individual speaker in testing phase is measured and the speaker is identified according to the minimum Euclidean distance. The code is developed in the MATLAB environment and performs the identification satisfactorily. [ABSTRACT FROM AUTHOR]

In recent years, significant advancements have been made in neural beamforming, leveraging spectral and spatial cues to enhance their performance in multi-channel speech enhancement. When the frame-wise processing mechanism is required, there exists a trade-off between performance and algorithmic delay for existing all-neural beamformers. However, from the perspective of multi-source information fusion, the network is often encapsulated into a black box to entangle and fuse the spatial and spectral features into a non-linear feature space, which hinders our understanding of how they work collaboratively for target speech extraction. In this regard, this paper proposes to decouple the spatial and spectral domain processing inspired by Taylor's approximation theory. Specifically, we reformulate the time-variant beamforming defined in the spatial domain into the adaptive weighting and mixing of different beam components in the beamspace domain. This reformulation enables us to model the recovery of target speech as a weighted sum operation in the beamspace domain, where each beam component is associated with an introduced unknown term for residual interference cancellation. By virtue of Taylor's series expansion, the recovery process can be decomposed into the superimposition of the 0th-order non-derivative and high-order derivative terms, where the former acts as spatial filtering in the spatial domain, and the latter serves as a residual interference canceller in the spectral domain. We conduct extensive experiments on the spatialized LibriSpeech and L3DAS Challenge datasets. Experimental results show that, compared with existing advanced approaches, the proposed method not only achieves competitive performance in terms of multiple objective metrics but also provides feasible guidance in multi-channel speech enhancement pipeline design. • This paper proposes to decouple the spatial and spectral processing inspired by Taylor's approximation theory. • Beamforming in spatial domain is reformulated into the adaptive weighting of beamed components in the beamspace domain. • Speech recovery is decomposed into the superimposition of the 0th-order non-derivative and high-order derivative terms. [ABSTRACT FROM AUTHOR]

In this paper, an audio-visual model for separating a speech of the target speaker from a combination of other speakers' speeches is proposed. It can be used in speech separation, automatic speech recognition systems (ASR) and also in creating single speaker speech databases. Speech separation is complicated problem using audio information only so visual and auditory signals are combined to complete the separation process. The proposed model consists of four modules, two for audio signal, one for visual feature and the last one used to concatenate the features resulted from the previous three modules to generate the separated signals. Our proposed model improved Short-time objective intelligibility (STOI) with 11%, Perceptual Evaluation of Speech Quality (PESQ) with 24%, and Frequency-weighted Segmental SNR (fwSNRseg) with 16% compared with previous works. It also improved Csig' which is the predicted rating of speech distortion with 13% and 'Covl' which is the predicted rating of overall quality with 18% compared with previous audio-visual models. [ABSTRACT FROM AUTHOR]

Recent advancements in deep learning (DL) have posed a significant challenge for automatic speech recognition (ASR). ASR relies on extensive training datasets, including confidential ones, and demands substantial computational and storage resources. Enabling adaptive systems improves ASR performance in dynamic environments. DL techniques assume training and testing data originate from the same domain, which is not always true. Advanced DL techniques like deep transfer learning (DTL), federated learning (FL), and deep reinforcement learning (DRL) address these issues. DTL allows high-performance models using small yet related datasets, FL enables training on confidential data without dataset possession, and DRL optimizes decision-making in dynamic environments, reducing computation costs. This survey offers a comprehensive review of DTL, FL, and DRL-based ASR frameworks, aiming to provide insights into the latest developments and aid researchers and professionals in understanding the current challenges. Additionally, Transformers, which are advanced DL techniques heavily used in proposed ASR frameworks, are considered in this survey for their ability to capture extensive dependencies in the input ASR sequence. The paper starts by presenting the background of DTL, FL, DRL, and Transformers and then adopts a well-designed taxonomy to outline the state-of-the-art (SOTA) approaches. Subsequently, a critical analysis is conducted to identify the strengths and weaknesses of each framework. Additionally, a comparative study is presented to highlight the existing challenges, paving the way for future research opportunities. • Recent DL advancements challenge ASR, demanding adaptive systems. • DTL, FL, RL tackle ASR challenges: data domain differences, resource demands. • Survey reviews DTL, FL, RL-based ASR frameworks, aiding understanding. • Transformers, key in ASR, capture extensive dependencies in input sequences. • Analyzes strengths, weaknesses, and future research in ASR frameworks. [ABSTRACT FROM AUTHOR]

The application of virtual reality (VR) technology in rehabilitation is a research area that has been attracting remarkable attention recently, especially as an intervention with learning and developmental disorders such as dyslexia and autism. Multi-modal VR-based rehabilitative systems can provide a rich experience to patients by responding to various types of bio-signals. This paper proposes a framework for incorporating voice input in rehabilitative games. The proposed framework equips collaborative learning environments with voice based interaction which allows for a more dynamic flow of therapy sessions between physically impaired patients and specialists in a rehabilitative VR environment. The proposed framework's pipeline begins with a speech recognition module to parse utterances into sentences. Next, comes the feature extraction module to identify the intent of the user and entities as well. Finally, we modulate the 3D environment in Unity. The framework was designed in a way that guarantees smooth integration with any game or 3D environment. Several technologies were tested in terms of accuracy and speed to increase the overall performance of the pipeline. To the best of our knowledge, this research is the first to do the following: 1) build a dataset of instructions, inspired by a product already used in the healthcare market, 2) based on which we could realize a multi-modal system, with voice as one of the modalities, for the context of VR-based rehabilitation. We evaluate every stage in the system pipeline, and we show that we can achieve considerable results on each of these stages including the speech recognition module, with a 5.64% word error rate, and on intent classification where the BiLSTM model outperforms BERT in the Named Entity Recognition by 11% in terms of the F1-score. We envisage the proposed system to lend itself well to emerging healthcare frameworks in the Metaverse. [ABSTRACT FROM AUTHOR]

The cross-attention mechanism enables Transformer to capture correspondences between the input and output. However, in the domain of end-to-end (E2E) speech-to-text translation (ST), the learned cross-attention weights often struggle to accurately correspond with actual alignments, given the need to align speech and text across different modalities and languages. In this paper, we present a simple yet effective method called regularized cross-attention learning, for end-to-end speech translation in a multitask learning (MTL) framework. RCAL leverages the knowledge from auxiliary automatic speech recognition (ASR) and machine translation (MT) tasks to generate a teacher cross-attention matrix, serving as prior alignment knowledge to enhance cross-attention learning within the ST task. An additional loss function is introduced as part of the MTL framework to facilitate this process. We conducted experiments on the MuST-C benchmark dataset to evaluate the effectiveness of RCAL. The results demonstrate that the proposed approach yields significant improvements over the baseline, with an average enhancement of +0.8 BLEU across four translation directions in two experimental settings, outperforming state-of-the-art E2E and cascaded speech translation models. Further analysis and visualization reveal that the model with RCAL effectively learns high-quality alignment information from auxiliary ASR and ST tasks, thereby improving the ST alignment quality. Moreover, the experiments with different sizes of MT and ST data provide strong evidence supporting our model's robustness in various scenarios. • Exploring how to mitigate the modality gap for E2E ST in the decoder. • Transfer alignment knowledge from the ASR and MT tasks to the E2E ST task. • Proposing two strategies to filter low-quality attention weights. [ABSTRACT FROM AUTHOR]

Language-independent spoken term detection (LI-STD) refers to the process of locating the occurrences of spoken queries from speech databases of any language. This paper alization of a multilingual broad phoneme classifier (BPC) and its application for the development of an LI-STD system. This work proposes a multi-stage architecture to address the task of LI-STD for low-resourced languages, where there is limited amount of labelled training data. The proposed LI-STD system contains three stages; one label sequence matching stage and two template matching stages. A deep neural network (DNN) based BPC trained using 16 handcrafted, signal-based features is the backbone of the proposed LI-STD system. In LI-STD system, stage 1 performs a broad phoneme sequence matching, while stage 2 and 3 perform template matching on posteriorgram and feature sequence, respectively. Concatenation of multiple stages results in search space reduction for the later computationally intensive template matching stages. In order to adapt to a new/unseen language, the BPC gets retrained using selected broad phoneme labelled data of the language generated by itself. The effectiveness of the proposed system is demonstrated on a set of low-resourced Indian languages. [ABSTRACT FROM AUTHOR]

The paper proposes an approach to the formal representation of natural language elements based on multi-agent recursive architectures for solving problems of speech recognition, understanding and synthesis. This approach, based on intelligent software agents with a developed cognitive architecture, makes it possible to represent linguistic information at all levels of the language: from specific morphological to abstract semantic. [ABSTRACT FROM AUTHOR]

Highlights • We perform a detailed analysis of acoustic phenomena produced in newborn cries, focusing specifically on the prevalence of fundamental frequency glide (F 0glide) and resonance frequencies dysregulation (RFsdys). • We propose an automated tool that uses (F0glide) and (RFsdys) to support the diagnosis of diseases in newborn infants. • The obtained results indicate the important association between the quantified features and some studied pathologies, and also an improvement in the identification of pathologic cries. Abstract Our challenge in the current study is to extend research on the cries of newborns for the early diagnosis of different pathologies. This paper proposes a recognition system for healthy and pathological cries using a probabilistic neural network classifier. Two different kinds of features have been used to characterize newborn cry signals: 1) acoustic features such as fundamental frequency glide (F 0glide) and resonance frequencies dysregulation (RFs dys); 2) conventional features such as mel-frequency cestrum coefficients. This paper describes the automatic estimation of the proposed characteristics and the performance evaluation of these features in identifying pathological cries. The adopted methods for F 0glides and RFs dys estimation are based on the derived function of the F 0 contour and the jump "J" of the RFs between two subsequent tunings, respectively. The database used contains 3250 cry samples of full-term and preterm newborns, and includes healthy and pathologic cries. The obtained results indicate the important association between the quantified features and some studied pathologies, and also an improvement in the identification of pathologic cries. The best result obtained is 88.71% for the correct identification of health status of preterm newborns, and 82% for the correct identification of full-term infants with a specific disease. We conclude that using the proposed characteristics improves the diagnosis of pathologies in newborns. Moreover, the method applied in the estimation of these characteristics allows us to extend this study to other uninvestigated pathologies. [ABSTRACT FROM AUTHOR]

This paper addresses the development of an Automatic Speech Recognition (ASR) system for the Moroccan Dialect. Dialectal Arabic (DA) refers to the day-to-day vernaculars spoken in the Arab world. In fact, Moroccan Dialect is very different from the Modern Standard Arabic (MSA) because it is highly influenced by the French Language. It is observed throughout all Arab countries that standard Arabic widely written and used for official speech, news papers, public administration and school but not used in everyday conversation and dialect is widely spoken in everyday life but almost never written. we propose to use the Mel Frequency Cepstral Coefficient (MFCC) features to specify the best speaker identification system. The extracted speech features are quantized to a number of centroids using vector quantization algorithm. These centroids constitute the codebook of that speaker. MFCC's are calculated in training phase and again in testing phase. Speakers uttered same words once in a training session and once in a testing session later. The Euclidean distance between the MFCC's of each speaker in training phase to the centroids of individual speaker in testing phase is measured and the speaker is identified according to the minimum Euclidean distance. The code is developed in the MATLAB environment and performs the identification satisfactorily. [ABSTRACT FROM AUTHOR]

• This paper proposes a supervised CatPCA approach for performing the imagined speech classification. • The objective function of an optimization problem for designing the unitary matrix is formulated based on both the PCA algorithm and the k means algorithm. • To find the analytical solution of the optimization problem, a property on computing the singular value decomposition (SVD) of a symmetric matrix is employed. • Then, the design of the cluster centers in the k means algorithm is formulated as a quadratic programming. • Finally, the above two procedures are iterated until the algorithm converges. • The required computational power for finding the solution of the optimization problem is low. The dimension of the feature vectors for performing the imagined speech is large. Hence, it is required to reduce their dimension as well as perform the classification simultaneously. This is a categorized principal component analysis (CatPCA) problem. However, the conventional CatPCA methods are unsupervised learning methods and the design of the unitary matrix in the conventional principal component analysis (PCA) algorithms is only based on the PCA approach. Therefore, the classification performance is limited. This paper proposes a supervised CatPCA approach for performing the imagined speech classification. Also, the objective function of an optimization problem for designing the unitary matrix is formulated based on both the PCA algorithm and the k means algorithm. To find the analytical solution of the optimization problem, a property on computing the singular value decomposition (SVD) of a symmetric matrix is employed. Then, the design of the cluster centers in the k means algorithm is formulated as a quadratic programming. Finally, the above two procedures are iterated until the algorithm converges. Since the analytical solutions of these two optimization problems can be derived analytically, the required computational power for finding the solution of the optimization problem is low. To demonstrate the effectiveness of our proposed CatPCA algorithm, the Track 3 of the International brain computer interface (BCI) Competition Database is employed as the dataset for performing the performance evaluation. The computer numerical simulation results show that our proposed CatPCA algorithm yields the higher classification accuracy compared to PCA variants and the dimension reduction related algorithms. Also, the required execution time of our proposed method is lower than those of the states of the arts methods. This demonstrates the effectiveness and the efficiency of our proposed CatPCA algorithm. [ABSTRACT FROM AUTHOR]

• Using speech materials recorded from speakers with different voice disorders this paper shows: how continuous speech recordings recorded in voice clinics can be automatically annotated to identify different phonetic regions. • That consideration of how voice disorders differentially affect phonetic regions can improve the discrimination between types of disorder. • That automated differential diagnosis of voice disorders from a read passage is possible at sensitivities and specificities typical of diagnostic screening tests. In this paper we evaluate the hypothesis that automated methods for diagnosis of voice disorders from speech recordings would benefit from contextual information found in continuous speech. Rather than basing a diagnosis on how disorders affect the average acoustic properties of the speech signal, the idea is to exploit the possibility that different disorders will cause different acoustic changes within different phonetic contexts. Any differences in the pattern of effects across contexts would then provide additional information for discrimination of pathologies. We evaluate this approach using two complementary studies: the first uses a short phrase which is automatically annotated using a phonetic transcription, the second uses a long reading passage which is automatically annotated from text. The first study uses a single sentence recorded from 597 speakers in the Saarbrucken Voice Database to discriminate structural from neurogenic disorders. The results show that discrimination performance for these broad pathology classes improves from 59% to 67% unweighted average recall when classifiers are trained for each phone-label and the results fused. Although the phonetic contexts improved discrimination, the overall sensitivity and specificity of the method seems insufficient for clinical application. We hypothesise that this is because of the limited contexts in the speech audio and the heterogeneous nature of the disorders. In the second study we address these issues by processing recordings of a long reading passage obtained from clinical recordings of 60 speakers with either Spasmodic Dysphonia or Vocal fold Paralysis. We show that discrimination performance increases from 80% to 87% unweighted average recall if classifiers are trained for each phone-labelled region and predictions fused. We also show that the sensitivity and specificity of a diagnostic test with this performance is similar to other diagnostic procedures in clinical use. In conclusion, the studies confirm that the exploitation of contextual differences in the way disorders affect speech improves automated diagnostic performance, and that automated methods for phonetic annotation of reading passages are robust enough to extract useful diagnostic information. [ABSTRACT FROM AUTHOR]

Recently, several countries have been trying hard to facilitate the integration of disabled people into their societies by ensuring equal opportunities through ease of access to social services, daily human necessities, and the labor market. Deafness is considered one of the major disabilities separating the deaf from their society. To integrate the deaf fully into society, a two-way mode of communication is required: one from the deaf to the hearing people, and the other from the hearing to the deaf. Communication from the hearing person to the deaf is generally easy and can be done through speech recognition and text-to-sign representations, but communication from the deaf to the hearing is somewhat difficult and requires a sign recognition module that recognizes the sign motions from the deaf and translates it to a text; following this, a speech synthesis module will translate this text to speech. To build the sign recognition module, a sign language dataset is required. This paper contributes to the literature by introducing a comprehensive survey of 17 Arabic sign language datasets and by developing a well-organized framework that is used to build a sign language dataset. This paper also contributes to the literature by creating the largest Saudi Sign Language (SSL) database—the King Saud University Saudi-SSL (KSU-SSL data-base)—with 293 signs, 33 signers, 145,035 samples, and 10 domains (healthcare, common, alpha-bets, verbs, pronouns and adverbs, numbers, days, kings, family, and regions). This paper also contributes to the literature by introducing a convolutional graph neural network (CGCN) architecture for sign language recognition and applying the proposed architecture to the built KSU-SSL database. The architecture is made up of a small number of separable 3DGCN layers, and is augmented with a spatial attention mechanism. This study is a part of the project that aims to develop a two-way communication system for Saudi deaf individuals. [ABSTRACT FROM AUTHOR]

Machine learning (ML), a branch of artificial intelligence (AI), is an area of computational science that is concerned with the analysis and interpretation of patterns and structures in data to enable learning and decision-making without the participation of a human. Hidden Markov models (HMMs), which have been acknowledged for decades but have recently made a significant revival in machine learning, are one of the most impressively powerful probabilistic models. HMMs are frequently employed in machine learning to model heterogeneous time series data. In this paper, we integrate independent component analysis (ICA) and ICA with a bounded multivariate generalized Gaussian mixture model (ICA-BMGGMM) into the HMM approach. One limitation of ICA is that it assumes the sources to be independent from each other. This assumption can be relaxed by combining independent vectors analysis (IVA) and IVA with the BMGGMM (IVA-BMGGMM) into the HMM approach to improve their modeling capability. We validate our proposed models using a variety of applications, such as human action recognition, speech recognition, and energy disaggregation. The results presented in the paper demonstrate the effectiveness of the proposed approaches for modeling different types of data. These data include KTH and Weizmann datasets for human action recognition, TIMIT and SDR for speech recognition, REDD dataset for energy disaggregation and EEG dataset for elliptic seizure classification. For all conducted experiments, our proposed models outperform other comparing models for all performance metrics such as accuracy, sensitivity, and precision. The best detection results were found using the IVABMGGMM-HMM for the reported experiments. [ABSTRACT FROM AUTHOR]

Speech Recognition is one of the several Artificial Intelligence applications. It helps us converting spoken words into text. It can be part of various daily use cases in order to deal with accessibility. Google Assistant and Amazon's Alexa are in the top of list of the well-known Speech Recognition tools. European companies cannot use these solutions as they should guarantee data sovereignty. Another important point is that these mentioned solutions are not customized. So that, it is not possible to deal with new accents or new vocabularies. To cope with these problems, one can either use European Automatic Speech Recognition (ASR) solutions or build his own personalized models using well-known open-source tools like Deep Speech or Kaldi. Choosing the best solution between both, Kaldi and DeepSpeech, is an important task. The criteria for judging the finest method are the Accuracy and the Inference Time. In this paper, we make theoretical and experimental study between DeepSpeech and Kaldi. Also, Vosk and LinTO, open-source solutions build in top of Kaldi, will be included in the comparison study. [ABSTRACT FROM AUTHOR]

Emotions are the base to comprehend speech and respond to each other. However, interpreting emotions is not a simple process even for people, not to mention machines. Based on a ground truth investigation that we have done on the RML (Ryerson Multimedia Lab) dataset, we have found that the same emotional speech might be interpreted differently by a number of individuals despite it has only one label. In this study we provide a fresh perspective to the speech emotion recognition field where a system may identify the presence of one or more potential emotions per utterance. Giving the computer the potential of identifying numerous categories at the same time, will allow it to assess different scenarios when making a decision. In this paper, we introduce a robust Speech Emotion Recognition (SER) system for multi-label classification. The model was tested on the RML and it achieved 83.88% which outperforms state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Several speech emotion recognition systems have been proposed, mostly targeting a single language; however, the recognition performance is significantly degraded for languages other than the target language. To solve this problem, a method of normalizing features using synthesized speech was proposed by Okabe et al.; however, this method requires a search for the optimal normalization operation. In this paper, we propose a multilingual emotion recognition method that automatically normalizes acoustic features and recognizes emotions using the "recognition rate maximization" criterion. Synthesized speech with the same content as emotional speech was generated and input to a convolutional recurrent neural network with time axis aligned. Feature normalization by comparison with synthesized speech and emotion recognition were performed in a single network, and efficient training was expected. In speech emotion recognition experiments with four languages, normalization improved the recognition accuracy from 47.5% to 49.0%. Because normalization was not effective for utterances in which the speaking rate of the synthesized and emotional speeches differed significantly, we proposed a method to adjust the speaking rate of the synthesized speech and further improved the accuracy by up to 50.4%. [ABSTRACT FROM AUTHOR]

Automatic speech recognition (ASR) in the military domain can enable voice user interfaces of command & control information systems (C2IS) and, thus, improve the usability of C2IS. It can also serve intelligence (COMINT) by supporting the monitoring of radio communication. However, military use cases are challenging as ASR has to cope with adverse conditions due to speech variations, noise and other channel disturbances, as well as strong requirements regarding reliability and robustness. Generic ASR solutions do not meet these requirements sufficiently. Therefore, tailor-fit ASR solutions have to be provided. In order to enable the quick development and testing of such solutions for specific use cases, we designed a Speech Recognition Lab (SR Lab). A demonstrator of the SR Lab and its components has been implemented and will be described in this paper. [ABSTRACT FROM AUTHOR]

Abstract Sentiment analysis on video is quite an unexplored field of research wherein the emotion and sentiment of the speaker are extracted by processing the frames, audio and text obtained from the video. In recent times, sentiment analysis from naturalistic audio has been an upcoming field of research. This is typically done by performing automatic speech recognition on audio, followed by extracting the sentiment exhibited by the speaker. On the other hand, techniques for extracting sentiments from text are quite developed and tech giants have already optimized these methods to process large amounts of customer review, feedback and reactions. In this paper, a new model for sentiment analysis from audio is proposed which is a hybrid of Keyword Spotting System (KWS) and Maximum Entropy (ME) Classifier System. This model is developed with the aim to outperform other conventional classifiers and to provide a single integrated system for audio and text processing. In addition, a web application for dynamic processing of YouTube videos is described. The WebApp provides an index-based result for each phrase that is detected in the video. Often, the emotion of the viewer of a video corresponds to its content. In this regard, it is useful to map these emotions to the text transcript of the video and assign a suitable weight to it while predicting the sentiment that the speaker exhibits. This paper describes such an application that was developed to analyze facial expressions using Affdex API. Thus, using the combined statistics from all the three aforementioned components, a robust and portable system for emotion detection is obtained that provides accurate predictions and can be deployed on any modern systems with minimal configuration changes. [ABSTRACT FROM AUTHOR]

Abstract Natural Language Understanding (NLU) is considered a core component in implementing dialogue systems. NLU has been greatly enhanced by deep learning techniques such as word embeddings and deep neural network architectures, but current NLP methods for Arabic language dialogue action classification or semantic decoding is mostly based on handcrafted rule-based systems and methods that use feature engineering, but without the benefit of any form of distributed representation of words. This paper presents an approach to use deep learning techniques for text classification and Named Entity Recognition for the domain of home automation in Arabic. To this end, we present an NLU module that can further be integrated with Automatic Speech Recognition (ASR), a Dialogue Manager (DM) and a Natural Language Generator (NLG) module to build a fully working dialogue system. The paper further describes our process of collecting and annotating the data, structuring the intent classifier and entity extractor models, and finally the evaluation of these methods on different benchmarks. [ABSTRACT FROM AUTHOR]

The acoustic and prosodic characteristics of speech exhibit alterations when individuals are affected by different health conditions. The field of biomedical engineering holds significant potential in the advancement of non-invasive diagnostic systems that utilize voice as a modality. The common cold is an infectious sickness that affects a large number of people all over the world each year. This paper presents the utilization of various spectral features and a transformer-based model with focal loss function for classifying cold-affected and healthy speech signals. A spectral feature consisting of Mel frequency cepstral coefficients (MFCC), Mel-spectrogram, chromagram, spectral contrast, spectral centroid, spectral bandwidth, spectral flatness, and spectral roll-off features. The efficacy of the proposed methodology is assessed using the URTIC database. The findings indicate that the proposed framework has better results compared to existing state-of-the-art approaches. We have achieved the UAR of 69.55% on the develop set and 70.48% on the test set of the URTIC database. These preliminary findings exhibit significant potential for future investigation in this domain. • A non-invasive diagnostic methodology for detecting the common cold through the analysis of speech signals. • The fusion of various spectral features for depicting distinct speech patterns observed in people suffering from a common cold. • Development of the state-of-the-art deep neural network-based transformer model for cold speech classification. • The optimization of transformer architecture using focal loss function to address the issue of imbalanced classes. • The proposed framework achieved the UAR of 69.55% on the develop set and 70.48% on the test set of the URTIC database. [ABSTRACT FROM AUTHOR]

Stuttering is a prevalent speech disorder affecting millions worldwide. To provide an automatic and objective stuttering assessment tool, Stuttering Event Detection (SED) is under extensive investigation for advanced speech research and applications. Despite significant progress achieved by various machine learning and deep learning models, SED directly from speech signal is still challenging due to stuttering speech's heterogeneous and overlapped nature. This paper presents a novel SED approach using multi-feature fusion and attention mechanisms. The model utilises multiple acoustic features extracted based on different pitch, time-domain, frequency domain, and automatic speech recognition feature to detect stuttering core behaviours more accurately and reliably. In addition, we exploit both spatial and temporal attention mechanisms as well as Bidirectional Long Short-Term Memory (BI-LSTM) modules to learn better representations to improve the SED performance. The experimental evaluation and analysis convincingly demonstrate that our proposed model surpasses the state-of-the-art models on two popular stuttering datasets, with 4% and 3% overall F1 scores, respectively. The superior results indicate the consistency of our proposed method, supported by both multi-feature and attention mechanisms in different stuttering events datasets. • We propose A novel attention-based model to learn stuttering events detection (SED). • Our multi-feature fusion approach can improve SED performance. • Our multi-feature fusion model outperforms state-of-the-art methods. • We introduced an attention map along two separate dimensions for SED. • We demonstrate the effectiveness of CBAM in focusing on the salient features. [ABSTRACT FROM AUTHOR]

Speech accent recognition (SAR) plays a crucial role in enhancing communication between customers and service providers, enabling personalized interactions based on geographical, birthplace, and cultural cues derived from accents. However, current approaches predominantly rely on training SAR models from scratch, overlooking the potential of transfer learning from other speech processing tasks, despite the relatively small size of accent datasets. This paper represents the first comprehensive investigation into the effectiveness of transfer learning methods derived from a diverse array of data-rich speech processing tasks for SAR. Through experiments on a practical Vietnamese telephone dataset provided by Viettel, the largest telecommunications provider in Southeast Asia, our study reveals that our best-performing model outperforms previous state-of-the-art SAR models by 46.7% in terms of relative accuracy. [ABSTRACT FROM AUTHOR]

Silent speech recognition (SSR) is a system that can recognize verbal expressions when speech signals are not accessible. It has found versatile applications not only in helping those who suffer a speech impediment, but also in human–computer interaction. In this paper, we propose an SSR system based on surface electromyography (sEMG) signals. We first design a Mandarin corpus which contains 135-class utterances and collect 8-channel sEMG data from 12 subjects. After preprocessing the raw sEMG signals, we extract a time–frequency hybrid feature map for each sample. Then, we use two data augmentation strategies to obtain more data in order to improve the performance of our model. Finally, we test our proposed models on two tasks: the multi-subject mixed classification task and the single subject classification task. We use a 5-fold cross-validation to evaluate our model on both tasks. The GRU model performs best, achieving a recognition accuracy of 88.01% on the multi-subject mixed classification task and a maximum recognition accuracy of 97.19% on the single subject classification task. Experiments results demonstrate the effectiveness of our proposed system. • A silent speech system based on surface electromyography signals and deep learning models. • Time–frequency hybrid features are extracted from surface electromyography signals. • Speed perturbation and time–frequency masking are used for data augmentation. • Several deep learning models are evaluated on two different tasks; The GRU model performed the best. [ABSTRACT FROM AUTHOR]

Most existing voice conversion methods focus primarily on separating speech content from speaker information while overlooking the decoupling of pitch information. Additionally, the quality of converted speech significantly degrades when the speech of the target speaker is contaminated by noises. To address these issues, this paper proposes a noise-robust voice conversion model with multi-feature decoupling based on adversarial training. The proposed framework utilizes three distinct encoders to encode speech content, speaker identity, and pitch information independently, which aims to enhance the performance of decoupling by minimizing their mutual information and reduce the correlations between feature vectors. Moreover, a gradient reversal layer and a noise decoupling discriminator are incorporated into the framework, which extracts noise-resistant speaker representations and content representations through adversarial training to facilitate the synthesis of high-quality speech. In order to optimize the learning process, a training strategy is developed which involves alternating between clean and noisy data during the training of the encoder. This strategy effectively guides and expedites the convergence of the model. Experimental results demonstrate that compared to the state-of-the-art baselines of noise-robust voice conversion, the proposed model achieves improvements around 0.31 and 0.39 in terms of speech naturalness and speaker similarity evaluation metrics, respectively. [ABSTRACT FROM AUTHOR]

Conventionally, the extraction of hand-crafted acoustic features has been separated from the task of establishing robust machine-learning models in speech processing. The manual approach of feature engineering is both time-consuming and necessitates specialist knowledge, posing significant hindrances. Moreover, the resulting features may not be ideal for the desired application. The speech community has adopted raw waveform modeling to enhance performance. These techniques learn an optimized representation of the input automatically. With deep learning (DL) advancements, raw waveform modeling has become valuable for tasks like classification and prediction. The primary aim of this survey is to offer valuable insights and fills a gap in the existing literature by providing a comprehensive review of the state-of-the-art in speech and speaker recognition using raw waveform modeling for both adult and children's speech. The article covers papers from 2013–2023 and is the first to review both adult and children's databases. The article focuses on the advantages of raw waveform models. It presents essential concepts and techniques while discussing the challenges and limitations of using raw waveforms for speech and speaker recognition in both adult and children's speech. The article also evaluates recent progress in DL architectures such as SincNet, ResNet, and RawNet, and outlines future research directions in the field. [ABSTRACT FROM AUTHOR]

Speech recognition is the field of research concerning the ability of machines to accept human voice as input and interpret it with the highest probability. This paper aims at simulating the driving of a robot by speech recognition of the dialect language "Darija" using the training of a deep-learning model, then the validation of the model will be done by simulation on the ROS environment, the simulation consists of driving a robot car that moves by voice commands in Darija. We develop here a deep-learning model based on TensorFlow to recognize this language. This model has been successfully tested in a car driving simulator (Robot Operating System, ROS) with the robot HUSKY with success rates up to 90% of recognition. [ABSTRACT FROM AUTHOR]

Communication between people and machines has been extended in the last two decades. Corresponding techniques have been founded to cover the need of voice understanding, including speech and speaker recognition on a large-scale. In this paper, the authors propose a simplified deep-learning approach to accomplish the large-scale speaker identification task using as little training data as possible. Fisher speech corpus has been explored to select the recordings of unique speakers having sufficient data. The authors are using the MFCC method to represent the feature vectors of a large set of more than 4 k speakers with about 343 h of speech signals. The solution includes omitting the pre-processing and considering longer segments of the voice signals. Various portions of training datasets have been tested, as well as dedicating larger percentages of the used data for testing. Bidirectional LSTM neural networks provided up to 76.9% accuracy rate for individual voice segments, and 99.5% when considering the segments of each speaker as a bundle. Doubling the amount of the training data yielded a perfect accuracy rate of 100%. [ABSTRACT FROM AUTHOR]

Dysarthria is a motor speech disability caused by weak muscles and organs involved in the articulation process, thereby affecting the speech intelligibility of individuals. Because this condition is linked to physical exhaustion disabilities, individuals not only have communication difficulties, but also have difficulty interacting with digital devices. Automatic speech recognition (ASR) makes an important difference for individuals with dysarthria since modern digital devices offer a better interaction medium that enables them to interact with their community and computers. Still, the performance of ASR technologies is poor in recognizing dysarthric speech, particularly for acute dysarthria. Multiple challenges, including dysarthric phoneme inaccuracy and labeling imperfection, are facing dysarthric ASR technologies. This paper proposes a spatio-temporal dysarthric ASR (DASR) system using Spatial Convolutional Neural Network (SCNN) and Multi-Head Attention Transformer (MHAT) to visually extract the speech features, and DASR learns the shapes of phonemes pronounced by dysarthric individuals. This visual DASR feature modeling eliminates phoneme-related challenges. The UA-Speech database is utilized in this paper, including different speakers with different speech intelligibility levels. However, because the proportion of usable speech data to the number of distinctive classes in the UA-speech database was small, the proposed DASR system leverages transfer learning to generate synthetic leverage and visuals. In benchmarking with other DASRs examined in this study, the proposed DASR system outperformed and improved the recognition accuracy for 20.72% of the UA-Speech database. The largest improvements were achieved for very-low (25.75%) and low intelligibility (33.67%). [ABSTRACT FROM AUTHOR]

This paper proposes a new method for calculating joint-state posteriors of mixed-audio features using deep neural networks to be used in factorial speech processing models. The joint-state posterior information is required in factorial models to perform joint-decoding. The novelty of this work is its architecture which enables the network to infer joint-state posteriors from the pairs of state posteriors of stereo features. This paper defines an objective function to solve an underdetermined system of equations, which is used by the network for extracting joint-state posteriors. It develops the required expressions for fine-tuning the network in a unified way. The experiments compare the proposed network decoding results to those of the vector Taylor series method and show 2.3% absolute performance improvement in the monaural speech separation and recognition challenge. This achievement is substantial when we consider the simplicity of joint-state posterior extraction provided by deep neural networks. [ABSTRACT FROM AUTHOR]

The use of mixed languages in daily conversations, referred to as “code-switching”, has become a common linguistic phenomenon among bilingual/multilingual communities. Code-switching involves the alternating use of distinct languages or “codes” at sentence boundaries or within the same sentence. With the rise of globalization, code-switching has become prevalent in daily conversations, especially among urban youth. This lead to an increasing demand on automatic speech recognition systems to be able to handle such mixed speech. In this paper, we present the first steps towards building a multilingual language model (LM) for code-switched Arabic-English. One of the main challenges faced when building a multilingual LM is the need of explicit mixed text corpus. Since code-switching is a behaviour used more commonly in spoken than written form, text corpora with code-switching are usually scarce. Therefore, the first aim of this paper is to introduce a code-switch Arabic-English text corpus that is collected by automatically downloading relevant documents from the web. The text is then extracted from the documents and processed to be useable by NLP tasks. For language modeling, a baseline LM was built from existing monolingual corpora. The baseline LM gave a perplexity of 11841.9 and Out-of-Vocabulary (OOV) rate of 4.07%. The gathered code-switch Arabic-English corpus, along with the existing monolingual corpora were then used to construct several LMs. The best LM achieved a great improvement over the baseline LM, with a perplexity of 275.41 and an OOV rate of 0.71%. [ABSTRACT FROM AUTHOR]

The most important shortcoming of the current speaker verification methods (based on knowledge or possession) is that this process is not sure whether the holder of the given ID is the entitled one or an imposter. Using biometrics in the verification system is designated for minimizing this problem and decreasing the necessity of carrying the tokens. In this paper, a novel Arabic text-independent speaker verification system is presented. First of all, new speech features are proposed for speaker characterization, which denoted as Wavelet Packet Four-Directional Features (WPFDF). With the objective of speaker verification, the paper proposes a Fuzzy Hidden Markov Model, termed FHMM, where the kernel fuzzy c-means (KFCM) is extended to calculate fuzzy memberships of HMMs training samples. Thus, information loss is reduced as well as recognition rate is increased. The proposed approach reached 98.38% of recognition rate. [ABSTRACT FROM AUTHOR]

This paper describes a suitable sign translator system that can be used for Arabic hearing impaired and any Arabic Sign Language (ArSL) users as well. The proposed system makes use of an automatic Arabic Speech Recognition module (ASR) integrated with Arabic Signer model to generate the ArSL signs. It translates from Arabic speech recognition (ASR) into ArSL. Therefore, the paper describes technologies for generating 3D Avatar signers from the Arabic speech recognition engine. The 3D avatar is based on character behaviours and gestures used by ArSL users. Consequently, sign language translator and the 3D avatar technology are used to sign the Arabic text script produced from the Arabic automatic speech recognition (ASR) engine. Therefore, the Arabic Text-to-Sign (ArTTS) translation is the goal of this paper. The machine translation with a set of transformational rules and language model with linguistic background used throughout our work will be described according to the model used for the proposed system. A typical presentation is developed to establish the values of the proposed solution. Experimental results show that the proposed ArTTS technique has a sign detection rate of 85 %. [ABSTRACT FROM AUTHOR]