Search

Childhood apraxia of speech (CAS) commonly affects the production of lexical stress contrast in polysyllabic words. Automated classification tools have the potential to increase reliability and efficiency in measuring lexical stress. Here, factors affecting the accuracy of a custom-built deep neural network (DNN)-based classification tool are evaluated. Sixteen children with typical development (TD) and 26 with CAS produced 50 polysyllabic words. Words with strong–weak (SW, e.g., dinosaur) or WS (e.g., banana) stress were fed to the classification tool, and the accuracy measured (a) against expert judgment, (b) for speaker group, and (c) with/without prior knowledge of phonemic errors in the sample. The influence of segmental features and participant factors on tool accuracy was analysed. Linear mixed modelling showed significant interaction between group and stress type, surviving adjustment for age and CAS severity. For TD, agreement for SW and WS words was &gt, 80%, but CAS speech was higher for SW (&gt, 80%) than WS (~60%). Prior knowledge of segmental errors conferred no clear advantage. Automatic lexical stress classification shows promise for identifying errors in children’s speech at diagnosis or with treatment-related change, but accuracy for WS words in apraxic speech needs improvement. Further training of algorithms using larger sets of labelled data containing impaired speech and WS words may increase accuracy.

Given the limited accessibility to Speech and Language Pathologists (SLPs) children in need often have, pediatric Computer-Aided Speech Therapy (CAST) tools can play an important role in the early diagnosis and treatment of speech disorders. However, various challenges impede the implementation of accurate automated analysis of speech disorders in children. In this article, we first discuss three key challenges in processing child disordered speech: 1) the unreliability of low-level annotation and scarcity of speech corpora, 2) speaker diarization of therapy sessions and 3) inaccurate children's acoustic models. We next explore opportunities to overcome some of these challenges. First, we investigate the effectiveness of high-level paralinguistic features in disordered speech detection to reduce the dependency on annotated data. A binary classifier trained using paralinguistic features extracted from both typically developing children and those suffering from Speech Sound Disorders (SSD) achieved 87% subject-level classification accuracy. Second, we tackle the speech disorder detection problem as an anomaly detection problem where models are trained merely on typically developing speech, reducing the need for disordered training data. A phoneme-level F 1 score of 0.77 was obtained from an anomaly detection-based system trained on speech attribute features to classify between typical and atypical phoneme pronunciations of children with speech disorder. Finally, we test the efficiency of an x-vector based speaker diarization technique in pediatric therapy sessions. The method successfully distinguished between therapist and child speech with a Diarization Error Rate (DER) of 10%.

Computer aided pronunciation training tools require accurate automatic pronunciation error detection algorithms to identify errors made by their users. However, the performance of these algorithms is highly dependent on the amount of mispronounced speech data used to train them and the reliability of its manual annotation. To overcome this problem, we turned the mispronunciation detection into an anomaly detection problem, which utilize algorithms trained with only correctly pronounced speech data. In this work we adopted the One-Class SVM as our anomaly detection model, with a specific model built for each phoneme. Each model was fed with a set of speech attribute features, namely the manners and places of articulation, extracted from a bank of binary DNN speech attribute detectors. We also applied multi-task learning and dropout approaches to alleviate the overfitting problem in the DNN speech attribute detectors. We trained the system using the WSJ0 and TIMIT standard data sets which contain only native English speech data and then evaluated it using three different data sets, a native English speaker corpus with artificial errors, a foreign-accented speech corpus and a children's disordered speech corpus. Finally, we compared our system with the conventional Goodness-of-Pronunciation (GOP) algorithm to demonstrate the effectiveness of our method. The results show that our method reduced the false-acceptance and false-rejection rates by 26% and 39% respectively compared to the GOP method.

In clinical practice, heart arrhythmias are manually diagnosed by a doctor, which is a time-consuming process. Furthermore, this process is error-prone due to noise from the recording equipment and biological non-idealities of patients. Thus, an automated arrhythmia classifier would be time and cost-effective as well as offer better generalization across patients. In this paper, we propose an adversarial multitask learning method to improve the generalization of heartbeat arrythmia classification. We built an end-to-end deep neural network (DNN) system consisting of three sub-networks; a generator, a heartbeat-type discriminator, and a subject (or patient) discriminator. Each of these two discriminators had its own loss function to control its impact. The generator was "friendly" to the heartbeat-type discrimination task by minimizing its loss function and "hostile" to the subject discrimination task by maximizing its loss function. The network was trained using raw ECG signals to discriminate between five types of heartbeats - normal heartbeats, right bundle branch blocks (RBBB), premature ventricular contractions (PVC), paced beats (PB) and fusion of ventricular and normal beats (FVN). The method was tested with the MIT-BIH arrhythmia dataset and achieved a 17% reduction in classification error compared to a baseline using a fully-connected DNN classifier.Clinical Relevance—This work validates that it is possible to develop a subject-independent automated heart arrhythmia detection system to assist clinicians in the diagnosis process.

Subjective screening of children with speech disorders is costly, time consuming and infeasible due to the limited availability of Speech and Language Pathologists (SLPs). Therefore, there is an increasing interest in automatic speech analysis of children with speech disorders as it can offer a practical alternative to human assessment. Paralinguistic features are a set of low-level descriptors commonly used in speech emotion recognition. However, they have not yet been examined with childhood speech sound disorders such as, apraxia-of-speech and phonological and articulation disorders. In this paper, we investigated the effectiveness of paralinguistic features in discriminating between typically developing children and those who suffer from different types of speech sound disorders. Two types of standard paralinguistic features were explored, the Geneva Minimalistic Acoustic Parameter Set (GeMAPS) and its extended version, (eGeMAPS) feature sets. We applied feature selection to find the most discriminant set of features and employed binary classification using a support vector machine (SVM) to discriminate between the two groups. The method was tested on a recently-released public speech corpus collected from typically developing children and children with various types of speech sound disorders. The system achieved segment-level and subject-level unweighted average recall (UAR) of around 78% and 87% respectively.

The automatic assessment of speech plays an important role in Computer Aided Pronunciation Learning systems. However, modeling both the correct and incorrect pronunciation of each phoneme to achieve accurate pronunciation verification is unfeasible due to the lack of sufficient mispronounced samples in training datasets. In this paper, we propose a novel approach that handles this unbalanced data distribution by building multiple one-class SVMs to evaluate each phoneme as correct or incorrect. We model the correct pronunciation of each individual phoneme with a one-class SVM trained using a set of speech attributes features, namely the manner and place of articulation. These features are extracted from a bank of pre-trained DNN speech attributes classifiers. The one-class SVM model measures the similarity between the new data and the training set and then classifies it as normal (correct) or an anomaly (incorrect). We evaluated the system using native speech corpus and disordered speech corpus and compared it with the conventional Goodness of Pronunciation (GOP) algorithm. The results show that our approach reduces the false-acceptance and false-rejection rates by around 26% and 39% respectively.

We present a multitier system for the remote administration of speech therapy to children with apraxia of speech. The system uses a client-server architecture model and facilitates task-oriented remote therapeutic training in both in-home and clinical settings. The system allows a speech language pathologist (SLP) to remotely assign speech production exercises to each child through a web interface and the child to practice these exercises in the form of a game on a mobile device. The mobile app records the child's utterances and streams them to a back-end server for automated scoring by a speech-analysis engine. The SLP can then review the individual recordings and the automated scores through a web interface, provide feedback to the child, and adapt the training program as needed. We have validated the system through a pilot study with children diagnosed with apraxia of speech, their parents, and SLPs. Here, we describe the overall client-server architecture, middleware tools used to build the system, speech-analysis tools for automatic scoring of utterances, and present results from a clinical study. Our results support the feasibility of the system as a complement to traditional face-to-face therapy through the use of mobile tools and automated speech analysis algorithms.

Automated tool to assess productions from children with apraxia of speech.Consists of clinician interface, mobile application and speech processing engine.Automatically detects groping errors, articulation errors and prosodic errors.Lattice-based Pronunciation Verification module detects articulation errors.Lexical Stress Pattern Verification module detects prosodic errors. Children with developmental disabilities such as childhood apraxia of speech (CAS) require repeated intervention sessions with a speech therapist, sometimes extending over several years. Technology-based therapy tools offer the potential to reduce the demanding workload of speech therapists as well as time and cost for families. In response to this need, we have developed "Tabby Talks," a multi-tier system for remote administration of speech therapy. This paper describes the speech processing pipeline to automatically detect common errors associated with CAS. The pipeline contains modules for voice activity detection, pronunciation verification, and lexical stress verification. The voice activity detector evaluates the intensity contour of an utterance and compares it against an adaptive threshold to detect silence segments and measure voicing delays and total production time. The pronunciation verification module uses a generic search lattice structure with multiple internal paths that covers all possible pronunciation errors (substitutions, insertions and deletions) in the child's production. Finally, the lexical stress verification module classifies the lexical stress across consecutive syllables into strong-weak or weak-strong patterns using a combination of prosodic and spectral measures. These error measures can be provided to the therapist through a web interface, to enable them to adapt the child's therapy program remotely. When evaluated on a dataset of typically developing and disordered speech from children ages 4-16years, the system achieves a pronunciation verification accuracy of 88.2% at the phoneme level and 80.7% at the utterance level, and lexical stress classification rate of 83.3%.

Background and Objectives: As English is a stress-timed language, lexical stress plays an important role in the perception and processing of speech by native speakers. Incorrect stress placement can reduce the intelligibility of the speaker and their ability to communicate more effectively. The accurate identification of lexical stress patterns is thus a key assessment tool of the speaker's pronunciation in applications such as second language (L2) learning, language proficiency testing and speech therapy. With the increasing use of Computer-Aided Language Learning (CALL) and Computer-Aided Speech and Language Therapy (CASLT) tools, the automatic assessment of lexical stress has become an important component of measuring the quality of the speaker's pronunciation. In this work we proposed a Deep Neural Network (DNN) classifier to discriminate between the unequal lexical stress patterns in English words, strong-weak (SW) and weak-strong (WS). The features used in training the deep neural network are derived from the duration, pitch and intensity of each of the two consecutive syllables along with a set of energies of different frequency bands. The robustness of our proposed lexical stress detector has been validated by testing it on the standard TIMIT dataset collected from adult male and female speakers distributed over 8 different dialect regions. Method: Our lexical stress classifier is applied on the speech signal along with the prompted word. Figure 1 shows a block diagram of the overall system. The speech signal is first force aligned with the predetermined phoneme sequence of the word to obtain the time boundaries of each phoneme. The alignment is performed using a Hidden Markov Model (HMM) Viterbi decoder along with set of HMM acoustic models trained from the same corpus to reduce the error caused by inaccurate phone level segmentation. A set of features is then extracted from each syllable and the features of each pair of consecutive syllables combined using the extracted features directly and concatenating them into one wide feature vector.Lexical stress is identified by the variation in the pitch, energy and duration produced between different syllables in a multi-syllabic word. The stressed syllable is characterized by increased energy and pitch as well as a longer duration compared to the other syllables within the same word. Therefore we extracted seven features f1–f7 related to these characteristics as listed in Table 1. The energy based features (f1, f2, f3) were extracted after applying the non-linear Teager energy operator (TEO) on the speech signal to obtain a better estimation of the speech signal energy and reduce the noise effect. These seven features are commonly used in the detection of the stressed syllable in a word. As the speech signal energy is distributed over different frequency bands, we also computed the energy in the Mel-scale frequency bands in each frame of the syllable nucleus. The speech signal was divided into 10 msec non-overlapped frames and the energy, pitch and the frequency bands energies calculated for each frame.As seen in Figure 1, to input the raw extracted features directly to the DNN, we concatenate the extracted features into one wide feature vector. Each syllable has 7 scalar values f1–f7 and 27*n Mel-coefficients where n is the number of frames in each syllable's vowel.To handle variable vowel lengths, we limit the number of input frames provided to the DNN to a maximum N frames for each syllable. This provides the DNN with a fixed length Mel-energy input vector and allows the DNN to use information about the distribution of the Mel-energy bands over the vowel. If the vowel length (n) is greater than N frames, only the middle N frames are used. If the length of the vowel (n) is smaller than N frames, inputs frames are padded to N frames. The final size of the input vector to the DNN is 2*(7+27*N) for a pair of consecutive syllables, with N tuned empirically.The DNN is trained using the mini-batch stochastic gradient decent method (MSGD) with adaptive learning rate. The learning rate starts with an initial value (typically 0.1) and after each epoch the loss in the error of the validation data set is computed. If the loss is greater than zero (i.e. the error increases) the training continues with the same learning rate.If the error continues increasing for 10 consecutive epochs, the learning rate is halved and the parameter of the classifier returned to the one that achieved minimum error. Training is terminated when the learning rate reaches its minimum value (typically 0.0001) or after 200 epochs, whichever is earlier. The performance of the DNN is then computed using a separate testing set. Experiments and Results: We extracted raw features from consecutive syllables belonging to the same word from the TIMIT speech corpus. With the TIMIT corpus, we achieved a minimum error rate of 12.6% using a DNN classifier with 6 hidden layers and 100 hidden units per layer. Due to the unavailability of sufficient male and female data, we were unable to build a separate model for each gender. In Fig. 2, we present the error rate for each gender using a model trained on both male and female data. The results show that the classification of the SW is better in male speakers compared to female speakers while the WS error rate is lower for female speakers. However, the overall misclassification rate for both male and female speakers is almost the same.To study the influence of the dialect on the algorithm, we compared the error rate when testing each dialect using a model trained with the training data of all dialects and when the model was trained with training data from all dialects except the tested one as shown in Fig. 3. As seen, the error rate of most of the dialects remains unchanged except for DR1 where the error rate increased significantly from 4.8 % to 8%. This can be explained by the small amount of test samples for this dialect (only 5% of the test samples). DR4 also shows a considerable increase in the error rate. Although the smallest amount of training samples was from the DR1 (New England) dialect, it produced the lowest error rate among the other dialects. Further work is needed to explain this behavior. Conclusion: In this work we present a DNN classifier to detect bisyllabic lexical stress patterns in multi-syllabic English words. The DNN classifier is trained using a set of features extracted from pairs of consecutive syllables related to pitch, intensity and duration along with energies in different frequency bands. The feature set of each pair of consecutive syllables is combined by concatenating the raw features into one wide vector. When applied on the standard TIMIT adult speech, the algorithm achieved a classification accuracy of 87.4%. The system performance show high stability over different dialects and gender.

Lexical stress is a key diagnostic marker of disordered speech as it strongly affects speech perception. In this paper we introduce an automated method to classify between the different lexical stress patterns in children's speech. A deep neural network is used to classify between strong-weak (SW), weak-strong (WS) and equal-stress (SS/WW) patterns in English by measuring the articulation change between the two successive syllables. The deep neural network architecture is trained using a set of acoustic features derived from pitch, duration and intensity measurements along with the energies in different frequency bands. We compared the performance of the deep neural classifier to a traditional single hidden layer MLP. Results show that the deep neural classifier outperforms the traditional MLP. The accuracy of the deep neural system is approximately 85% when classifying between the unequal stress patterns (SW/WS) and greater than 70% when classifying both equal and unequal stress patterns.

This paper introduces a pronunciation verification method to be used in an automatic assessment therapy tool of child disordered speech. The proposed method creates a phonebased search lattice that is flexible enough to cover all probable mispronunciations. This allows us to verify the correctness of the pronunciation and detect the incorrect phonemes produced by the child. We compare between two different acoustic models, the conventional GMM-HMM and the hybrid DNN-HMM. Results show that the hybrid DNNHMM outperforms the conventional GMM-HMM for all experiments on both normal and disordered speech. The total correctness accuracy of the system at the phoneme level is above 85% when used with disordered speech.

Background and Objectives Childhood Apraxia of Speech (CAS) is a speech disorder characterized by articulation errors, i.e. the replacement of certain phonemes with alternatives. In previous work we proposed a simple method to evaluate the child's speech as correct or incorrect with an overall accuracy of 88.2%. In this work we present an enhanced method that increases the accuracy of the correct/incorrect evaluation to 92.7%, in addition to identifying the incorrect phonemes with an accuracy of 60%. Method The goal of the mispronunciation detection system is to compare each phoneme in the child's production to their given prompt and identify mispronunciations. Figure 1 shows the block diagram of the system, which uses a search lattice for each prompt in the child's speech therapy treatment protocol to identify errors made. Each prompt is transcribed as per the corresponding phoneme sequence using the CMU pronunciation dictionary and then passed to the lattice generator along with the expected mispronunciation rules to generate the search lattice. Mel Frequency Cepstral Coefficients (MFCC) are extracted from the speech signal with delta and acceleration to produce a 39- dimensional feature vector per frame. The extracted features are then fed to the speech recognizer along with the created lattice and the Hidden Markov Model (HMM) acoustic models to generate a sequence of phones from the child's utterance. An evaluation report is then generated by matching the recognized phoneme sequence with the correct phoneme sequence and specifying the errors made by the child. We use a search lattice with a specific number of alternative pronunciations for each phoneme; this limits the decoder search, making it faster and more accurate. Each phoneme in the correct phoneme sequence is compared with expected mispronunciation rules developed by a therapist after an assessment of 20 children with CAS; if a rule is matched, the pronunciation variants are added as alternative arcs to the current phoneme sequence. The mispronunciation rules depend on the type of the phoneme (consonant/vowel), the phoneme position in the word (Initial/Medial/Final) and the context of the phoneme. The lattice is then created using the matched rules as shown in Figure 2, where the garbage model absorbs any mispronounced phoneme not in the lattice. PA and PG are insertion penalties added to the alternative and the garbage arcs respectively so the decoder does not align the speech to the alternative error phonemes or the garbage node unless it is confident enough. Results The system overall system accuracy is 92.7% where the Correct Acceptance (CA) is 97.6% and the Correct Rejection (CR) is 83.1%. The system also detects phoneme errors made by the child with 60% accuracy. Conclusion In this paper we proposed a mispronunciation detection tool that can detect phonemes mispronounced by children with CAS and specify the errors made.

We present a multi-tier system for the remote administration of speech therapy to children with apraxia of speech. The system uses a client-server architecture model and facilitates task-oriented remote therapeutic training in both in-home and clinical settings. Namely, the system allows a speech therapist to remotely assign speech production exercises to each child through a web interface, and the child to practice these exercises on a mobile device. The mobile app records the child's utterances and streams them to a back-end server for automated scoring by a speech-analysis engine. The therapist can then review the individual recordings and the automated scores through a web interface, provide feedback to the child, and adapt the training program as needed. We validated the system through a pilot study with children diagnosed with apraxia of speech, and their parents and speech therapists. Here we describe the overall client-server architecture, middleware tools used to build the system, the speech-analysis tools for automatic scoring of recorded utterances, and results from the pilot study. Our results support the feasibility of the system as a complement to traditional face-to-face therapy through the use of mobile tools and automated speech analysis algorithms.

Background and Objectives Language production and speech articulation can be delayed in children due to developmental disabilities and neuromotor disorders such as childhood apraxia of speech (CAS). One of the behaviors that are commonly associated with the CAS is the articulation errors where the child mispronounced some of the produced phonemes. The presented Pronunciation Verification (PV) method automatically evaluates the child speech and detects any insertion, deletion or substitution errors made by the child on the phoneme level. Method The proposed PV method based on a search lattice with different competing paths to allow the system to detect insertions, deletions and substitutions of phonemes. Fig. 1 shows a block diagram of the lattice based PV component. The prompted word is first phonetically transcribed to obtain the expected phoneme sequence. The lattice generator then uses the phoneme sequence to generate a search lattice fed to the speech recognizer. The generated lattice is flexible enough to cover all the possible pronunciation errors (insertion, deletion and substitution) by adding alternative paths to the correct path for each of the expected errors. The deletion path can be represented as a null arc to allow the recognizer to skip the phoneme node during decoding while the garbage node is used as an alternative to collect phoneme other than the expected one (substitution error). A garbage loop is also added between two consecutive phonemes to collect inserted phonemes frames. Fig. 2 (a) shows an example of the lattice for the word "chair" where PG and PD are the penalties attached to the garbage and deletion arcs respectively, these penalties are added to avoid the recognizer skipping phonemes or aligning speech to the garbage node unless the fit is better than the correct path. The garbage node is composed of all the phonemes connected in parallel as shown in Fig. 2 (b). The Mel Frequency Cepstral Coefficients (MFCC) are extracted from the speech signal with delta and acceleration to produce a 39 dimension feature vector per frame. The extracted features are then fed to the speech recognizer along with the created lattice and the Hidden Markov Model (HMM) acoustic models to generate a sequence of phonemes from the child's utterance. The Context Dependent (CD) HMM model consists of multi-mixture tied-state tri-phones while the garbage model consists of single mixture mono-phones to reduce the complexity and speed up the recognition process. The output phoneme sequence is then compared to the expected phoneme sequence, if matched the utterance is marked as correct otherwise incorrect. Results The system overall accuracy is 88.2% where the Correct Acceptance (CA) is 91.5% and the Correct Rejection (CR) is 83.4%. Conclusion A PV method that uses a search lattice with different alternative paths and a garbage model was used to detect the articulation errors made by the child with overall accuracy around 88%.

Technology based speech therapy systems are severely handicapped due to the absence of accurate prosodic event identification algorithms. This paper introduces an automatic method for the classification of strong-weak (SW) and weak-strong (WS) stress patterns in children speech with American English accent, for use in the assessment of the speech dysprosody. We investigate the ability of two sets of features used to train classifiers to identify the variation in lexical stress between two consecutive syllables. The first set consists of traditional features derived from measurements of pitch, intensity and duration, whereas the second set consists of energies of different filter banks. Three different classifiers were used in the experiments: an Artificial Neural Network (ANN) classifier with a single hidden layer, Support Vector Machine (SVM) classifier with both linear and Gaussian kernels and the Maximum Entropy modeling (MaxEnt). these features. Best results were obtained using an ANN classifier and a combination of the two sets of features. The system correctly classified 94% of the SW stress patterns and 76% of the WS stress patterns.

Background and Objectives: In dysprosodic speech, the prosody does not match the expected intonation pattern and can result in robotic-like speech, with each syllable produced with equal stress. These errors are manifested through inconsistent lexical stress as measured by perceptual judgments and/or acoustic variables. Lexical stress is produced through variations in syllable duration, peak intensity and fundamental frequency. The presented technique automatically evaluates the unequal lexical stress patterns Strong-Weak (SW) and Week-Strong (WS) in American English continuous speech production based upon a multi-layer feed forward neural network with seven acoustic features chosen to target the lexical stress variability between two consecutive syllables. Methods: The speech corpus used in this work is the PTDB-TUG. Five females and three males were chosen to form a training set and one female and one male for testing. The CMU pronouncing dictionary with lexical stress levels marked was used to assign stress levels to each syllable in all words in the speech corpus. Lexical stress is phonetically realized through the manipulation of signal intensity, the fundamental frequency (F0) and its dynamics and the syllable/vowel duration. The nucleus duration, syllable duration, mean pitch, maximum pitch over nucleus, the peak-to-peak amplitude integral over syllable nucleus, energy mean and maximum energy over nucleus were calculated for each syllable in the collected speech. As lexical stress errors are identified by evaluating the variability between consecutive syllables in a word, we computed the pairwise variability index ("PVI") for each acoustic measure. The PVI for any acoustic feature f_i is given by: PVI_i= (f_i1-f_i2)/(( f_i1+f_i2)/2)(1), where f_i1,f_i2 are the acoustic features of the first and second syllables consecutively. A multi-layer feed forward neural network which consisted of input, hidden and output layers was used to classify the stress patterns in the words in the database. Results: The presented system had an overall accuracy of 87.6%. It correctly classified 92.4% of the SW stress patterns and 76.5% of the WS stress pattern. Conclusions: A feed-forward neural network was used to classify between the SW and WS stress patterns in American English continuous speech with overall accuracy around 87 percent.