Search

MEng (Computer and Electronic Engineering), North-West University, Potchefstroom Campus In this study, we investigate the use of generative adversarial networks (GANs) to improve speech recognition performance of poor quality audio obtained from a real-world source. A GAN is developed to transform acoustic features of noisy audio prior to downstream acoustic modelling. The system utilises a baseline acoustic model trained on good quality data to improve the performance on mismatched data. This is achieved without requiring manual creation of parallel datasets. The practical relevance of the GAN is realised when a strong commercial-grade speech recognition system { which has already been optimised for a given set of conditions { is required to decode new mismatched data. The GAN can then act as a front-end to the existing system. We compare the GAN-based front-end to multi-style training (MTR) on three datasets in a controlled environment. The GAN system is much faster to train than a comparable MTR system with similar performance. The developed GAN is applied to a South African call centre dataset and achieves consistent improvements over a baseline model. Therefore, this provides a practical approach to improve ASR systems in mismatched environments. Masters

MEng (Computer en Electronic Engineering), North-West University, Potchefstroom Campus The generalisation ability of deep neural networks differs somewhat from that of more traditional models. Speciőcally, large networks that have the ability to łmemorisež the training data can still generalise well, regardless of regularisation. This generalisation ability is still not completely understood. As part of a larger approach to studying the generalisation ability of neural networks by measuring the utilisation of training data, we aim to őnd a method of measuring mutual information and redundancy, speciőcally in the context of information processing in neural networks. To this end, we study various existing redundancy estimators, and, using these as inspiration, we develop a new, nearest-neighbours-based redundancy estimator that can be used with discrete-continuous mixture distributions. We evaluate this new estimator on synthetically generated data and compare its behaviour to that of existing estimators. As a demonstration of the use of this estimator in neural network analysis, we calculate various node-based mutual information estimates in fully connected, feedforward networks trained for classiőcation. Our demonstration reveals interesting regularities and differences between networks with different generalisation characteristics. Overall, we implement and evaluate several redundancy estimators and show that node-based redundancy estimates can be used to analyse neural networks. Masters

MEng (Computer Engineering), North-West University, Potchefstroom Campus Speaker diarisation systems aim to answer the question \who spoke when?" and are useful in providing valuable metadata to downstream applications, such as automatic speech recognition systems. However, speaker diarisation systems, like most applications in the field of speech recognition, are especially challenged by domain-mismatch conditions. In this study, we investigate methods with which to adapt a pre-trained diarisation system to a new target domain when only a small in-domain corpus is available and retraining is therefore not an option. We also develop a method for fine-tuning the adaptation process of a pre-trained speaker diarisation system using cluster analysis. Our domain adaptation process focuses on retraining and adapting the statistical components in a speaker diarisation pipeline, which are inherently domain specific, to the target domain. Lastly, we demonstrate this domain adaptation process in a real-world scenario by adapting a pre-trained diarisation system using a small in-domain dataset consisting of telephonic speech from South African call centres. We show that the adapted system can be used to provide metadata which aids the performance of automatic speech recognition systems through speaker-specific adaptations. Masters

MEng (Computer and Electronic Engineering), North-West University, Potchefstroom Campus Despite their impressive performances on a range of widespread tasks, deep neural networks (DNNs) are generally considered `black box' models due to the lack of transparency behind their decision-making processes. Researchers address this issue through the use of interpretability techniques which, in the context of this study, uses some set of rules to map the output of the network back onto its inputs. In recent works, sample set analysis has been proposed as a novel methodology to better study the generalisation capabilities of DNNs through analysing the natural sample clusters formed by the network itself. By being able to directly identify the nodes that process the largest number of class samples, this methodology does o er some potential as a possible means for improving DNN interpretations. In this exploratory study, we investigate the applicability of sample set analysis as a tool for DNN interpretability purposes. We do this by analysing the inner workings of networks trained on the MNIST data set through using sample set analysis in conjunction with the Layer-wise Relevance Propagation (LRP) interpretability technique, while verifying the results using a custom generated synthetic data set. Our analysis led to the introduction of encoding sample sets, an additional sample set category that groups class samples according to their binary node activation patterns in a given layer. Through encoding sample sets, we further introduce the concepts of core and variation nodes, which refer to the nodes that activates for all encoding sample sets within a layer or only a subset of them, respectively. When used in conjunction with LRP, encoding sample sets are capable of generating interpretations which represent groups of samples rather then representing them individually. We coined this approach set interpretations and found that it provides interpretations highly similar to its individual counterparts while simplifying the interpretation process. Masters

MEng (Computer and Electronic Engineering), North-West University, Potchefstroom Campus Deep neural networks (DNNs) have shown impressive performance on a wide variety of applications, but it remains di cult to interpret these models. For regression modelling, DNNs generally do not explicitly provide any information about the utility of each of the input parameters in terms of their contribution to model accuracy. With this in mind, we develop the pairwise network, an adaptation to the multilayer perceptron (MLP) that allows the ranking of input parameters according to their contribution to model output. The technique is developed using synthetic data, before its application is demonstrated in the context of a space physics problem. Geomagnetic storms are multi-day events characterised by signi cant perturbations to the magnetic eld of the Earth, driven by solar activity. Previous storm forecasting e orts typically use solar wind measurements as input parameters to a regression problem tasked with predicting a perturbation index such as the 1-minute cadence symmetric-H (SYM-H) index. We re-visit the task of predicting SYM-H from solar wind parameters, with two `twists': (i) Geomagnetic storm phase information is incorporated as model inputs and shown to increase prediction performance. (ii) We describe the pairwise network structure and training process { rst validating ranking ability on synthetic data, before using the network to analyse the SYM-H problem. We found that the proposed pairwise network achieved slightly better performance than an MLP on this task while providing feature rankings that correspond to our understanding of the underlying physics. Masters

PhD (Computer and Electronic Engineering), North-West University, Potchefstroom Campus We present an investigation of how simple artificial neural networks (specifcally, feed-forward networks with full connections between each successive pair of layers) generalize to out-of-sample data. By emphasizing the substructures formed within these networks we are able to shed light on several phenomena and relevant open questions in the literature. Specifically, we show that hidden units with piecewise linear activation functions are optimized on the train set in a distributed manner, meaning each sub-unit is only optimized to reduce the loss of a specific sub-population of the train set. This mechanism gives rise to a type of modularity that is not often considered in investigations of artificial neural networks and generalization. We are able to uncover informative regularity in sub-unit behavior and elucidate known phenomena such as: different artificial neural networks tend to prioritize similar samples, over-parametization does not necessarily lead to poor generalization, artificial neural networks are able to interpolate large amounts of noise and still generalize appropriately, and generalization error as a function of representational capacity undergoes a second descent beyond the point of interpolation (a.k.a the double descent phenomenon). We motivate a perspective of generalization in deep learning that is less focused on the complexity of hypothesis spaces, and looks to substructures and the manner by which training data is compartmentalized as a method of understanding the observed ability of these networks to generalize. This perspective contradicts classical ideas of generalization and complexity under certain conditions. Doctoral

MEng (Computer and Electronic Engineering), North-West University, Potchefstroom Campus Although Convolutional Neural Networks (CNNs) are widely used, their translation in-variance (ability to deal with translated inputs) is still subject to some controversy. We explore this question using translation-sensitivity maps to quantify how sensitive a standard CNN is to a translated input. We propose the use of cosine similarity as sensitivity metric over Euclidean distance, and discuss the importance of restricting the dimensionality of either of these metrics when comparing architectures. Our main focus is to investigate the effect of different architectural components of a stan-dard CNN on that network’s sensitivity to translation. To study the effects of max-pool kernel size on translation invariance, we train several CNN architectures with differently shaped max-pool kernels and compare their translation invariance. The results indicate that larger max-pool kernels result in more translation invariance than smaller max-pool kernels. By varying convolutional kernel sizes and amounts of zero padding, we control the size of the feature maps produced, allowing us to quantify the extent to which these elements influence translation invariance. We also measure translation invariance at different loca-tions within the CNN to determine the extent to which convolutional and fully connected layers, respectively, contribute to the translation invariance of a CNN as a whole. Our analysis indicates that both convolutional kernel size and feature map size have a system-atic influence on translation invariance. We also see that convolutional layers contribute less than expected to translation invariance, when not specifically forced to do so. The effects of various CNN components on distortion-sensitivity is also analysed in this study. We analyse the differences between how CNNs deal with translation and distortion. Using distortion-sensitivity functions that we define, we are able to quantify how sensitive a system is to distorted inputs. The results indicate that larger max-pool kernels result in more distortion-sensitivity for CNNs trained on MNIST, similar to translation invariance. Convolutional kernel size has less of an effect on distortion sensitivity for CNNs trained in MNIST while all networks, regardless of their architectural variations, learn to be less sensitive to distortion when trained on CIFAR10. All in all, it seems that convolutional layers are not fully utilised to deal with spatial information if the training task is not difficult enough, forcing the fully connected layers (spatially unaware) to compensate by learning similar elements at different inputs. By reducing feature map size to 1, forcing the convolutional layers to better deal with trans-lation, we obtain the most translation invariant system studied, when evaluated on the unseen test set. Also, we observe a few similarities in how CNNs deal with translation and distortion and attribute these similarities to the network’s ability to pinpoint impor-tant features in general, rather than specifically the movement of kernel across the input during convolution. Masters

MEng (Computer and Electronic Engineering), North-West University, Potchefstroom Campus Convolutional Neural Networks (CNNs) have become the standard for image classification tasks, however, they are not completely invariant to transformations of the input image. We empirically investigate to which degree CNNs can handle transformed input images, and also compare their abilities to multilayer perceptrons (MLPs) and spatial transformer networks (STNs). We measure invariance to three affine transformations, namely: translation, rotation and scale; and specifically focus on translation. The lack of translation invariance in CNNs is attributed to the use of stride which sub-samples the input, resulting in a loss of information, and fully connected layers which lack spatial reasoning. We first theoretically show that stride can greatly benefit translation invariance given that it is combined with sufficient similarity between neighbouring pixels, a characteristic which we refer to as local homogeneity. We then empirically verify this hypothesis, and also observe that this characteristic is dataset-specific, which dictates the required relationship between pooling kernel size and stride for translation invariance. Furthermore we find that a trade-off exists between generalization and translation invariance in the case of pooling kernel size and stride, as larger kernel sizes and strides lead to better invariance but poorer generalization. We then compare the translation, scale, and rotation invariance of CNNs to STN-CNNs and MLPs. As expected, we find that MLPs fair far worse than CNNs and STN-CNNs in terms of transformation invariance and generalization. We find that STNs can improve the transformation invariance of a CNN architecture, given that it is exposed to enough transformed samples during the training process. Furthermore, we observe that without explicit regularization, STNs do not provide any benefits over CNNs in terms of generalization ability. Masters

MEng (Computer and Electronic Engineering), North-West University, Potchefstroom Campus The ability of machine learning algorithms to generalize is arguably their most important aspect as it determines their ability to perform appropriately on unseen data. The impres-sive generalization abilities of deep neural networks (DNNs) are not yet well understood. In particular, the influence of activation functions on the learning process has received limited theoretical attention, even though phenomena such as vanishing gradients, node saturation and sparsity have been identified as possible contributors when comparing different activation functions. In this study, we present findings based on a comparison of several DNN architectures trained with two popular activation functions, and investigate the effect of these ac-tivation functions on training and generalization. We aim to determine the principal factors that contribute towards the superior generalization performance of rectified linear networks when compared with sigmoidal networks. We investigate these factors using fully-connected feedforward networks trained on three standard benchmark tasks. We find that the most salient differences between networks trained with these activation functions relate to the way in which class-distinctive information is separated and prop-agated through the network. We find that the behavior of nodes in ReLU and sigmoidal networks shows similar regularities in some cases. We also find that there are relationships in the ability of hidden layers to accurately use the information available to them and the capacity (specifically depth and width) of the models. The study contributes towards open questions regarding the generalization performance of deep neural networks, specif-ically giving an informed perspective on the role of two historically popular activation functions. Masters

Code switching (CS) is a natural phenomenon that is often observed in multilingual speakers. These speakers use words, phrases or sentences from foreign languages and embed them in sentences in the primary language. Automatic speech recognition (ASR) systems find code-switched speech difficult to process, and ASR performance is known to degrade in CS environments. We study the Sepedi/English CS phenomenon in the context of Sepedi ASR. Using experimentation, data collection and quantitative data analysis, we analyse techniques that can be used to effectively model code-switched speech in resource-scarce environments. The focus is on techniques that modify the pronunciation dictionary, in order to improve recognition accuracy. For this purpose, three new speech resources are designed, collected and curated: (1) the Radio Broadcast Corpus contains real examples of code-switching as observed during radio broadcasts; (2) the Sepedi Prompted Code-Switched (SPCS) Corpus is based on true code switching prompts, with each individual prompt recorded by multiple speakers in order to capture pronunciation variability occurring in code-switched speech; and (3) the National Center for Human Language Technology (NCHLT) Sepedi-English code switched subset (NSECSS) corpus does not contain naturally occurring code-switched speech, but rather English as spoken by Sepedi speakers. The latter corpus is particularly useful as its recording conditions and format match two related corpora: English produced by English speakers and Sepedi produced by Sepedi speakers. As part of corpus development, resource collection and analysis tools were developed and evaluated. Utilising these corpora, the implications of code-switched speech for ASR systems were evaluated. Various approaches to pronunciation modelling of code-switched speech were investigated and a novel method for pronunciation prediction developed. This new variant selection approach to modelling code-switched speech requires a two-step process: after grapheme-to-phoneme prediction of foreign words, phoneme-to-phoneme prediction (mapping the foreign phonemes to in-language phonemes) does not only take phoneme identity into account, but also graphemic context. A practical implementation of such an algorithm performed well during recognition experiments, both as a single approach and in combination with other existing approaches. The best overall results were obtained when multiple variants were generated per CS word, and variant-selection included in this process. Even though specifically applied to the Sepedi/English task, the methods themselves are language-independent. In addition, the methods, frequency of and reasons for code switching observed among Sepedi speakers were studied using corpus analysis. Among other results, it was found that the prevalence of code switching within naturally occurring Sepedi speech was much higher than initially anticipated, making this a task well worth studying. Doctoral

The ability to predict the pronunciation of proper names is of importance to speech recognition applications that utilise names, such as directory enquiry systems. One of the factors that has been shown to improve the modelling of proper names, is the ability to identify the language of origin of a particular name. Proper names present specific challenges, which typically result in poor language identification (LID) accuracy: they are short, can be spelled in idiosyncratic ways and may have multiple language origins. In South Africa, the difficulty of identifying the language of origin of a name is exacerbated by two factors: co-existence of multiple languages and the scarcity of resources for model training. In this thesis, we first investigate existing LID approaches applicable to words in isolation, specifically focusing on those techniques that have been identified to produce high accuracy when resources are limited. We assess the strengths and weaknesses of existing LID techniques when applied to generic words and highlight various factors that influence the performance accuracy with which the language of individual words can be classified. A novel approach to LID of isolated words is then developed using an existing pronunciation modelling technique. Specifically, the LID task is recast as a pronunciation modelling task, and ‘joint sequence models’ are applied to obtain accurate single-word predictions. We evaluated the algorithm and found that the approach outperformed other conventional LID techniques in terms of identification accuracy, with low training data requirements. The results show that this new approach is able to reach identification accuracies greater than 97% on generic words. Given that suitable corpora for South African names were not available prior to the study, we developed two corpora as part of this work: the ‘Southern African corpus for multilingual name pronunciation’ (Multipron corpus) contains names in four languages (Afrikaans, English, Sesotho and isiZulu) as produced by speakers of the particular languages; the ‘South African directory enquiry’ (SADE) corpus contains a wide variety of names produced in a directory enquiries system, produced by speakers of the same four languages as above. When applying this technique to the above corpora, one finds that LID of proper names is a difficult task, but identification accuracy of over 80% was still obtained. In practice, there are cases where words belong to more than one language of origin. This has not been studied extensively (for either generic words or proper names), even though it is of practical importance. We investigate the ability of the proposed technique to perform LID of multilingual words, specifically for under-resourced languages. This thesis concludes by investigating the implications of LID of proper names for pronunciation prediction by analysing G2P accuracy of dictionaries developed using the auto-generated LID information, as well as the recognition accuracy of an automatic speech recognition system developed using these dictionaries. We define a new G2P performance metric – bilateral VPA – which deals with variants in a way that is conceptually more consistent than existing performance metrics. We show that the new G2P accuracy measure correlates well with the ASR results observed. Based on an analysis of different approaches to dictionary creation, we provide guidelines for incorporating LID information during pronunciation modelling of proper names. Doctoral