Search

Les avancées récentes dans l’apprentissage auto-supervisé ont permis aux systèmes de reconnaissance automatique de la parole (ASR) d’atteindre l’etat de l’art des taux d’erreurs de reconnaissance de mots (WER) tout en ne nécessitant qu’une fraction des données étiquetées nécessaires à leurs prédécesseurs. Néanmoins, bien que de tels modèles atteignent des résultats de pointe dans des scénarios d’entraînement/test correspondants, leurs performances se dégradent considérablement lorsqu’ils sont testés dans des conditions non vues. Pour palier à ce problème, des stratégies telles que l’augmentation de données et/ou l’adaptation au domaine ont été explorées. Cependant, les modèles disponibles sont encore trop volumineux pour être considérés pour des applications vocales sur des appareils aux ressources limitées ; ainsi, des outils de compression de modèle tels que l’élagage de paramètres, la quantification de poids et la distillation de connaissances sont nécessaires. Dans notre étude sur l’apprentissage de la représentation de la parole auto-supervisée (S3RL), nous abordons d’abord les effets de non-correspondance d’entraînement/test sur les modèles compressés, en investiguant l’impact de la quantification des paramètres et de l’élagage du modèle sur le modèle robust wav2vec 2.0 dans des conditions bruyantes, réverbérantes et mixtes. De plus, nous améliorons la recette de distillation DistilHuBERT avec des têtes de prédiction optimisées, un enrichissement du jeu de données spécifique pour différents environnements, et un estimateur d’environnement en temps réel pour la sélection du modèle lors de l’inférence. Des expériences sur l’ensemble de données LibriSpeech, corrompu par différents niveaux de bruit et de réverbération, démontrent qu’une diversité de données pendant l’entraînement renforce la robustesse du modèle contre la compression, le bruit et la réverbération. Notre méthode proposée surpassent les modèles de références jusqu’à 48,4% et 89,2% dans le taux

As well known, the huge memory and compute costs of both artificial neural networks (ANNs) and spiking neural networks (SNNs) greatly hinder their deployment on edge devices with high efficiency. Model compression has been proposed as a promising technique to improve the running efficiency via parameter and operation reduction, whereas this technique is mainly practiced in ANNs rather than SNNs. It is interesting to answer how much an SNN model can be compressed without compromising its functionality, where two challenges should be addressed: 1) the accuracy of SNNs is usually sensitive to model compression, which requires an accurate compression methodology and 2) the computation of SNNs is event-driven rather than static, which produces an extra compression dimension on dynamic spikes. To this end, we realize a comprehensive SNN compression through three steps. First, we formulate the connection pruning and weight quantization as a constrained optimization problem. Second, we combine spatiotemporal backpropagation (STBP) and alternating direction method of multipliers (ADMMs) to solve the problem with minimum accuracy loss. Third, we further propose activity regularization to reduce the spike events for fewer active operations. These methods can be applied in either a single way for moderate compression or a joint way for aggressive compression. We define several quantitative metrics to evaluate the compression performance for SNNs. Our methodology is validated in pattern recognition tasks over MNIST, N-MNIST, CIFAR10, and CIFAR100 datasets, where extensive comparisons, analyses, and insights are provided. To the best of our knowledge, this is the first work that studies SNN compression in a comprehensive manner by exploiting all compressible components and achieves better results.

Producción Científica, A key requirement for RRAM in neural network accelerators with a large number of synaptic parameters is the multilevel programming. This is hindered by resistance imprecision due to cycle-to-cycle and device-to-device variations. Here, we compare two multilevel programming algorithms to minimize resistance variations in a 4-kbit array of HfO 2 RRAM. We show that gate-based algorithms have the highest reliability. The optimized scheme is used to implement a neural network with 9-level weights, achieving 91.5% (vs. software 93.27%) in MNIST recognition.

Due to controllable conductance and non-volatility, flexible memristors are regarded as a key enabler for building artificial neural network (ANN)-based learning algorithms in flexible and wearable systems. However, the existing flexible memristors are suffering from limited number of conductance values, issues limiting large-scale integration, and insufficient accuracy that cannot support accurate computation of ANN. In this paper, solutions are proposed for the three major challenges of the flexible memristor; the feasibility of a three-layer fully connected neural network on MNIST and a 13-layer convolutional neural network (CNN) on CIFAR-10 using the flexible memristor based on single-walled carbon nanotubes network/polymer composite and hydrophilic Al2O3 dielectric are studied. The evaluation result shows that in the fully connected neural network system, it is able to recognize MNIST with an accuracy above 90% after 4-bit quantization, 52.05% decrease in interconnection numbers in the circuit and up to 40% random error introduced, and in the CNN on CIFAR-10, the system can retain an accuracy above 86% with less than 4% accuracy loss after 5-bit quantization, 59.34% decrease in interconnection numbers in the circuit and up to 40% random error injected., QC 20180904

Compact architectures, ternary weights and binary activations are two methods suitable for making neural networks more efficient. We introduce a) a dithering binary activation which improves accuracy of ternary weight networks with binary activations by randomizing quantization error, and b) a method of implementing ternary weight networks with binary activations using binary operations. Despite these new approaches, training a compact SqueezeNet architecture with ternary weights and full precision activations on ImageNet degrades classification accuracy significantly more than when training a less compact architecture the same way. Therefore ternary weights in their current form cannot be called the best method for reducing network size. However, the effect of weight decay on ternary weight network training should be investigated more in order to have more certainty in this finding., Kompakta arkitekturer, ternära vikter och binära aktiveringar är två metoder som är lämpliga för att göra neurala nätverk effektivare. Vi introducerar a) en dithering binär aktivering som förbättrar noggrannheten av ternärviktsnätverk med binära aktiveringar genom randomisering av kvantiseringsfel, och b) en metod för genomförande ternärviktsnätverk med binära aktiveringar med användning av binära operationer. Trots dessa nya metoder, att träna en kompakt SqueezeNet-arkitektur med ternära vikter och fullprecisionaktiveringar på ImageNet försämrar klassificeringsnoggrannheten betydligt mer än om man tränar en mindre kompakt arkitektur på samma sätt. Därför kan ternära vikter i deras nuvarande form inte kallas bästa sättet att minska nätverksstorleken. Emellertid, effekten av weight decay på träning av ternärviktsnätverk bör undersökas mer för att få större säkerhet i detta resultat.

Compact architectures, ternary weights and binary activations are two methods suitable for making neural networks more efficient. We introduce a) a dithering binary activation which improves accuracy of ternary weight networks with binary activations by randomizing quantization error, and b) a method of implementing ternary weight networks with binary activations using binary operations. Despite these new approaches, training a compact SqueezeNet architecture with ternary weights and full precision activations on ImageNet degrades classification accuracy significantly more than when training a less compact architecture the same way. Therefore ternary weights in their current form cannot be called the best method for reducing network size. However, the effect of weight decay on ternary weight network training should be investigated more in order to have more certainty in this finding., Kompakta arkitekturer, ternära vikter och binära aktiveringar är två metoder som är lämpliga för att göra neurala nätverk effektivare. Vi introducerar a) en dithering binär aktivering som förbättrar noggrannheten av ternärviktsnätverk med binära aktiveringar genom randomisering av kvantiseringsfel, och b) en metod för genomförande ternärviktsnätverk med binära aktiveringar med användning av binära operationer. Trots dessa nya metoder, att träna en kompakt SqueezeNet-arkitektur med ternära vikter och fullprecisionaktiveringar på ImageNet försämrar klassificeringsnoggrannheten betydligt mer än om man tränar en mindre kompakt arkitektur på samma sätt. Därför kan ternära vikter i deras nuvarande form inte kallas bästa sättet att minska nätverksstorleken. Emellertid, effekten av weight decay på träning av ternärviktsnätverk bör undersökas mer för att få större säkerhet i detta resultat.