Search

Superficially, read and spontaneous speech—the two main kinds of training data for automatic speech recognition—appear as complementary, but are equal: pairs of texts and acoustic signals. Yet, spontaneous speech is typically harder for recognition. This is usually explained by different kinds of variation and noise, but there is a more fundamental deviation at play: for read speech, the audio signal is produced by recitation of the given text, whereas in spontaneous speech, the text is transcribed from a given signal. In this review, we embrace this difference by presenting a first introduction of causal reasoning into automatic speech recognition, and describing causality as a tool to study speaking styles and training data. After breaking down the data generation processes of read and spontaneous speech and analysing the domain from a causal perspective, we highlight how data generation by annotation must affect the interpretation of inference and performance. Our work discusses how various results from the causality literature regarding the impact of the direction of data generation mechanisms on learning and prediction apply to speech data. Finally, we argue how a causal perspective can support the understanding of models in speech processing regarding their behaviour, capabilities, and limitations. [ABSTRACT FROM AUTHOR]

Automated construction of location graphs is instrumental but challenging, particularly in logistics optimisation problems and agent-based movement simulations. Hence, we propose an algorithm for automated construction of location graphs, in which vertices correspond to geographic locations of interest and edges to direct travelling routes between them. Our approach involves two steps. In the first step, we use a routing service to compute distances between all pairs of L locations, resulting in a complete graph. In the second step, we prune this graph by removing edges corresponding to indirect routes, identified using the triangle inequality. The computational complexity of this second step is O (L 3) , which enables the computation of location graphs for all towns and cities on the road network of an entire continent. To illustrate the utility of our algorithm in an application, we constructed location graphs for four regions of different size and road infrastructures and compared them to manually created ground truths. Our algorithm simultaneously achieved precision and recall values around 0.9 for a wide range of the single hyperparameter, suggesting that it is a valid approach to create large location graphs for which a manual creation is infeasible. [ABSTRACT FROM AUTHOR]

In this theory paper, we investigate training deep neural networks (DNNs) for classification via minimizing the information bottleneck (IB) functional. We show that the resulting optimization problem suffers from two severe issues: First, for deterministic DNNs, either the IB functional is infinite for almost all values of network parameters, making the optimization problem ill-posed, or it is piecewise constant, hence not admitting gradient-based optimization methods. Second, the invariance of the IB functional under bijections prevents it from capturing properties of the learned representation that are desirable for classification, such as robustness and simplicity. We argue that these issues are partly resolved for stochastic DNNs, DNNs that include a (hard or soft) decision rule, or by replacing the IB functional with related, but more well-behaved cost functions. We conclude that recent successes reported about training DNNs using the IB framework must be attributed to such solutions. As a side effect, our results indicate limitations of the IB framework for the analysis of DNNs. We also note that rather than trying to repair the inherent problems in the IB functional, a better approach may be to design regularizers on latent representation enforcing the desired properties directly. [ABSTRACT FROM AUTHOR]

In 1959, Rényi proposed the information dimension and the $d$ -dimensional entropy to measure the information content of general random variables. This paper proposes a generalization of information dimension to stochastic processes by defining the information dimension rate as the entropy rate of the uniformly quantized stochastic process divided by minus the logarithm of the quantizer step size $1/m$ in the limit as $m\to \infty $. It is demonstrated that the information dimension rate coincides with the rate-distortion dimension, defined as twice the rate-distortion function $R(D)$ of the stochastic process divided by $-\log (D)$ in the limit as $D\downarrow 0$. It is further shown that among all multivariate stationary processes with a given (matrix-valued) spectral distribution function (SDF), the Gaussian process has the largest information dimension rate and the information dimension rate of multivariate stationary Gaussian processes is given by the average rank of the derivative of the SDF. The presented results reveal that the fundamental limits of almost zero-distortion recovery via compressible signal pursuit and almost lossless analog compression are different in general. [ABSTRACT FROM AUTHOR]

The generator matrices of polar codes and Reed-Muller codes are submatrices of the Kronecker product of a lower-triangular binary square matrix. For polar codes, the submatrix is generated by selecting rows according to their Bhattacharyya parameter, which is related to the error probability of sequential decoding. For Reed-Muller codes, the submatrix is generated by selecting rows according to their Hamming weight. In this work, we investigate the properties of the index sets selecting those rows, in the limit as the blocklength tends to infinity. We compute the Lebesgue measure and the Hausdorff dimension of these sets. We furthermore show that these sets are finely structured and self-similar in a well-defined sense, i.e., they have properties that are common to fractals. [ABSTRACT FROM AUTHOR]

Finite precision approximations of discrete probability distributions are considered, applicable for distribution synthesis, e.g., probabilistic shaping. Two algorithms are presented that find the optimal $M$ -type approximation Q$ of a distribution P$ in terms of the variational distance and the informational divergence $D(Q\Vert P)$ . Bounds on the approximation errors are derived and shown to be asymptotically tight. Several examples illustrate that the variational distance optimal approximation can be quite different from the informational divergence optimal approximation. [ABSTRACT FROM PUBLISHER]

The information loss in deterministic, memoryless systems is investigated by evaluating the conditional entropy of the input random variable given the output random variable. It is shown that for a large class of systems the information loss is finite, even if the input has a continuous distribution. For systems with infinite information loss, a relative measure is defined and shown to be related to Rényi information dimension. As deterministic signal processing can only destroy information, it is important to know how this information loss affects the solution of inverse problems. Hence, we connect the probability of perfectly reconstructing the input to the information lost in the system via Fano-type bounds. The theoretical results are illustrated by example systems commonly used in discrete-time, nonlinear signal processing and communications. [ABSTRACT FROM AUTHOR]

The approximation of a discrete probability distribution t by an M-type distribution p is considered. The approximation error is measured by the informational divergence D(t‖p), which is an appropriate measure, e.g., in the context of data compression. Properties of the optimal approximation are derived and bounds on the approximation error are presented, which are asymptotically tight. A greedy algorithm is proposed that solves this M-type approximation problem optimally. Finally, it is shown that different instantiations of this algorithm minimize the informational divergence D(p‖t) or the variational distance ‖p-t‖1. [ABSTRACT FROM AUTHOR]

Acquisition is a search in two continuous dimensions, where the digital algorithms require a partitioning of the search space into cells. Depending on the partitioning of the Doppler frequency domain, more than one cell might contain significant signal energy. We present an expression for the expected values of the cells' energies to analyze the impact of the Doppler bin width on detection and false alarm probabilities. [ABSTRACT FROM AUTHOR]

Information plane analysis, describing the mutual information between the input and a hidden layer and between a hidden layer and the target over time, has recently been proposed to analyze the training of neural networks. Since the activations of a hidden layer are typically continuous-valued, this mutual information cannot be computed analytically and must thus be estimated, resulting in apparently inconsistent or even contradicting results in the literature. The goal of this paper is to demonstrate how information plane analysis can still be a valuable tool for analyzing neural network training. To this end, we complement the prevailing binning estimator for mutual information with a geometric interpretation. With this geometric interpretation in mind, we evaluate the impact of regularization and interpret phenomena such as underfitting and overfitting. In addition, we investigate neural network learning in the presence of noisy data and noisy labels. [ABSTRACT FROM AUTHOR]

Acknowledging the issues in [[12]], these NNs are trained to minimize an upper bound on HT ht , combining variational and non-parametric approaches for bounding. In this case, it is known that sweeping the paramter HT ht in (1) is not sufficient to sweep the Pareto frontier of optimal HT ht pairs [[12]]. The authors show that this shortcoming can be removed by optimizing HT ht instead, where I u i is a strictly convex function. [Extracted from the article]

In this short note, we relate the variational bounds proposed in Alemi et al. (2017) and Fischer (2020) for the information bottleneck (IB) and the conditional entropy bottleneck (CEB) functional, respectively. Although the two functionals were shown to be equivalent, it was empirically observed that optimizing bounds on the CEB functional achieves better generalization performance and adversarial robustness than optimizing those on the IB functional. This work tries to shed light on this issue by showing that, in the most general setting, no ordering can be established between these variational bounds, while such an ordering can be enforced by restricting the feasible sets over which the optimizations take place. The absence of such an ordering in the general setup suggests that the variational bound on the CEB functional is either more amenable to optimization or a relevant cost function for optimization in its own regard, i.e., without justification from the IB or CEB functionals. [ABSTRACT FROM AUTHOR]

Semiconductor manufacturing is a highly innovative branch of industry, where a high degree of automation has already been achieved. For example, devices tested to be outside of their specifications in electrical wafer test are automatically scrapped. In this paper, we go one step further and analyze test data of devices still within the limits of the specification, by exploiting the information contained in the analog wafermaps. To that end, we propose two feature extraction approaches with the aim to detect patterns in the wafer test dataset. Such patterns might indicate the onset of critical deviations in the production process. The studied approaches are: 1) classical image processing and restoration techniques in combination with sophisticated feature engineering and 2) a data-driven deep generative model. The two approaches are evaluated on both a synthetic and a real-world dataset. The synthetic dataset has been modeled based on real-world patterns and characteristics. We found both approaches to provide similar overall evaluation metrics. Our in-depth analysis helps to choose one approach over the other depending on data availability as a major aspect, as well as on available computing power and required interpretability of the results. [ABSTRACT FROM AUTHOR]

We present an information-theoretic cost function for co-clustering, i.e., for simultaneous clustering of two sets based on similarities between their elements. By constructing a simple random walk on the corresponding bipartite graph, our cost function is derived from a recently proposed generalized framework for information-theoretic Markov chain aggregation. The goal of our cost function is to minimize relevant information loss, hence it connects to the information bottleneck formalism. Moreover, via the connection to Markov aggregation, our cost function is not ad hoc, but inherits its justification from the operational qualities associated with the corresponding Markov aggregation problem. We furthermore show that, for appropriate parameter settings, our cost function is identical to well-known approaches from the literature, such as “Information-Theoretic Co-Clustering” by Dhillon et al. Hence, understanding the influence of this parameter admits a deeper understanding of the relationship between previously proposed information-theoretic cost functions. We highlight some strengths and weaknesses of the cost function for different parameters. We also illustrate the performance of our cost function, optimized with a simple sequential heuristic, on several synthetic and real-world data sets, including the Newsgroup20 and the MovieLens100k data sets. [ABSTRACT FROM AUTHOR]

Many concepts in digital signal processing are intuitive, despite being mathematically challenging. The lecturer not only has to teach the complicated math but should also help students develop intuition about the concept. To aid the lecturer in this task, the Magnitude Response Learning Tool has been introduced, a computer-based learning game addressing the connection between the pole/zero chart of a discrete-time linear time-invariant (LTI) system and its magnitude response. In each game round, the student has to draw a magnitude response given only the poles and zeros of the LTI system. The accuracy of the student's estimate acts as a game score, allowing leaderboards and player-versus-player comparisons on the tool's webpage. This paper analyzes how the tool affects student performance. While the qualitative results show that the students are satisfied with the tool and consider it a useful supplement to the available teaching material, the quantitative part of the study suggests a strong positive correlation between the number of rounds played and the student performance in related tasks. [ABSTRACT FROM AUTHOR]

A comparison between two widespread global navigation satellite system (GNSS) acquisition strategies is presented. The first strategy bases its decision on comparing the energy within a cell to a threshold, while the second one uses the ratio between the two largest cell energies. It is shown that the first method outperforms the second one in terms of receiver operating characteristics (ROCs) in many practically relevant cases. Moreover, despite the purported simplicity of the ratio detection method, it is further shown that its complexity is comparable to or even higher than the one of threshold comparison with adaptive threshold setting. [ABSTRACT FROM PUBLISHER]

An optimal control of the combustion process of an engine ensures lower emissions and fuel consumption plus high efficiencies. Combustion parameters such as the peak firing pressure (PFP) and the crank angle (CA) corresponding to 50% of mass fraction burned (MFB50) are essential for a closed-loop control strategy. These parameters are based on the measured in-cylinder pressure that is typically gained by intrusive pressure sensors (PSs). These are costly and their durability is uncertain. To overcome these issues, the potential of using a virtual sensor based on the vibration signals acquired by a knock sensor (KS) for control of the combustion process is investigated. The present work introduces a data-driven approach where a signal-processing technique, designated as discrete wavelet transform (DWT), will be used as the preprocessing step for extracting informative features to perform regression tasks of the selected combustion parameters with extreme gradient boosting (XGBoost) regression models. The presented methodology will be applied to data from two different spark-ignited, single cylinder gas engines. Finally, an analysis is obtained where the important features based on the model's decisions are identified. [ABSTRACT FROM AUTHOR]

In this paper, we consider a lossy single-user caching problem with correlated sources. We first describe the fundamental interplay between the source correlations, the capacity of the user’s cache, the user’s reconstruction distortion requirements, and the final delivery-phase (compression) rate. We then illustrate this interplay using a multivariate Gaussian source example and a binary symmetric source example. To fully explore the effect of the user’s distortion requirements, we formulate the caching problem using \mathsf f -separable distortion functions recently introduce by Shkel and Verdú. The class of \mathsf f -separable distortion functions includes separable distortion functions as a special case, and our analysis covers both the expected- and excess-distortion settings in detail. We also determine what “common information” should be placed in the cache, and what information should be transmitted during the delivery phase. To this end, two new common-information measures are introduced for caching, and their relationship to the common-information measures of Wyner, Gács, and Körner is discussed in detail. [ABSTRACT FROM PUBLISHER]

We introduce a novel approach to reconstructing the in-cylinder pressure trace from vibration signals recorded with common knock sensors. The proposed methodology is purely data-driven and employs a convolutional neural network that has two distinct branches. Each branch is allowed to learn individual aspects of the mapping process, with boundary conditions within the model architecture set to incentivize the individual branches to learn low-frequency and high-frequency contents of the pressure trace. The reconstruction achieves calculated Pearson coefficients and coefficients of determination above 0.99 for all investigated datasets and a Mean Absolute Error of under 2.7 bar across all processed cycles. Furthermore, peak firing pressure and peak pressure position were extracted from the reconstructed cycles. Hereby, the method achieves Mean Absolute Error values of under 4.3 bar for peak firing pressure and under 1°crank angle for peak pressure position across all processed datasets, despite them not explicitly being targets of the underlying task. Deeper investigation of the results shows that combustion anomalies such as knocking do not negatively influence model fit. Moreover, model limitations were identified for high-pressure cycles and cycles exemplifying rather slow combustion. • Highly accurate in-cylinder pressure reconstruction from engine block vibrations. • Coefficient of determination and Pearson coefficient are above 0.99. • Errors for peak pressure and its position are below 0.5% and 0.2°, respectively. • Investigations show most difficulties are faced dealing with slow combustion. • Reconstruction performance is stable when dealing with knocking combustion. • The model is informed by engineers' knowledge and outperforms physics-based methods. [ABSTRACT FROM AUTHOR]

The phenomenon of knock is an abnormal combustion occurring in spark-ignition (SI) engines and forms a barrier that prevents an increase in thermal efficiency while simultaneously reducing CO 2 emissions. Since knocking combustion is highly stochastic, a cyclic analysis of in-cylinder pressure is necessary. In this study we propose an approach for efficient and robust detection and identification of knocking combustion in three different internal combustion engines. The proposed methodology includes a signal processing technique, called continuous wavelet transformation (CWT), which provides a simultaneous analysis of the in-cylinder pressure traces in the time and frequency domains with coefficients. These coefficients serve as input for a convolutional neural network (CNN) which extracts distinctive features and performs an image recognition task in order to distinguish between non-knock and knock. The results revealed the following: (i) The CWT delivered a stable and effective feature space with the coefficients that represents the unique time-frequency pattern of each individual in-cylinder pressure cycle; (ii) the proposed approach was superior to the state-of-the-art threshold value exceeded (TVE) method with a maximum amplitude pressure oscillation (MAPO) criterion improving the overall accuracy by 6.15 percentage points (up to 92.62%); and (iii) The CWT + CNN method does not require calibrating threshold values for different engines or operating conditions as long as enough and diverse data is used to train the neural network. [ABSTRACT FROM AUTHOR]