Search

In this work, we investigate the use of three information-theoretic quantities-entropy, mutual information with the class variable, and a class selectivity measure based on Kullback-Leibler (KL) divergence-to understand and study the behavior of already trained fully connected feedforward neural networks (NNs). We analyze the connection between these information-theoretic quantities and classification performance on the test set by cumulatively ablating neurons in networks trained on MNIST, FashionMNIST, and CIFAR-10. Our results parallel those recently published by Morcos et al., indicating that class selectivity is not a good indicator for classification performance. However, looking at individual layers separately, both mutual information and class selectivity are positively correlated with classification performance, at least for networks with ReLU activation functions. We provide explanations for this phenomenon and conclude that it is ill-advised to compare the proposed information-theoretic quantities across layers. Furthermore, we show that cumulative ablation of neurons with ascending or descending information-theoretic quantities can be used to formulate hypotheses regarding the joint behavior of multiple neurons, such as redundancy and synergy, with comparably low computational cost. We also draw connections to the information bottleneck theory for NNs.

In this paper, we consider a lossy single-user caching problem with correlated sources. We first describe the fundamental interplay between the source correlations, the capacity of the user’s cache, the user’s reconstruction distortion requirements, and the final delivery-phase (compression) rate. We then illustrate this interplay using a multivariate Gaussian source example and a binary symmetric source example. To fully explore the effect of the user’s distortion requirements, we formulate the caching problem using \mathsf f -separable distortion functions recently introduce by Shkel and Verdú. The class of \mathsf f -separable distortion functions includes separable distortion functions as a special case, and our analysis covers both the expected- and excess-distortion settings in detail. We also determine what “common information” should be placed in the cache, and what information should be transmitted during the delivery phase. To this end, two new common-information measures are introduced for caching, and their relationship to the common-information measures of Wyner, Gács, and Körner is discussed in detail. [ABSTRACT FROM PUBLISHER]

The information loss in deterministic, memoryless systems is investigated by evaluating the conditional entropy of the input random variable given the output random variable. It is shown that for a large class of systems the information loss is finite, even if the input has a continuous distribution. For systems with infinite information loss, a relative measure is defined and shown to be related to Rényi information dimension. As deterministic signal processing can only destroy information, it is important to know how this information loss affects the solution of inverse problems. Hence, we connect the probability of perfectly reconstructing the input to the information lost in the system via Fano-type bounds. The theoretical results are illustrated by example systems commonly used in discrete-time, nonlinear signal processing and communications. [ABSTRACT FROM AUTHOR]

Finite precision approximations of discrete probability distributions are considered, applicable for distribution synthesis, e.g., probabilistic shaping. Two algorithms are presented that find the optimal $M$ -type approximation Q$ of a distribution P$ in terms of the variational distance and the informational divergence $D(Q\Vert P)$ . Bounds on the approximation errors are derived and shown to be asymptotically tight. Several examples illustrate that the variational distance optimal approximation can be quite different from the informational divergence optimal approximation. [ABSTRACT FROM PUBLISHER]

A lumping of a Markov chain is a coordinate wise projection of the chain. We characterise the entropy rate preservation of a lumping of an aperiodic and irreducible Markov chain on a finite state space by the random growth rate of the cardinality of the realisable preimage of a finite-length trajectory of the lumped chain and by the information needed to reconstruct original trajectories from their lumped images. Both are purely combinatorial criteria, depending only on the transition graph of the Markov chain and the lumping function. A lumping is strongly ¿-lumpable, if and only if the lumped process is a Ath-order Markov chain for each starting distribution of the original Markov chain. We characterise strong fc-lumpability via tightness of stationary entropie bounds. In the sparse setting, we give sufficient conditions on the lumping to both preserve the entropy rate and be strongly ¿-lumpable. [ABSTRACT FROM AUTHOR]

Consider the problem of approximating a Markov chain by another Markov chain with a smaller state space that is obtained by partitioning the original state space. An information-theoretic cost function is proposed that is based on the relative entropy rate between the original Markov chain and a Markov chain defined by the partition. The state space aggregation problem can be sub-optimally solved by using the information bottleneck method. [ABSTRACT FROM AUTHOR]

Acknowledging the issues in [[12]], these NNs are trained to minimize an upper bound on HT ht , combining variational and non-parametric approaches for bounding. In this case, it is known that sweeping the paramter HT ht in (1) is not sufficient to sweep the Pareto frontier of optimal HT ht pairs [[12]]. The authors show that this shortcoming can be removed by optimizing HT ht instead, where I u i is a strictly convex function. [Extracted from the article]