Search

Entropy is re-examined as a quantification of ignorance in the predictability of a one dimensional continuous phenomenon. Although traditional estimators for entropy have been widely utilized in this context, we show that both the thermodynamic and Shannon's theory of entropy are fundamentally discrete, and that the limiting process used to define differential entropy suffers from similar problems to those encountered in thermodynamics. In contrast, we consider a sampled data set to be observations of microstates (unmeasurable in thermodynamics and nonexistent in Shannon's discrete theory), meaning, in this context, it is the macrostates of the underlying phenomenon that are unknown. To obtain a particular coarse-grained model we define macrostates using quantiles of the sample and define an ignorance density distribution based on the distances between quantiles. The geometric partition entropy is then just the Shannon entropy of this finite distribution. Our measure is more consistent and informative than histogram-binning, especially when applied to complex distributions and those with extreme outliers or under limited sampling. Its computational efficiency and avoidance of negative values can also make it preferable to geometric estimators such as k-nearest neighbors. We suggest applications that are unique to this estimator and illustrate its general utility through an application to time series in the approximation of an ergodic symbolic dynamics from limited observations. [ABSTRACT FROM AUTHOR]

Considering both biological and non-biological polygonal shape organizations, in this paper we introduce a quantitative method which is able to determine informational entropy as spatial differences between heterogeneity of internal areas from simulation and experimental samples. According to these data (i.e., heterogeneity), we are able to establish levels of informational entropy using statistical insights of spatial orders using discrete and continuous values. Given a particular state of entropy, we establish levels of information as a novel approach which can unveil general principles of biological organization. Thirty-five geometric aggregates are tested (biological, non-biological, and polygonal simulations) in order to obtain the theoretical and experimental results of their spatial heterogeneity. Geometrical aggregates (meshes) include a spectrum of organizations ranging from cell meshes to ecological patterns. Experimental results for discrete entropy using a bin width of 0.5 show that a particular range of informational entropy (0.08 to 0.27 bits) is intrinsically associated with low rates of heterogeneity, which indicates a high degree of uncertainty in finding non-homogeneous configurations. In contrast, differential entropy (continuous) results reflect negative entropy within a particular range (−0.4 to −0.9) for all bin widths. We conclude that the differential entropy of geometrical organizations is an important source of neglected information in biological systems. [ABSTRACT FROM AUTHOR]

Kendall transformation is a conversion of an ordered feature into a vector of pairwise order relations between individual values. This way, it preserves ranking of observations and represents it in a categorical form. Such transformation allows for generalisation of methods requiring strictly categorical input, especially in the limit of small number of observations, when quantisation becomes problematic. In particular, many approaches of information theory can be directly applied to Kendall-transformed continuous data without relying on differential entropy or any additional parameters. Moreover, by filtering information to this contained in ranking, Kendall transformation leads to a better robustness at a reasonable cost of dropping sophisticated interactions which are anyhow unlikely to be correctly estimated. In bivariate analysis, Kendall transformation can be related to popular non-parametric methods, showing the soundness of the approach. The paper also demonstrates its efficiency in multivariate problems, as well as provides an example analysis of a real-world data. [ABSTRACT FROM AUTHOR]

In several applications, the assumption of normality is often violated in data with some level of skewness, so skewness affects the mean's estimation. The class of skew–normal distributions is considered, given their flexibility for modeling data with asymmetry parameter. In this paper, we considered two location parameter (μ) estimation methods in the skew–normal setting, where the coefficient of variation and the skewness parameter are known. Specifically, the least square estimator (LSE) and the best unbiased estimator (BUE) for μ are considered. The properties for BUE (which dominates LSE) using classic theorems of information theory are explored, which provides a way to measure the uncertainty of location parameter estimations. Specifically, inequalities based on convexity property enable obtaining lower and upper bounds for differential entropy and Fisher information. Some simulations illustrate the behavior of differential entropy and Fisher information bounds. [ABSTRACT FROM AUTHOR]

We propose a theory to characterize the information and information processing abilities of metasurfaces, and demonstrate the relation between the information of the metasurface and its radiation pattern in the far-field region. By incorporating a general aperture model with uncertainty relation in L 2-space, we propose a theory to predict the upper bound of information contained in the radiation pattern of a metasurface, and reveal the theoretical upper limit of orthogonal radiation states. The proposed theory also provides guidance for inverse design of the metasurface with respect to given functionalities. Through investigation of the information of disordered-phase modulated metasurfaces, we find the information invariance (1−γ, where γ is Euler's constant) of chaotic radiation patterns. That is to say, the information of the disordered-phase modulated radiation patterns is always equal to 1− γ , regardless of variations in size, the number of elements and the phase pattern of metasurface. This value might be the lower bound of radiation-pattern information of the metasurface, which can provide a theoretical limit for information modulation applications, including computational imaging, stealth technologies and wireless communications. [ABSTRACT FROM AUTHOR]

Gradient of the information measures can be considered as the building block of the fundamental relations between information and estimation theories established in the literature. This paper introduces the gradients of two fundamental information measures, i.e., differential entropy and mutual information in some general cases. Our approach is based on the utilizing a general functional model to describe the stochastic systems. This approach enables us to give a simple new proof for De Bruijn's identity. We also derive a general expression for the gradient of the mutual information in systems with both additive and multiplicative noises. Systems with additive noise, as the most common model for the communication channels, is considered as well, where in the Gaussian case some of the available results in the literature are recovered. Moreover, the aforementioned general functional model allows us to study the sensitivity of the input–output mutual information to the system parameters in some other practical scenarios. Specifically, Orthogonal Frequency Division Multiplexing (OFDM) systems with High Power Amplifier (HPA) are considered and an information theoretic analysis for the behavior of these systems is given. [ABSTRACT FROM AUTHOR]

At present, research on hesitant fuzzy operations and measures is based on equal length processing, and an equal length processing method will inevitably destroy the original data structure and change the data information. This is an urgent problem to be solved in the development of hesitant fuzzy sets. Aiming at solving this problem, this paper firstly defines a hesitant fuzzy entropy function as the measure of the degree of uncertainty of hesitant fuzzy information and then proposes the concept of hesitant fuzzy information feature vector. The hesitant fuzzy distance measure and similarity measure are studied based on the information feature vector. Finally, the hesitant fuzzy network clustering method based on similarity measure is given, and the effectiveness of our algorithm through a numerical example is illustrated. [ABSTRACT FROM AUTHOR]

The entropy of a random variable is well-known to equal the exponential growth rate of the volumes of its typical sets. In this paper, we show that for any log-concave random variable X , the sequence of the \lfloor n\theta \rfloor ^{\text {th}} intrinsic volumes of the typical sets of X in dimensions n \geq 1 grows exponentially with a well-defined rate. We denote this rate by h_{X}(\theta ) , and call it the \theta ^{\text {th}} intrinsic entropy of X . We show that h_{X}(\theta ) is a continuous function of \theta $ over the range [0, 1], thereby providing a smooth interpolation between the values 0 and $h(X)$ at the endpoints 0 and 1, respectively. [ABSTRACT FROM PUBLISHER]

This paper investigates the design of information theoretic-based fitness function for embedded evolutionary robotics (ERs). Such fitness relies on the assumption that interesting behaviors result in a high sensorimotor (individual) diversity. The current simple entropy as a diversity metric only considers individuals' difference but ignores their spatial relationship. The sensorimotor stream can be analyzed to construct a simple directed graph that has unique entry and exit nodes. This paper proposes a hierarchic entropy as a diversity metric by incorporating the simple entropy and the spatial relationship based graph entropy. Maximizing the hierarchic entropy, achieved by on-board evolutionary algorithm, thus defines a self-driven fitness function enforcing the controller visiting diverse sensorimotor states. The proposed algorithm achieves better performance than the published results of other entropy-based methods only relying on simple entropy, without requiring additional computational resources. [ABSTRACT FROM AUTHOR]

We consider the independent sum of a given random variable with a Gaussian variable and an infinitely divisible one. We find a novel tight upper bound on the entropy of the sum which still holds when the variable possibly has an infinite second moment. The proven bound has several implications on both information theoretic problems and infinitely divisible noise channels' transmission rates. [ABSTRACT FROM AUTHOR]

Accurate short-term traffic forecasting is the cornerstone for Intelligent Transportation Systems. In the past several decades, many models have been proposed to continuously improve the predictive accuracy. A key but unsolved question is whether there is a theoretical bound to the accuracy with which traffic can be predicted and whether that limit can be directly estimated from data. To answer this question, we use core concepts in information theory to derive the limit of predictability in short-term traffic forecasting. Theoretical analysis proves that conditional differential entropy poses a rigorous lower bound of negative-log-likelihood (NLL) for probabilistic models. And the continuous form of Fano's theorem further gives a loose lower bound of mean-square-error (MSE) for deterministic models. Based on the special properties of traffic dynamics, two assumptions are made in the estimate of entropy metrics: cyclostationarity (traffic phenomena show strong periodicity) and localized spatial correlation (due to kinematic wave propagation). They allow formulating the limit of predictability as a function of longitudinal space and time-of-day which finds the most uncertain locations and periods solely from data. Experiments on univariate traffic accumulation forecasting and network-level speed forecasting show that the selected models, including some state-of-the-art deep learning models, indeed cannot outperform the estimated lower bounds but just approach them. The limit of predictability depends on time-of-day, network locations, observation range, and prediction horizon. The results reveal that the stochastic nature of traffic dynamics and improper assumptions on the prior distribution of output are two major factors that restrict the predictive performance. In summary, the proposed method estimates a trustworthy performance boundary for most traffic forecasting models. These conclusions are helpful for further studies in this domain. • Estimate the limit of predictability for data-driven traffic forecasting. • Embed spatiotemporal correlations in the lower bound estimation scheme. • Identify the most unpredictable time slots and locations directly from data. • Clarify the major bottlenecks for further improving predictive accuracy. [ABSTRACT FROM AUTHOR]

Blanched-Sadri and Woodhouse in 2013 have proven the conjecture of Cassaigne, stating that any pattern with m distinct variables and of length at least 2m is avoidable over a ternary alphabet and if the length is at least 3 2m−1 it is avoidable over a binary alphabet. They conjectured that similar theorems are true for partial words - sequences, in which some characters are left 'blank'. Using method of entropy compression, we obtain the partial words version of the theorem for ternary words [ABSTRACT FROM AUTHOR]

Overlapping genes are two protein-coding sequences sharing a significant part of the same DNA locus in different reading frames. Although in recent times an increasing number of examples have been found in bacteria the underlying mechanisms of their evolution are unknown. In this work we explore how selective pressure in a protein-coding sequence influences its overlapping genes in alternative reading frames. We model evolution using a time-continuous Markov process and derive the corresponding model for the remaining frames to quantify selection pressure and genetic noise. Our findings lead to the presumption that, once information is embedded in the reverse reading frame −2 (relative to the mother gene in +1) purifying selection in the protein-coding reading frame automatically protects the sequences in both frames. We also found that this coincides with the fact that the genetic noise measured using the conditional entropy is minimal in frame −2 under selection in the coding frame. [ABSTRACT FROM AUTHOR]

Highlights: [•] Shannon and Fisher entropies are calculated for confined hydrogen. [•] They are analyzed in terms of the box radius and barrier strength. [•] They evolve smoothly along the box radius for hard confinement. [•] They change rapidly over short radial intervals for soft confinement. [Copyright &y& Elsevier]

Many metrics of design complexity have been proposed in the literature. Of these, the most popular are information theoretic metrics, such as information content based on Suh's axiomatic theory and differential entropy based on Shannon's information theory. In this paper, we show that these metrics do not provide common sense measures of complexity, and they also do not possess proper mathematical properties. At best, they are geared towards measuring a designs goodness-of-fit rather than its complexity. At least in part, the inappropriate use of these metrics can be attributed to misuse of terms such as ‘entropy’ and ‘minimise information content’. [ABSTRACT FROM AUTHOR]

Abstract: Organisms on the move face uncertainty regarding the state of their environments, and characterizing the magnitude of this uncertainty is important because of its influence on organismal decision making. Two common measures of the uncertainty inherent in a distribution of possible outcomes are variance and entropy, yet there is currently no standard for when one measure should be used over the other. This paper explores this question using two models of resource uncertainty. For small numbers of discrete possible outcomes, variance is the better measure because it captures the spread between outcomes as well as their differential possibilities. However, variance can categorically fail as a measure of uncertainty when distributions are multimodal or discontinuous, in which case entropy should be used to characterize uncertainty. [Copyright &y& Elsevier]

This study is intended to investigate the connection between the complexity of a capital market and the occurrence of dramatic decreases in transaction prices. The work hypothesis is that such episodes, characterized by sudden and dramatic decreases in transaction prices mostly occur in period of market inefficiency, when the level of complexity reaches a local minimum. In this regard, we introduce a complexity estimator, through differential entropy. The connection between the market complexity level and the appearance of extreme returns is illustrated in a logistic regression model. [ABSTRACT FROM AUTHOR]

This paper focuses on illustrating 1) the equivalence between Stein's identity and De Bruijn's identity, and 2) two extensions of De Bruijn's identity. First, it is shown that Stein's identity is equivalent to De Bruijn's identity under additive noise channels with specific conditions. Second, for arbitrary but fixed input and noise distributions under additive noise channels, the first derivative of the differential entropy is expressed by a function of the posterior mean, and the second derivative of the differential entropy is expressed in terms of a function of Fisher information. Several applications over a number of fields, such as signal processing and information theory, are presented to support the usefulness of the developed results in this paper. [ABSTRACT FROM AUTHOR]

Recent research has demonstrated significant achievable performance gains by exploiting circularity/noncircularity or properness/improperness of complex-valued signals. In this paper, we investigate the influence of these properties on important information theoretic quantities such as entropy, divergence, and capacity. We prove two maximum entropy theorems that strengthen previously known results. The proof of the first maximum entropy theorem is based on the so-called circular analog of a given complex-valued random vector. The introduction of the circular analog is additionally supported by a characterization theorem that employs a minimum Kullback–Leibler divergence criterion. In the proof of the second maximum entropy theorem, results about the second-order structure of complex-valued random vectors are exploited. Furthermore, we address the capacity of multiple-input multiple-output (MIMO) channels. Regardless of the specific distribution of the channel parameters (noise vector and channel matrix, if modeled as random), we show that the capacity-achieving input vector is circular for a broad range of MIMO channels (including coherent and noncoherent scenarios). Finally, we investigate the situation of an improper and Gaussian distributed noise vector. We compute both capacity and capacity-achieving input vector and show that improperness increases capacity, provided that the complementary covariance matrix is exploited. Otherwise, a capacity loss occurs, for which we derive an explicit expression. [ABSTRACT FROM PUBLISHER]

Expressions for the entropy, the Kullback-Leibler information, and the Iα-information are established for distributions of progressively Type-II censored order statistics. These results are used to identify minimum and maximum information censoring plans. In particular, we find minimum and maximum entropy plans for DFR, exponential, Pareto, reflected power, and Weibull distributions. The results for Kullback-Leibler and Iα-information hold for any continuous distribution. [ABSTRACT FROM AUTHOR]

While most useful information theoretic inequalities can be deduced from the basic properties of entropy or mutual information, up to now Shannon's entropy power inequality (EPI) is an exception: Existing information theoretic proofs of the EPI hinge on representations of differential entropy using either Fisher information or minimum mean-square error (MMSE), which are derived from de Bruijn's identity. In this paper, we first present an unified view of these proofs, showing that they share two essential ingredients: 1) a data processing argument applied to a covariance-preserving linear transformation; 2) an integration over a path of a continuous Gaussian perturbation. Using these ingredients, we develop a new and brief proof of the EPI through a mutual information inequality, which replaces Stam and Blachman's Fisher information inequality (FII) and an inequality for MMSE by Guo, Shamai, and Verdú used in earlier proofs. The result has the advantage of being very simple in that it relies only on the basic properties of mutual information. These ideas are then generalized to various extended versions of the EPI: Zamir and Feder's generalized EPI for linear transformations of the random variables, Takano and Johnson's EPI for dependent variables, Liu and Viswanath's covariance-constrained EPI, and Costa's concavity inequality for the entropy power. [ABSTRACT FROM AUTHOR]

In this paper, an analytical expression is developed for the differential entropy of a sinusoid with a Beta-distributed phase angle. This particular signal model is prevalent in optical communications, however an expression for the associated differential entropy does not currently exist. The expression we derive is approximate as it relies on a series expansion for one of the key terms needed in the derivation. However, we are able to show that the approximation is accurate (error ≤ 5%) for a wide variety of Beta parameter choices. [ABSTRACT FROM AUTHOR]

Estimation of the differential entropy from observations of a random variable is of great importance for a wide range of signal processing applications such as source coding, pattern recognition, hypothesis testing, and blind source separation. In this paper, we present a method for estimation of the Shannon differential entropy that accounts for embedded manifolds. The method is based on high-rate quantization theory and forms an extension of the classical nearest-neighbor entropy estimator. The estimator is consistent in the mean square sense and an upper bound on the rate of convergence of the estimator is given. Because of the close connection between compression and Shannon entropy, the proposed method has an advantage over methods estimating the Renyi entropy. Through experiments on uniformly distributed data on known manifolds and real-world speech data we show the accuracy and usefulness of our proposed method. [ABSTRACT FROM AUTHOR]

Recently, Rao et al. [(2004) Cumulative residual entropy: a new measure of information. IEEE Trans. Inform. Theory 50(6), 1220–1228] have proposed a new measure of uncertainty, called cumulative residual entropy (CRE), in a distribution function F and obtained some properties and applications of that. In the present paper, we propose a dynamic form of CRE and obtain some of its properties. We show how CRE (and its dynamic version) is connected with well-known reliability measures such as the mean residual life time. [Copyright &y& Elsevier]

We prove a new external inequality, motivated by the vector Gaussian broadcast channel and the distributed source coding with a single quadratic distortion constraint problems. As a corollary, this inequality yields a generalization of the classical entropy-power inequality (EPI). As another corollary, this inequality sheds insight into maximizing the differential entropy of the sum of two dependent random variables. [ABSTRACT FROM AUTHOR]

In this paper, the conditions for identifiability, separability and uniqueness of linear complex valued independent component analysis (ICA) models are established. These results extend the well-known conditions for solving real-valued ICA problems to complex-valued models. Relevant properties of complex random vectors are described in order to extend the Darmois-Skitovich theorem for complex-valued models. This theorem is used to construct a proof of a theorem for each of the above ICA model concepts. Both circular and noncircular complex random vectors are covered. Examples clarifying the above concepts are presented. [ABSTRACT FROM AUTHOR]

In a causal source coding system, the reconstruction of the present source sample is restricted to be a function of the present and past source samples, while the code stream itself may be noncausal and have variable rate. Neuhoff and Gilbert showed that for memoryless sources, optimum performance among all causal source codes is achieved by time-sharing at most two memoryless codes (quantizers) followed by entropy coding. In this work, we extend Neuhoff and Gilbert's result in the limit of small distortion (high resolution) to two new settings. First, we show that at high resolution, an optimal causal code for a stationary source with finite differential entropy rate consists of a uniform quantizer followed by a (sequence) entropy coder. This implies that the price of causality at high resolution is approximately 0.254 bit, i.e., the space-filling loss of the uniform quantizer. Then, we consider individual sequences and introduce a deterministic analogue of differential entropy, which we call "Lempel-Ziv differential entropy." We show that for any bounded individual sequence with finite Lempel-Ziv differential entropy, optimum high-resolution performance among all finite-memory variable-rate causal codes is achieved by dithered scalar uniform quantization followed by Lempel-Ziv coding. As a by-product, we also prove an individual-sequence version of the Shannon lower bound. [ABSTRACT FROM AUTHOR]

A broad set of sufficient conditions that guarantees the existence of the maximum entropy (maxent) distribution consistent with specified bounds on certain generalized moments is derived. Most results in the literature are either focused on the minimum cross-entropy distribution or apply only to distributions with a bounded-volume support or address only equality constraints. The results of this work hold for general moment inequality constraints for probability distributions with possibly unbounded support, and the technical conditions are explicitly on the underlying generalized moment functions. An analytical characterization of the maxent distribution is also derived using results from the theory of constrained optimization in infinite-dimensional norms linear spaces. Several auxiliary results of independent Interest pertaining to certain properties of convex coercive functions are also presented. [ABSTRACT FROM AUTHOR]

Abstract: The enthalpy–entropy compensation theory was applied to the experimental moisture adsorption and desorption isotherm data (water activity (a w) range 0.006–0.982) of raw potato, potato starch gel, potato starch powder, highly amylopectin corn starch powder and highly amylose corn starch powder in the temperature range 30–60 °C. A linear relation existed between differential enthalpy (ΔH) and differential entropy (ΔS) for all the experimental data considered, thus satisfying the enthalpy–entropy compensation theory. Further analysis of the data indicated an enthalpy-controlled (isokinetic temperature (T β )>harmonic mean temperature (T hm)) and spontaneous (−ΔG) sorption process. [Copyright &y& Elsevier]

We explore properties of the entropy, Kullback-Leibler information, and mutual information for order statistics. The probability integral transformation plays a pivotal role in developing our results. We provide bounds for the entropy of order statistics and some results that relate entropy ordering of order statistics to other well-known orderings of random variables. We show that the discrimination information between order statistics and data distribution, the discrimination information among the order statistics, and the mutual information between order statistics are all distribution free and are computable using the distributions of the order statistics of the samples from the uniform distribution. We also discuss information properties of spacings for uniform and exponential samples and provide a large sample distribution-free result on the entropy of spacings. The results show interesting symmetries of information orderings among order statistics. [ABSTRACT FROM AUTHOR]

We explore a well-known integral representation of the logarithmic function, and demonstrate its usefulness in obtaining compact, easily computable exact formulas for quantities that involve expectations and higher moments of the logarithm of a positive random variable (or the logarithm of a sum of i.i.d. positive random variables). The integral representation of the logarithm is proved useful in a variety of information-theoretic applications, including universal lossless data compression, entropy and differential entropy evaluations, and the calculation of the ergodic capacity of the single-input, multiple-output (SIMO) Gaussian channel with random parameters (known to both transmitter and receiver). This integral representation and its variants are anticipated to serve as a useful tool in additional applications, as a rigorous alternative to the popular (but non-rigorous) replica method (at least in some situations). [ABSTRACT FROM AUTHOR]

We give a brief survey of the literature on the empirical estimation of entropy, differential entropy, relative entropy, mutual information and related information measures. While those quantities are of central importance in information theory, universal algorithms for their estimation are increasingly important in data science, machine learning, biology, neuroscience, economics, language, and other experimental sciences. [ABSTRACT FROM AUTHOR]

Recent work has focused on the problem of nonparametric estimation of information divergence functionals between two continuous random variables. Many existing approaches require either restrictive assumptions about the density support set or difficult calculations at the support set boundary which must be known a priori. The mean squared error (MSE) convergence rate of a leave-one-out kernel density plug-in divergence functional estimator for general bounded density support sets is derived where knowledge of the support boundary, and therefore, the boundary correction is not required. The theory of optimally weighted ensemble estimation is generalized to derive a divergence estimator that achieves the parametric rate when the densities are sufficiently smooth. Guidelines for the tuning parameter selection and the asymptotic distribution of this estimator are provided. Based on the theory, an empirical estimator of Rényi- α divergence is proposed that greatly outperforms the standard kernel density plug-in estimator in terms of mean squared error, especially in high dimensions. The estimator is shown to be robust to the choice of tuning parameters. We show extensive simulation results that verify the theoretical results of our paper. Finally, we apply the proposed estimator to estimate the bounds on the Bayes error rate of a cell classification problem. [ABSTRACT FROM AUTHOR]