Search

We leverage the interplay between microscopic variability and macroscopic order to connect physical descriptions across scales directly from data, without underlying equations. We reconstruct a state space by concatenating measurements in time, building a maximum entropy partition of the resulting sequences, and choosing the sequence length to maximize predictive information. Trading non-linear trajectories for linear, ensemble evolution, we analyze reconstructed dynamics through transfer operators. The evolution is parameterized by a transition timeτ: capturing the source entropy rate at smallτand revealing timescale separation with collective, coherent states through the operator spectrum at largerτ. Applicable to both deterministic and stochastic systems, we illustrate our approach through the Langevin dynamics of a particle in a double-well potential and the Lorenz system. Applied to the behavior of the nematode wormC. elegans, we derive a “run-and-pirouette” navigation strategy directly from posture dynamics. We demonstrate how sequences simulated from the ensemble evolution capture both fine scale posture dynamics and large scale effective diffusion in the worm’s centroid trajectories and introduce a top-down, operator-based clustering which reveals subtle subdivisions of the “run” behavior.POPULAR SUMMARYComplex structure is often composed from a limited set of relatively simple building blocks; such as novels from letters or proteins from amino acids. In musical composition, e.g., sounds and silences combine to form longer time scale structures; motifs form passages which in turn form movements. The challenge we address is how to identify collective variables which distinguish structures across such disparate time scales. We introduce a principled framework for learning effective descriptions directly from observations. Just as a musical piece transitions from one movement to the next, the collective dynamics we infer consists of transitions between macroscopic states, like jumps between metastable states in an effective potential landscape.The statistics of these transitions are captured compactly by transfer operators. These operators play a central role, guiding the construction of maximally-predictive short-time states from incomplete measurements and identifying collective modes via eigenvalue decomposition. We demonstrate our analysis in both stochastic and deterministic systems, and with an application to the movement dynamics of an entire organism, unravelling new insight in long time scale behavioral states directly from measurements of posture dynamics. We can, in principle, also make connections to both longer or shorter timescales. Microscopically, postural dynamics result from the fine scale interactions of actin and myosin in the muscles, and from electrical impulses in the brain and nervous system. Macroscopically, behavioral dynamics may be extended to longer time scales, to moods or dispositions, including changes during aging, or over generations due to ecological or evolutionary adaptation. The generality of our approach provides opportunity for insights on long term dynamics within a wide variety of complex systems.

The IoT market has grown significantly in recent years, and it is estimated that it will continue to do so. For this reason, the need to identify new solutions to ensure security is vital for the future development in this field. Inadequate sources of entropy are one of the factors that negatively influence security. In this study, inspired by NIST’s latest entropy estimation recommendations, we proposed a methodology for analyzing and validating a sensor-based entropy source, highlighted by an innovative experiment design. Moreover, the proposed solution is analyzed in terms of resistance to multiple types of attacks. Following an analysis of the influence of sensor characteristics and settings on the entropy rate, we obtain a maximum entropy value of 0.63 per bit, and a throughput of 3.12 Kb/s, even when no motion is applied on the sensors. Our results show that a stable and resistant entropy source can be built based on the data obtained from the sensors. Our assessment of the proposed entropy source also achieves a higher complexity than previous studies, in terms of the variety of approached situations and the types of the performed experiments.

To get a deep understanding of a diffusion process and realizing the most efficient methods for investigating a real network, has always been of great interest and utility to us. In this work, we aim to study and compare our mobility in a network using a normal random walk and a long-range navigation strategy such as Levy Walk. We study the Global Mean First Traverse Distance (GMFTD) and the entropy rate for this long-range navigation process and later, compare with a normal walk strategy. For this study, GMFTD is utilized instead of Global Mean First Passage Time (GMFPT). The reason for such a choice is the fact that for more accurate and precise results, we need to also take into account the cost of a large step in case of a long-range navigation strategy. Next, we continue by calculating the entropy rate in both procedures and we derive the corresponding expressions of these quantities on a Treelike fractal. Eventually, we show that while GMFTD decreases in a long-range navigation system, entropy rate will increase. We perform an analytical comparison between the two cases and clearly demonstrate the superiority of a Levy Walk strategy. Eventually, our simulation results, based on the derived expressions, give us a very reliable estimation for the optimum value of exponent parameter, α, which are in strong agreement with each other for both cases of a minimum GMFTD and a maximum entropy rate.

This paper provides a comprehensive analysis of the following optimization problem: Maximize the entropy rate generated by a Markov chain over a connected graph of order $n$ and subject to a prescribed stationary distribution. First, we show that this problem is strictly convex with global optimum lying in the interior of the feasible space. Second, using Lagrange multipliers, we provide a closed-form expression for the maxentropic Markov chain as a function of an $n$ -dimensional vector, referred to as the maxentropic vector; we provide a provably converging iteration to compute this vector. Third, we show that the maxentropic Markov chain is reversible, compute its entropy rate and describe special cases, among other results. Fourth, through analysis and simulations, we show that our proposed procedure is more computationally efficient than semidefinite programming methods. Finally, we apply these results to robotic surveillance problems. We show realizations of the maxentropic Markov chains over prototypical robotic roadmaps and find that maxentropic Markov chains outperform minimum mean hitting time Markov chains for the so-called “intelligent intruders” with short attack durations. A comprehensive analysis of the following optimization problem: maximize the entropy rate generated by a Markov chain over a connected graph of order $n$ and subject to a prescribed stationary distribution.

We propose a mathematical model of the entropy estimator in the Linux random number generator. First, we construct a probability model for random event times in entropy sources, and then precisely derive probability distributions for the first, second, and third time differences. Second, we obtain the probability distribution for the minimum of absolute values of these differences, which is used for the estimated entropy in the Linux system. Moreover, we provide several simulations that display the accuracy of our results for various parameters.

Recently, Sati and Gupta (2015) have introduced a generalized cumulative residual entropy based on the non-additive Tsallis entropy. The cumulative residual entropy, introduced by Rao et al. (2004) is a generalized measure of uncertainty which is applied in reliability and image alignment and non-additive measures of entropy. This entropy finds justifications in many physical, biological and chemical phenomena. In this paper, we derive the weighted form of this measure and call it Weighted Cumulative Residual Tsallis Entropy (WCRTE). Being a “length-biased” shift-dependent information measure, WCRTE is related to the differential information in which higher weight is assigned to large values of observed random variables. Based on the dynamic version of this new information measure, we propose ageing classes and it is shown that it can uniquely determine the survival function and Rayleigh distribution. Several properties, including linear transformations, bounds and related results to stochastic orders are obtained for these measures. Also, we identify classes of distributions in which some well-known distributions are maximum dynamic version of WCRTE. The empirical WCRTE is finally proposed to estimate the new information measure.

In this article, we propose high order, semi-discrete, entropy stable, finite difference schemes for Ten-Moment Gaussian Closure equations. The crucial components of these schemes are an entropy conservative flux and suitable high order entropy dissipative operators to ensure entropy stability. We design two numerical fluxes, one is approximately entropy conservative, and another is entropy conservative flux. For the construction of appropriate entropy dissipative operators, we also derive entropy scaled right eigenvectors. This is used for sign preserving reconstruction of scaled entropy variables, which results in second and third order entropy stable schemes. We also extend these schemes to a plasma flow model with source term. Several numerical results are presented for homogeneous and non-homogeneous cases to demonstrate stability and performance of these schemes.

Partial entropy is a device to measure how much of entropy of an uncertain random variable belongs to uncertain variables. However, partial entropy may fail to measure the uncertainty of an uncertain random variable. In this paper, for refining this problem, a definition of partial triangular entropy is presented. In addition, several properties of partial triangular entropy are obtained. Moreover, partial triangular entropy of an uncertain random variable is applied to mean-variance portfolio selection.

Characterising the capacity region for a network can be extremely difficult. Even with independent sources, determining the capacity region can be as hard as the open problem of characterising all information inequalities. The majority of computable outer bounds in the literature are relaxations of the Linear Programming bound which involves entropy functions of random variables related to the sources and link messages. When sources are not independent, the problem is even more complicated. Extension of Linear Programming bounds to networks with correlated sources is largely open. Source dependence is usually specified via a joint probability distribution, and one of the main challenges in extending linear program bounds is the difficulty (or impossibility) of characterising arbitrary dependencies via entropy functions. This paper tackles the problem by answering the question of how well entropy functions can characterise correlation among sources. We show that by using carefully chosen auxiliary random variables, the characterisation can be fairly "accurate" Using such auxiliary random variables we also give implicit and explicit outer bounds on the capacity of networks with correlated sources. The characterisation of correlation or joint distribution via Shannon entropy functions is also applicable to other information measures such as Renyi entropy and Tsallis entropy., Comment: 24 pager, 1 figure, submitted to IEEE Transactions on Information Theory. arXiv admin note: text overlap with arXiv:1309.1517

We propose a statistical monitoring method to detect the increase of entropy in categorical data. First, we propose a distribution estimation method to approximate the probability distribution of t...

Uncertain variables, applied to modelling imprecise quantities, are fundamental concepts in uncertainty theory. In order to characterize the information deficiency of an uncertain variable, this paper proposes a definition of quadratic entropy. Compared with the traditional entropy, it is much easier to compute and has a wider range. Furthermore, the principle of maximum entropy is applied to quadratic entropy, and two theorems of maximum quadratic entropy with moment constraint are proved.

We present a series of closed-form upper bounds of the differential entropy of univariate continuous distributions based on the maximum entropy principle. We apply those bounds to Gaussian mixture models, and study their tightness properties.

Recently, Di Crescenzo and Longobardi (2006) have studied “length-biased” shift-dependent information measure and its dynamic versions. On the other hand, Renyi's entropy plays a vital role in the ...

In this paper, recently introduced permutation entropy and sample entropy are further developed to the fractional cases, weighted fractional permutation entropy (WFPE) and fractional sample entropy (FSE). The fractional order generalization of information entropy is utilized in the above two complexity approaches, to detect the statistical characteristics of fractional order information in complex systems. The effectiveness analysis of proposed methods on the synthetic data and the real-world data reveals that tuning the fractional order allows a high sensitivity and more accurate characterization to the signal evolution, which is useful in describing the dynamics of complex systems. Moreover, the numerical research on nonlinear complexity behaviors is compared between the returns series of Potts financial model and the actual stock markets. And the empirical results confirm the feasibility of the proposed model.

The Hamburger and Stieltjes moment problem and the sequence of maximum entropy (MaxEnt) approximates with given moments are considered. MaxEnt approximate converges in entropy to the unknown distribution when the Hamburger (or Stieltjes) moment problem is determinate and the underlying distribution has finite entropy.

In this paper, we consider a fuzzy c-means (FCM) clustering algorithm combined with the deterministic annealing method and the Tsallis entropy maximization. The Tsallis entropy is a q-parameter extension of the Shannon entropy. By maximizing the Tsallis entropy within the framework of FCM, membership functions similar to statistical mechanical distribution functions can be derived. One of the major considerations when using this method is how to determine appropriate q values and the highest annealing temperature, Thigh , for a given data set. Accordingly, in this paper, a method for determining these values simultaneously without introducing any additional parameters is presented. In our approach, the membership function is approximated by a series of expansion methods and the K-means clustering algorithm is utilized as a preprocessing step to estimate a radius of each data distribution. The results of experiments indicate that the proposed method is effective and both q and Thigh can be determined automatically and algebraically from a given data set.

Tsallis entropy is a generalized form of entropy and tends to be Shannon entropy when q → 1. Using Tsallis entropy, an alternative estimation methodology (generalized maximum Tsallis entropy) is introduced and used to estimate the parameters in a linear regression model when the basic data are ill-conditioned. We describe the generalized maximum Tsallis entropy and for q = 2 we call that GMET2 estimator. We apply the GMET2 estimator for estimating the linear regression model Y = Xβ + e where the design matrix X is subject to severe multicollinearity. We compared the GMET2, generalized maximum entropy (GME), ordinary least-square (OLS), and inequality restricted least-square (IRLS) estimators on the analyzed dataset on Portland cement.

Recently, quantile-based study of entropy measures found greater interest among researchers as an alternative method of measuring uncertainty of a random variable. The quantile-based entropy measures possess some unique properties than its distribution function approach. Motivated by this, in the present paper, we introduce a quantile-based entropy of order statistics and study its properties. We also propose a quantile-based residual entropy of order statistics, an alternative method to measure the uncertainty of ordered observations for used items. Unlike distribution function approach, we derive an explicit relationship between the quantile density function and quantile-based residual entropy of order statistics.

We take up the problem of determining the distribution of the hitting time of a parabola by a Brownian motion. We use the maximum entropy method to obtain a good approximation to the true density from its translated fractional moments.

We examine the entropy of low-significance bits in analog-to-digital measurements of chaotic dynamical systems. We find the partition of measurement space corresponding to low-significance bits has a corrugated structure. Using simulated measurements of a map and experimental data from a circuit, we identify two consequences of this corrugated partition. First, entropy rates for sequences of low-significance bits more closely approach the metric entropy of the chaotic system, because the corrugated partition better approximates a generating partition. Second, accurate estimation of the entropy rate using low-significance bits requires long block lengths as the corrugated partition introduces more long-term correlation, and using only short block lengths overestimates the entropy rate. This second phenomenon may explain recent reports of experimental systems producing binary sequences that pass statistical tests of randomness at rates that may be significantly beyond the metric entropy rate of the physical source.

The purpose of this study is to indicate fundamental propositions of the pre-image entropy related to a system proposed by Bufetov [Bufetov A. Topological entropy of free semigroup actions and skew-product transformations. J Dyn Control Syst 1999;5:137–143. 1.] for generating free semigroup actions. This study reveals the formula for the pre-image entropy of skew-product transformation with respect to the one-sided shift space. Finally, one example is presented to show how to obtain the pre-image entropy value for the skew-product transformation.

The entropy is a measurement of disorder or a measurement of heterogeneous information and it is known that amount of information cannot be negative. There are many methods about entropies by applied derivative to a probability distribution functions. In this paper, we applied fractional order derivative to probability distribution function, and this fractional order derivative is different than many of fractional order derivative methods in literature. Tsallis entropy was obtained by applying fractional order derivative methods in literature. In this paper, a new method for fractional order derivative was applied to probability distribution functions and two new definitions for entropy computation were obtained. One of them concludes in negative values for some cases, it is known that entropy cannot be negative, so, we selected the other definition (Definition 2) for entropy computation. The obtained new definition for entropy was applied to four probability distribution functions and obtained results were compared with Shannon entropies. The experimental results illustrated that the proposed method is superior to Shannon entropy method and the better entropy values were obtained for orders in interval (0,1).

Many chemical reaction networks in biological systems present complex oscillatory dynamics. In systems such as regulatory gene networks, cell cycle, and enzymatic processes, the number of molecules involved is often far from the thermodynamic limit. Although stochastic models based on the probabilistic approach of the Chemical Master Equation (CME) have been proposed, studies in the literature have been limited by the challenges of solving the CME and the lack of computational power to perform large-scale stochastic simulations. In this paper, we show that the infinite set of stationary moment equations describing the stochastic Brusselator and Schnakenberg oscillatory reactions networks can be truncated and solved using maximization of the entropy of the distributions. The results from our numerical experiments compare with the distributions obtained from well-established kinetic Monte Carlo methods and suggest that the accuracy of the prediction increases exponentially with the closure order chosen for the system. We conclude that maximum entropy models can be used as an efficient closure scheme alternative for moment equations to predict the non-equilibrium stationary distributions of stochastic chemical reactions with oscillatory dynamics. This prediction is accomplished without any prior knowledge of the system dynamics and without imposing any biased assumptions on the mathematical relations among species involved.

The Kolmogorov-Sinai (K-S) entropy is used to quantify the average amount of uncertainty of a dynamical system through a sequence of observations. Sequence probabilities therefore play a central role for the computation of the entropy rate to determine if the dynamical system under study is deterministically non-chaotic, deterministically chaotic, or random. This paper extends the notion of the K-S entropy to measure the entropy rate of imprecise systems using sequence membership grades, in which the underlying deterministic paradigm is replaced with the degree of fuzziness. While constructing sequential probabilities for the calculation of the K-S entropy is difficult in practice, the estimate of the K-S entropy in the setting of fuzzy sets in an image is feasible and can be useful for modeling uncertainty of pixel distributions in images. The fuzzy K-S entropy is illustrated as an effective feature for image analysis and texture classification. HighlightsDevelopment of a new method for feature extraction using chaos and fuzzy sets.Modeling Kolmogorov-Sinai (K-S) entropy with fuzzy logic is feasible.Fuzzy K-S entropy is an effective feature for analysis and classification of textural images.

Recently, Abbasnejad et al. (2010) proposed a measure of uncertainty based on survival function, called the survival entropy of order α. A dynamic form of the survival entropy of order α is also proposed by them. In this paper, we derive the weighted form of these measures. The properties of the new measures are also discussed.

The Shannon entropy and the cumulative residual entropy (CRE) of a random variable are useful tools in probability theory. Recently, a new concept called generalized cumulative residual entropy (GCRE) of order n was introduced and studied. It is related with the record values of a sequence of i.i.d. random variables and with the relevation transform. In this paper, we show that, under some assumptions, the GCRE function of a fixed order n uniquely determines the distribution function. Some characterizations of particular probability models are obtained from this general result.

Resultant information concepts combining descriptors of both the probability and phase/current distributions in molecular electronic states are summarized. Continuity equations for these fundamental densities are explored and the equilibrium states of molecular systems are determined from the relevant information principles. The extremum of the nonclassical entropy/information terms alone determines the system vertical-equilibrium state of the vanishing current. The maximum of the resultant entropy, which defines the system horizontal-equilibrium, corresponds to the optimum “thermodynamic” phase determined by the system particle distribution and predicts its finite current related to the gradient of electron density. This equilibrium phase-transformation conserves the probability density while modifying the current of the original state. Generalized measures of the entropy/information content give rise to a thermodynamic description of molecular equilibrium states, which establishes the entropy source and flux concepts in the underlying information continuity equation. The influence of the equilibrium-phase transformation on the continuity relations for the state probability distribution, phase density, and local entropy production is examined.

In the literature of information theory, the concept of generalized entropy has been proposed and the length-based shift dependent information measure has been studied. In this paper, the concept o...

This paper presents a similarity solution analysis of entropy generation due to heat transfer and fluid flow which has been carried out for laminar free convection from a constant temperature vertical plate in an infinite quiescent fluid. The governing partial differential equations are transformed into a set of ordinary differential equations using similarity variables. So an analytical expression, in terms of entropy generation, entropy generation number, Bejan number and irreversibility distribution ratio are derived using velocity and temperature similarity (exact) solution. The rate of entropy generation is investigated and discussed in details. The results presented by the similarity solution are compared with integral method results. The similarity solution presents more appropriate and correct distribution of entropy generation in boundary layer because more accuracy than integral method. It shows true position of maximum entropy generation and value of it. Also, the result shows that the exact solution minimizes the rate of total entropy generation in the boundary layer compared to integral solution. By introducing group parameter (GP number) which is the ratio of friction entropy to thermal entropy generation, one can recognize that one of the thermal entropy and friction entropy generation is dominated in the boundary layer.

Dynamics of non-stationary nonlinear processes that follow the maximum entropy principle (MaxEnt) for continuous probability distributions is considered. A set of equations describing the dynamics of probability density function (pdf) under the mass conservation and the energy conservation constraints is derived. The asymptotic convergence of evolving pdf and convergence of differential entropy are examined. It is shown that for pdfs with compact carrier the limit pdf is unique and coincides with the pdf obtained from Jaynes’s MaxEnt principle.

In the present study, several characterizations of order statistics are obtained on the basis of the generalized entropy. Under some conditions, it is shown that the parent distribution can be uniquely determined by equality of generalized entropy of order statistics.

In this paper, we consider and study a generalized entropy for record values distribution from some continuous probability models. Further, we study generalized residual entropy for series system and record values. At the end, we prove a characterization result for the generalized entropy of record values.

We suggest a transformation to obtain the negation of a probability distribution. We investigate the properties of this negation. Using the Dempster–Shafer theory of evidence, we show of all the possible negations our proposed negation is one having a maximal type entropy.

Biological networks are open systems that can utilize nutrients and energy from their environment for use in their metabolic processes, and produce metabolic products. System entropy is defined as the difference between input and output signal entropy, i.e., the net signal entropy of the biological system. System entropy is an important indicator for living or non-living biological systems, as biological systems can maintain or decrease their system entropy. In this study, system entropy is determined for the first time for stochastic biological networks, and a computation method is proposed to measure the system entropy of nonlinear stochastic biological networks that are subject to intrinsic random fluctuations and environmental disturbances. We find that intrinsic random fluctuations could increase the system entropy, and that the system entropy is inversely proportional to the robustness and stability of the biological networks. It is also determined that adding feedback loops to shift all eigenvalues to the farther left-hand plane of the complex s-domain could decrease the system entropy of a biological network.

We study approximations of non-Gaussian stationary processes having long range correlations with microcanonical models. These models are conditioned by the empirical value of an energy vector, evaluated on a single realization. Asymptotic properties of maximum entropy microcanonical and macrocanonical processes and their convergence to Gibbs measures are reviewed. We show that the Jacobian of the energy vector controls the entropy rate of microcanonical processes. Sampling maximum entropy processes through MCMC algorithms require too many operations when the number of constraints is large. We define microcanonical gradient descent processes by transporting a maximum entropy measure with a gradient descent algorithm which enforces the energy conditions. Convergence and symmetries are analyzed. Approximations of non-Gaussian processes with long range interactions are defined with multiscale energy vectors computed with wavelet and scattering transforms. Sparsity properties are captured with $\bf l^1$ norms. Approximations of Gaussian, Ising and point processes are studied, as well as image and audio texture synthesis., Accepted to MSL (Mathematical Statistics and Learning)

When information about a distribution consists of statistical moments only, a self-consistent approach to deriving a subjective probability density function (pdf) is Maximum Entropy. Nonetheless, the available information may have uncertainty, and statistical moments maybe known only to lie in a certain domain. If Maximum Entropy is used to find the distribution with the largest entropy whose statistical moments lie within the domain, the information at only a single point in the domain would be used and other information would be discarded. In this paper, the bounded information on statistical moments is used to construct a family of Maximum Entropy distributions, leading to an uncertain probability function. This uncertainty description enables the investigation of how the uncertainty in the probabilistic assignment affects the predicted performance of an engineering system with respect to safety, quality and design constraints. It is shown that the pdf which maximizes (or equivalently minimizes) an engineering metric is potentially different from the pdf which maximizes the entropy. The feasibility of the proposed uncertainty model is shown through its application to: (i) fatigue failure analysis of a structural joint; (ii) evaluation of the probability that a response variable of an engineering system exceeds a critical level, and (iii) random vibration.

In the literature of information theory, there exist many well known measures of entropy suitable for entropy optimization principles towards applications in different disciplines of science and technology. The object of this article is to develop a new generalized measure of entropy and to establish the relation between entropy and queueing theory. To fulfill our aim, we have made use of maximum entropy principle which provides the most uncertain probability distribution subject to some constraints expressed by mean values.

In this paper, we investigate a lossy source coding problem, where an upper limit on the permitted distortion is defined for every dataset element. It can be seen as an alternative approach to rate distortion theory where a bound on the allowed average error is specified. In order to find the entropy, which gives a statistical length of source code compatible with a fixed distortion bound, a corresponding optimization problem has to be solved. First, we show how to simplify this general optimization by reducing the number of coding partitions, which are irrelevant for the entropy calculation. In our main result, we present a fast and feasible for implementation greedy algorithm, which allows one to approximate the entropy within an additive error term of log2 e. The proof is based on the minimum entropy set cover problem, for which a similar bound was obtained.

A theoretical method based on maximum Shannon entropy framework (MSEF) in the presence of the geometric/or shifted fractional geometric mean of the queue size is applied to study the finite buffer system. Analytical expression of the loss probability for large buffer size is found to depict power law behavior. The maximum entropy framework is extended to incorporate additional shifted fractional arithmetic mean constraint to yield the expression of loss probability which is similar to the one derived by Kim and Shroff who have employed an entirely different approach based on maximum variance asymptotic (MVA) approximation. An important finding is the relationship between fractional order and the Hurst parameter. The advantage of MSEF is that it has enabled one to derive the analytical closed form generalized expression of the probability distribution of queue size in finite buffer system. Further, MSEF with shifted fractional geometric mean constraint gives the same distribution of queue size as the one obtained by maximizing Tsallis entropy subject to the fractional arithmetic mean of queue size.

In this paper, we present entropy satisfying schemes for solving an integro-differential equation that describes the evolution of a population structured with respect to a continuous trait. In [P.-E. Jabin and G. Raoul, J. Math. Biol., 63 (2011), pp. 493--517] solutions are shown to converge toward the so-called evolutionary stable distribution (ESD) as time becomes large, using the relative entropy. At the discrete level, the ESD is shown to be the solution to a quadratic programming problem and can be computed by any well-established nonlinear programing algorithm. The schemes are then shown to satisfy the entropy dissipation inequality on the set where initial data are positive and the numerical solutions tend toward the discrete ESD in time. An alternative algorithm is presented to capture the global ESD for nonnegative initial data, which is made possible due to the mutation mechanism built into the modified scheme. A series of numerical tests are given to confirm both accuracy and the entropy satisf...

There are at least three distinct ways to conceptualize entropy: entropy as an extensive thermodynamic quantity of physical systems (Clausius, Boltzmann, Gibbs), entropy as a measure for information production of ergodic sources (Shannon), and entropy as a means for statistical inference on multinomial processes (Jaynes maximum entropy principle). Even though these notions represent fundamentally different concepts, the functional form of the entropy for thermodynamic systems in equilibrium, for ergodic sources in information theory, and for independent sampling processes in statistical systems, is degenerate, H(p)=-∑_{i}p_{i}logp_{i}. For many complex systems, which are typically history-dependent, nonergodic, and nonmultinomial, this is no longer the case. Here we show that for such processes, the three entropy concepts lead to different functional forms of entropy, which we will refer to as S_{EXT} for extensive entropy, S_{IT} for the source information rate in information theory, and S_{MEP} for the entropy functional that appears in the so-called maximum entropy principle, which characterizes the most likely observable distribution functions of a system. We explicitly compute these three entropy functionals for three concrete examples: for Pólya urn processes, which are simple self-reinforcing processes, for sample-space-reducing (SSR) processes, which are simple history dependent processes that are associated with power-law statistics, and finally for multinomial mixture processes.

An analytical model to predict the velocity-dip-position is presented in this study. Unlike the previous studies where empirical or semiempirical models were suggested, in this study the model is derived by using entropy theory. Using the principle of maximum entropy, the model for dip-position is derived by maximizing the Shannon entropy function after assuming dimensionless dip-position as a random variable. No estimation of empirical parameter is required for calculating dip-position from the proposed model. The model is capable of predicting the velocity-dip-position at any section from sidewall region (along lateral direction) of an open channel with any aspect ratio. The ratio of mean to maximum value of dip-position is analyzed from data, and it is found that ratio is almost constant for narrow open channels and it increases with aspect ratio for wide open channels. A relation is also proposed to predict this ratio in case of wide open channels. The developed model of velocity-dip-position is tested with existing experimental data for a wide range of aspect ratio and is also compared with other empirical models. The present model shows good agreement with the observed data and is comparable with the existing models.

In this paper, we identify a class of absolutely continuous probability distributions, and show that the differential entropy is uniformly convergent over this space under the metric of total variation distance. One of the advantages of this class is that the requirements could be readily verified for a given distribution., 4 pages

Objective : A critical point in any definition of entropy is the selection of the parameters employed to obtain an estimate in practice. We propose a new definition of entropy aiming to reduce the significance of this selection. Methods: We call the new definition Bubble Entropy . Bubble Entropy is based on permutation entropy, where the vectors in the embedding space are ranked. We use the bubble sort algorithm for the ordering procedure and count instead the number of swaps performed for each vector. Doing so, we create a more coarse-grained distribution and then compute the entropy of this distribution. Results: Experimental results with both real and synthetic HRV signals showed that bubble entropy presents remarkable stability and exhibits increased descriptive and discriminating power compared to all other definitions, including the most popular ones. Conclusion: The definition proposed is almost free of parameters. The most common ones are the scale factor r and the embedding dimension m . In our definition, the scale factor is totally eliminated and the importance of m is significantly reduced. The proposed method presents increased stability and discriminating power. Significance: After the extensive use of some entropy measures in physiological signals, typical values for their parameters have been suggested, or at least, widely used. However, the parameters are still there, application and dataset dependent, influencing the computed value and affecting the descriptive power. Reducing their significance or eliminating them alleviates the problem, decoupling the method from the data and the application, and eliminating subjective factors.

The scale, inflexion point and maximum point are important scaling parameters for studying growth phenomena with a size following the lognormal function. The width of the size function and its entropy depend on the scale parameter (or the standard deviation) and measure the relative importance of production and dissipation involved in the growth process. The Shannon entropy increases monotonically with the scale parameter, but the slope has a minimum at p 6/6. This value has been used previously to study spreading of spray and epidemical cases. In this paper, this approach of minimizing this entropy slope is discussed in a broader sense and applied to obtain the relationship between the inflexion point and maximum point. It is shown that this relationship is determined by the base of natural logarithm e ' 2.718 and exhibits some geometrical similarity to the minimal surface energy principle. The known data from a number of problems, including the swirling rate of the bathtub vortex, more data of droplet splashing, population growth, distribution of strokes in Chinese language characters and velocity profile of a turbulent jet, are used to assess to what extent the approach of minimizing the entropy slope can be regarded as useful.

We study the entropy production of systems out of equilibrium using random networks. We focus on systems with a finite number of states described by a master equation close to equilibrium. The dynamics are mapped into a network of states (nodes) connected by transition rates (links). Using this framework, we analyze the entropy production of ensembles of randomly generated networks owing to specific constraints (e.g., size or symmetries) and identify the most important parameters that determine its value. This analysis gives a null-model estimation for the entropy production that can be used for comparison with specific systems.

True random number generators (TRNGs) are essential for cryptographic systems, and they are usually evaluated by the concept of entropy. In general, the entropy of a TRNG is estimated from its stochastic model, and reflected in the statistical results of the generated raw bits. Oscillator-based TRNGs are widely used in practical cryptographic systems due to its elegant structure, and its stochastic model has been studied in different aspects. In this paper, we investigate the applicability of the different entropy estimation methods for oscillator-based TRNGs, including the bit-rate entropy, the lower bound and the approximate entropy. Particularly, we firstly analyze the two existing stochastic models (one of which is phase-based and the other is time-based), and deduce consistent bit-rate entropy results from these two models. Then, we design an approximate entropy calculation method on the output raw bits of a simulated oscillator-based TRNG, and this statistical calculation result well matches the bit-rate entropy from stochastic models. In addition, we discuss the extreme case of tiny randomness where some methods are inapplicable, and provide the recommendations for these entropy evaluation methods. Finally, we design a hardware verification method in a real oscillator-based TRNG, and validate these estimation methods in the hardware platform.

Similar to fuzzy set on a possibility space, uncertain set is a set-valued function on an uncertainty space, and attempts to model unsharp concepts. Entropy provides a quantitative measurement of the uncertainty associated with an uncertain set. This paper presents a formula for calculating the entropy of an uncertain set via its inverse membership function. Based on the formula, the entropy operator is shown to satisfy positive linearity property. In addition, this paper proposes a concept of relative entropy to describe the divergence between the membership functions of two uncertain sets.

By assuming latent sources are statistically independent, independent component analysis separates underlying sources from a given linear mixture. Since in many applications, latent sources are both non-Gaussian and have sample dependence, it is desirable to exploit both properties jointly. In this paper, we use mutual information rate to construct a general framework for analysis and derivation of algorithms that take both properties into account. We discuss two types of source models for entropy rate estimation-a Markovian and an invertible filter model-and give the general independent component analysis cost function, update rule, and performance analysis based on these. We also introduce four algorithms based on these two models, and show that their performance can approach the Cramer-Rao lower bound. In addition, we demonstrate that the algorithms with flexible models exhibit very desirable performance for “natural” data.