Search

In various applications, multiscale entropy (MSE) is often used as a feature to characterize the complexity of the signals in order to classify them. It consists of estimating the sample entropies (SEs) of the signal under study and its coarse-grained (CG) versions, where the CG process amounts to (1) filtering the signal with an average filter whose order is the scale and (2) decimating the filter output by a factor equal to the scale. In this paper, we propose to derive a new variant of the MSE. Its novelty stands in the way to get the sequences at different scales by avoiding distortions during the decimation step. To this end, a linear-phase or null-phase low-pass filter whose cutoff frequency is well suited to the scale is used. Interpretations on how the MSE behaves and illustrations with a sum of sinusoids, as well as white and pink noises, are given. Then, an application to detect attentional tunneling is presented. It shows the benefit of the new approach in terms of p value when one aims at differentiating the set of MSEs obtained in the attentional tunneling state from the set of MSEs obtained in the nominal state. It should be noted that CG versions can be replaced not only for the MSE but also for other variants. [ABSTRACT FROM AUTHOR]

Abstract Regularity is an important aspect of physical activity that can provide valuable insights into how individuals engage in physical activity over time. Accurate measurement of regularity not only advances our understanding of physical activity behavior but also facilitates the development of human activity modeling and forecasting. Furthermore, it can inform the design and implementation of tailored interventions to improve population health outcomes. In this paper, we aim to assess the regularity of physical activities through longitudinal sensor data, which reflects individuals’ all physical activities over an extended period. We explore three entropy models, including entropy rate, approximate entropy, and sample entropy, which can potentially offer a more comprehensive evaluation of physical activity regularity compared to metrics based solely on periodicity or stability. We propose a framework to validate the performance of entropy models on both synthesized and real-world physical activity data. The results indicate entropy rate is able to identify not only the magnitude and amount of noise but also macroscopic variations of physical activities, such as differences on duration and occurrence time. Simultaneously, entropy rate is highly correlated with the predictability of real-world samples, further highlighting its applicability in measuring human physical activity regularity. Leveraging entropy rate, we further investigate the regularity for 686 individuals. We find the composition of physical activities can partially explain the difference in regularity among individuals, and the majority of individuals exhibit temporal stability of regularity.

Regularity is an important aspect of physical activity that can provide valuable insights into how individuals engage in physical activity over time. Accurate measurement of regularity not only advances our understanding of physical activity behavior but also facilitates the development of human activity modeling and forecasting. Furthermore, it can inform the design and implementation of tailored interventions to improve population health outcomes. In this paper, we aim to assess the regularity of physical activities through longitudinal sensor data, which reflects individuals' all physical activities over an extended period. We explore three entropy models, including entropy rate, approximate entropy, and sample entropy, which can potentially offer a more comprehensive evaluation of physical activity regularity compared to metrics based solely on periodicity or stability. We propose a framework to validate the performance of entropy models on both synthesized and real-world physical activity data. The results indicate entropy rate is able to identify not only the magnitude and amount of noise but also macroscopic variations of physical activities, such as differences on duration and occurrence time. Simultaneously, entropy rate is highly correlated with the predictability of real-world samples, further highlighting its applicability in measuring human physical activity regularity. Leveraging entropy rate, we further investigate the regularity for 686 individuals. We find the composition of physical activities can partially explain the difference in regularity among individuals, and the majority of individuals exhibit temporal stability of regularity. [ABSTRACT FROM AUTHOR]

Entropy measures the randomness or uncertainty of a stochastic process, and the entropy rate refers to the limit of the time average of entropy. The generalized entropy rate in the form of delayed averages can overcome the redundancy of initial information while ensuring stationarity. Therefore, it has better practical value. A Hidden Markov Model (HMM) contains two stochastic processes, a stochastic process in which all states can be observed and a Markov chain in which all states cannot be observed. The entropy rate is an important characteristic of HMMs. The transition matrix of a homogeneous HMM is unique, while a Nonhomogeneous Hidden Markov Model (NHMM) requires the transition matrices to be dependent on time variables. From the perspective of model structure, NHMMs are novel extensions of homogeneous HMMs. In this paper, the concepts of the generalized entropy rate and NHMMs are defined and fully explained, a strong limit theorem and limit properties of a norm are presented, and then generalized entropy ergodic theorems with an almost surely convergence for NHMMs are obtained. These results provide concise formulas for the computation and estimation of the generalized entropy rate for NHMMs. [ABSTRACT FROM AUTHOR]

Estimating the entropy rate of physical random number generators with uncertainty is crucial for information security applications. We evaluate the sample entropy of chaotic temporal waveforms generated experimentally by a semiconductor laser with time-delayed optical feedback. We demonstrate random number generation with uncertainty using a quantitative measurement of the entropy rate.

In various applications, multiscale entropy (MSE) is often used as a feature to characterize the complexity of the signals in order to classify them. It consists of estimating the sample entropies (SEs) of the signal under study and its coarse-grained (CG) versions, where the CG process amounts to (1) filtering the signal with an average filter whose order is the scale and (2) decimating the filter output by a factor equal to the scale. In this paper, we propose to derive a new variant of the MSE. Its novelty stands in the way to get the sequences at different scales by avoiding distortions during the decimation step. To this end, a linear-phase or null-phase low-pass filter whose cutoff frequency is well suited to the scale is used. Interpretations on how the MSE behaves and illustrations with a sum of sinusoids, as well as white and pink noises, are given. Then, an application to detect attentional tunneling is presented. It shows the benefit of the new approach in terms of p value when one aims at differentiating the set of MSEs obtained in the attentional tunneling state from the set of MSEs obtained in the nominal state. It should be noted that CG versions can be replaced not only for the MSE but also for other variants.

Entropy measures the randomness or uncertainty of a stochastic process, and the entropy rate refers to the limit of the time average of entropy. The generalized entropy rate in the form of delayed averages can overcome the redundancy of initial information while ensuring stationarity. Therefore, it has better practical value. A Hidden Markov Model (HMM) contains two stochastic processes, a stochastic process in which all states can be observed and a Markov chain in which all states cannot be observed. The entropy rate is an important characteristic of HMMs. The transition matrix of a homogeneous HMM is unique, while a Nonhomogeneous Hidden Markov Model (NHMM) requires the transition matrices to be dependent on time variables. From the perspective of model structure, NHMMs are novel extensions of homogeneous HMMs. In this paper, the concepts of the generalized entropy rate and NHMMs are defined and fully explained, a strong limit theorem and limit properties of a norm are presented, and then generalized entropy ergodic theorems with an almost surely convergence for NHMMs are obtained. These results provide concise formulas for the computation and estimation of the generalized entropy rate for NHMMs.

This paper is devoted to the study of the properties of entropy as a function of the Hurst index, which corresponds to the fractional Gaussian noise. Since the entropy of the Gaussian vector depends on the determinant of the covariance matrix, and the behavior of this determinant as a function of the Hurst index is rather difficult to study analytically at high dimensions, we also consider simple alternative entropy functionals, whose behavior, on the one hand, mimics the behavior of entropy and, on the other hand, is not difficult to study. Asymptotic behavior of the normalized entropy (so called entropy rate) is also studied for the entropy and for the alternative functionals. [ABSTRACT FROM AUTHOR]

In this paper, we analyze the values of the entropic efficiency of longitudinal fins by investigating the coupling between the function describing the fin profile and the corresponding steady-state temperature distribution along the fin. By starting from different boundary conditions, we look at the distribution temperature maximizing the efficiency of the fin. From this temperature distribution and by requesting that the fin must comply with natural physical constraints, such as the maximum fin thickness, we obtain an optimal profile for a purely convective fin and a convecting–radiating fin. For different boundary conditions and for a maximum fin thickness equal to four (in dimensionless units), both the profiles are increasing starting from the fin base until they reach the maximum value and then decrease to zero at the tip. Analytic and numerical results, together with different plots, are presented. [ABSTRACT FROM AUTHOR]

This paper focuses on the ultimate limit theory of image compression. It proves that for an image source, there exists a coding method with shapes that can achieve the entropy rate under a certain condition where the shape-pixel ratio in the encoder/decoder is O (1 / log t) . Based on the new finding, an image coding framework with shapes is proposed and proved to be asymptotically optimal for stationary and ergodic processes. Moreover, the condition O (1 / log t) of shape-pixel ratio in the encoder/decoder has been confirmed in the image database MNIST, which illustrates the soft compression with shape coding is a near-optimal scheme for lossless compression of images. [ABSTRACT FROM AUTHOR]

We study the multiplicative version of the classical Furstenberg's filtering problem, where instead of the sum X + Y one considers the product X · Y (X and Y are bilateral, real, finitely-valued, stationary independent processes, Y is taking values in { 0 , 1 } ). We provide formulas for H (X · Y | Y) . As a consequence, we show that if H (X) > H (Y) = 0 and X ⨿ Y , then H (X · Y) < H (X) (and thus X cannot be filtered out from X · Y ) whenever X is not bilaterally deterministic, Y is ergodic and Y first return to 1 can take arbitrarily long with positive probability. On the other hand, if almost surely Y visits 1 along an infinite arithmetic progression of a fixed difference (with possibly some more visits in between) then we can find X that is not bilaterally deterministic and such that H (X · Y) = H (X) . As a consequence, a B -free system (X η , S) is proximal if and only if there is always an entropy drop h (κ ∗ ν η) < h (κ) for any κ corresponding to a non-bilaterally deterministic process of positive entropy. These results partly settle some open problems on invariant measures for B -free systems. [ABSTRACT FROM AUTHOR]

Motivated by DNA storage in living organisms, and by known biological mutation processes, we study the reverse-complement string-duplication system. We fully classify the conditions under which the system has full expressiveness, for all alphabets and all fixed duplication lengths. We then focus on binary systems with duplication length 2 and prove that they have full capacity, yet surprisingly, have zero entropy-rate. Finally, by using binary single burst-insertion correcting codes, we construct codes that correct a single reverse-complement duplication of odd length, over any alphabet. The redundancy (in bits) of the constructed code does not depend on the alphabet size. [ABSTRACT FROM AUTHOR]

Do speakers use less redundant language with more proficient interlocutors? Both the communicative efficiency framework and the language development literature predict that speech directed to younger infants should be more redundant than speech directed to older infants. Here, we test this by quantifying redundancy in infant-directed speech using entropy rate – an information-theoretic measure reflecting average degree of repetitiveness. While IDS is often described as repetitive, entropy rate provides a novel holistic measure of redundancy in this speech genre. Using two developmental corpora, we compare entropy rates of samples taken from different ages. We find that parents use less redundant speech when talking to older children, illustrating an effect of perceived interlocutor proficiency on redundancy. The developmental decrease in redundancy reflects a decrease in lexical repetition, but also a decrease in repetitions of multi-word sequences, highlighting the importance of larger sequences in early language learning. [ABSTRACT FROM AUTHOR]

Objective: Temporal patterns in neuronal spiking encode stimulus uncertainty, and convey information about high-level functions such as memory and cognition. Estimating the associated information content and understanding how that evolves with time assume significance in the investigation of neuronal coding mechanisms and abnormal signaling. However, existing estimators of the entropy rate, a measure of information content, either ignore the inherent nonstationarity, or employ dictionary-based Lempel-Ziv (LZ) methods that converge too slowly for one to study temporal variations in sufficient detail. Against this backdrop, we seek estimates that handle nonstationarity, are fast converging, and hence allow meaningful temporal investigations. &#xD;Approach: We proposed a homogeneous Markov model approximation of spike trains within windows of suitably chosen length and an entropy rate estimator based on empirical probabilities that converges quickly. &#xD;Main results: We constructed mathematical families of nonstationary Markov processes with certain bi/multi-level properties (inspired by neuronal responses) with known entropy rates, and validated the proposed estimator against those. Further statistical validations were presented on data collected from hippocampal (and primary visual cortex) neuron populations in terms of single neuron behavior as well as population heterogeneity. &#xD;Our estimator appears to be statistically more accurate and converges faster than &#xD;existing LZ estimators, and hence well suited for temporal studies. &#xD;Significance: The entropy rate analysis revealed not only informational and process memory heterogeneity among neurons, but distinct statistical patterns in neuronal populations (from two different brain regions) under basal and post-stimulus conditions. Taking inspiration, we envision future large-scale studies of different brain regions enabled by the proposed tool (estimator), potentially contributing to improved functional modeling of the brain and identification of statistical signatures of neurodegenerative diseases.&#xD., (© 2024 IOP Publishing Ltd.)

We introduce the minimal maximally predictive models (ϵ-machines) of processes generated by certain hidden semi-Markov models. Their causal states are either discrete, mixed, or continuous random variables and causal-state transitions are described by partial differential equations. As an application, we present a complete analysis of the ϵ-machines of continuous-time renewal processes. This leads to closed-form expressions for their entropy rate, statistical complexity, excess entropy, and differential information anatomy rates.

Key to biological success, the requisite variety that confronts an adaptive organism is the set of detectable, accessible, and controllable states in its environment. We analyze its role in the thermodynamic functioning of information ratchets—a form of autonomous Maxwellian Demon capable of exploiting fluctuations in an external information reservoir to harvest useful work from a thermal bath. This establishes a quantitative paradigm for understanding how adaptive agents leverage structured thermal environments for their own thermodynamic benefit. General ratchets behave as memoryful communication channels, interacting with their environment sequentially and storing results to an output. The bulk of thermal ratchets analyzed to date, however, assume memoryless environments that generate input signals without temporal correlations. Employing computational mechanics and a new information-processing Second Law of Thermodynamics (IPSL) we remove these restrictions, analyzing general finite-state ratchets interacting with structured environments that generate correlated input signals. On the one hand, we demonstrate that a ratchet need not have memory to exploit an uncorrelated environment. On the other, and more appropriate to biological adaptation, we show that a ratchet must have memory to most effectively leverage structure and correlation in its environment. The lesson is that to optimally harvest work a ratchet’s memory must reflect the input generator’s memory. Finally, we investigate achieving the IPSL bounds on the amount of work a ratchet can extract from its environment, discovering that finite-state, optimal ratchets are unable to reach these bounds. In contrast, we show that infinite-state ratchets can go well beyond these bounds by utilizing their own infinite “negentropy”. We conclude with an outline of the collective thermodynamics of information-ratchet swarms.

In this paper, a random walk on t-designs are considered. We assign a weight to each block and walk randomly on the vertices with a probability proportional to the weight of blocks. This stochastic process is a Markov chain. We obtain a stationary distribution for this process and compute its entropy rate. It is seen that, when the blocks have the same weight, the uniform distribution on the vertices is a stationary distribution and the entropy rate depends only on the number of vertices.

We give exact formulae for a wide family of complexity measures that capture the organization of hidden nonlinear processes. The spectral decomposition of operator-valued functions leads to closed-form expressions involving the full eigenvalue spectrum of the mixed-state presentation of a process's ε-machine causal-state dynamic. Measures include correlation functions, power spectra, past-future mutual information, transient and synchronization informations, and many others. As a result, a direct and complete analysis of intrinsic computation is now available for the temporal organization of finitary hidden Markov models and nonlinear dynamical systems with generating partitions and for the spatial organization in one-dimensional systems, including spin systems, cellular automata, and complex materials via chaotic crystallography.

We introduce a family of Maxwellian Demons for which correlations among information bearing degrees of freedom can be calculated exactly and in compact analytical form. This allows one to precisely determine Demon functional thermodynamic operating regimes, when previous methods either misclassify or simply fail due to approximations they invoke. This reveals that these Demons are more functional than previous candidates. They too behave either as engines, lifting a mass against gravity by extracting energy from a single heat reservoir, or as Landauer erasers, consuming external work to remove information from a sequence of binary symbols by decreasing their individual uncertainty. Going beyond these, our Demon exhibits a new functionality that erases bits not by simply decreasing individual-symbol uncertainty, but by increasing inter-bit correlations (that is, by adding temporal order) while increasing single-symbol uncertainty. In all cases, but especially in the new erasure regime, exactly accounting for informational correlations leads to tight bounds on Demon performance, expressed as a refined Second Law of thermodynamics that relies on the Kolmogorov-Sinai entropy for dynamical processes and not on changes purely in system configurational entropy, as previously employed. We rigorously derive the refined Second Law under minimal assumptions and so it applies quite broadly - for Demons with and without memory and input sequences that are correlated or not. We note that general Maxwellian Demons readily violate previously proposed, alternative such bounds, while the current bound still holds. As such, it broadly describes the minimal energetic cost of any computation by a thermodynamic system.

This paper tries to achieve a solution for problems that concern condensation around a flat plate, circular and elliptical tube in by numerical and analytical methods. Also, it calculates entropy production rates. At first, a problem was solved with mesh dynamic and rational assumptions; next it was compared with the numerical solution that the result had acceptable errors. An additional supporting relation is applied based on the characteristic of the condensation phenomenon for condensing elements. As it is shown here, due to higher rates of heat transfer for elliptical tubes, they have more entropy production rates, in comparison to circular ones. Findings showed that the two methods were efficient. Furthermore, analytical methods can be used to optimize the problem and reduce the entropy production rate.

This paper focuses on the ultimate limit theory of image compression. It proves that for an image source, there exists a coding method with shapes that can achieve the entropy rate under a certain condition where the shape-pixel ratio in the encoder/decoder is O(1/logt). Based on the new finding, an image coding framework with shapes is proposed and proved to be asymptotically optimal for stationary and ergodic processes. Moreover, the condition O(1/logt) of shape-pixel ratio in the encoder/decoder has been confirmed in the image database MNIST, which illustrates the soft compression with shape coding is a near-optimal scheme for lossless compression of images.

In this paper, we analyze the values of the entropic efficiency of longitudinal fins by investigating the coupling between the function describing the fin profile and the corresponding steady-state temperature distribution along the fin. By starting from different boundary conditions, we look at the distribution temperature maximizing the efficiency of the fin. From this temperature distribution and by requesting that the fin must comply with natural physical constraints, such as the maximum fin thickness, we obtain an optimal profile for a purely convective fin and a convecting–radiating fin. For different boundary conditions and for a maximum fin thickness equal to four (in dimensionless units), both the profiles are increasing starting from the fin base until they reach the maximum value and then decrease to zero at the tip. Analytic and numerical results, together with different plots, are presented.

Purpose: Develop a diabetes diagnostic tool based on two markers of continuous glucose monitoring (CGM) dynamics: CGM entropy rate (ER) and Poincaré plot (PP) ellipse area (S)., Methods: 5,754 daily CGM profiles from 843 individuals with type 1, type 2 diabetes, or healthy individuals with or without islet autoantibody status were used to compute two individual dynamic markers: ER (in bits per transition; BPT) of daily probability matrices describing CGM transitions between eight glycemic states, and the area S (mg2/dL2) of individual CGM PP ellipses using standard PP descriptors. The Youden's index was used to determine "optimal" cut-points for ER and S for health vs. diabetes (case 1); type 1 vs. type 2 (case 2); and low vs. high type 1 immunological risk (case 3). The markers' discriminative power was assessed through the area under the receiver operating characteristics curves (AUC)., Results: Optimal cut-off points were determined for ER and S for each of the three cases. ER and S discriminated case 1 with AUC = 0.98 (95% CI: 0.97-0.99) and AUC = 0.99 (95% CI: 0.99-1.00), respectively, (cut-offs ERcase1 = 0.76 BPT, Scase1 = 1993.91 mg2/dL2), case 2 with AUC = 0.81 (95% CI, 0.77-0.84) and AUC = 0.76 (95% CI, 0.72-0.81), respectively (ERcase2 = 1.00 BPT, Scase2 = 5112.98 mg2/dL2), and case 3 with AUC = 0.81 (95% CI, 0.77-0.84) and AUC = 0.76 (95% CI, 0.72-0.81), respectively (ERcase3 = 0.52 BPT, Scase3 = 923.65 mg2/dL2)., Conclusions: CGM dynamics markers can be an alternative to fasting plasma glucose or glucose tolerance testing and identifying individuals at higher immunological risk of progressing to type 1 diabetes., (© The Author(s) 2024. Published by Oxford University Press on behalf of the Endocrine Society. All rights reserved. For commercial re-use, please contact reprints@oup.com for reprints and translation rights for reprints. All other permissions can be obtained through our RightsLink service via the Permissions link on the article page on our site—for further information please contact journals.permissions@oup.com. See the journal About page for additional terms.)

The mutual information between stimulus and spike-train response is commonly used to monitor neural coding efficiency, but neuronal computation broadly conceived requires more refined and targeted information measures of input-output joint processes. A first step toward that larger goal is to develop information measures for individual output processes, including information generation (entropy rate), stored information (statistical complexity), predictable information (excess entropy), and active information accumulation (bound information rate). We calculate these for spike trains generated by a variety of noise-driven integrate-and-fire neurons as a function of time resolution and for alternating renewal processes. We show that their time-resolution dependence reveals coarse-grained structural properties of interspike interval statistics; e.g., τ-entropy rates that diverge less quickly than the firing rate indicated by interspike interval correlations. We also find evidence that the excess entropy and regularized statistical complexity of different types of integrate-and-fire neurons are universal in the continuous-time limit in the sense that they do not depend on mechanism details. This suggests a surprising simplicity in the spike trains generated by these model neurons. Interestingly, neurons with gamma-distributed ISIs and neurons whose spike trains are alternating renewal processes do not fall into the same universality class. These results lead to two conclusions. First, the dependence of information measures on time resolution reveals mechanistic details about spike train generation. Second, information measures can be used as model selection tools for analyzing spike train processes.

Renewal processes are broadly used to model stochastic behavior consisting of isolated events separated by periods of quiescence, whose durations are specified by a given probability law. Here, we identify the minimal sufficient statistic for their prediction (the set of causal states), calculate the historical memory capacity required to store those states (statistical complexity), delineate what information is predictable (excess entropy), and decompose the entropy of a single measurement into that shared with the past, future, or both. The causal state equivalence relation defines a new subclass of renewal processes with a finite number of causal states despite having an unbounded interevent count distribution. We use the resulting formulae to analyze the output of the parametrized Simple Nonunifilar Source, generated by a simple two-state hidden Markov model, but with an infinite-state ε-machine presentation. All in all, the results lay the groundwork for analyzing more complex processes with infinite statistical complexity and infinite excess entropy.

The overarching goal of the present work is to contribute to the understanding of the relations between fetal heart rate (FHR) temporal dynamics and the well-being of the fetus, notably in terms of predicting the evolution of lactate, pH and cardiovascular decompensation (CVD). It makes uses of an established animal model of human labor, where 14 near-term ovine fetuses subjected to umbilical cord occlusions (UCO) were instrumented to permit regular intermittent measurements of metabolites lactate and base excess, pH, and continuous recording of electrocardiogram (ECG) and systemic arterial blood pressure (to identify CVD) during UCO. ECG-derived FHR was digitized at the sampling rate of 1,000 Hz and resampled to 4 Hz, as used in clinical routine. We focused on four FHR variability features which are tunable to temporal scales of FHR dynamics, robustly computable from FHR sampled at 4 Hz and within short-time sliding windows, hence permitting a time-dependent, or local, analysis of FHR which helps dealing with signal noise. Results show the sensitivity of the proposed features for early detection of CVD, correlation to metabolites and pH, useful for early acidosis detection and the importance of coarse time scales (2.5–8 s) which are not disturbed by the low FHR sampling rate. Further, we introduce the performance of an individualized self-referencing metric of the distance to healthy state, based on a combination of the four features. We demonstrate that this novel metric, applied to clinically available FHR temporal dynamics alone, accurately predicts the time occurrence of CVD which heralds a clinically significant degradation of the fetal health reserve to tolerate the trial of labor.

The hallmark of deterministic chaos is that it creates information - the rate being given by the Kolmogorov-Sinai metric entropy. Since its introduction half a century ago, the metric entropy has been used as a unitary quantity to measure a system's intrinsic unpredictability. Here, we show that it naturally decomposes into two structurally meaningful components: A portion of the created information - the ephemeral information - is forgotten and a portion - the bound information - is remembered. The bound information is a new kind of intrinsic computation that differs fundamentally from information creation: it measures the rate of active information storage. We show that it can be directly and accurately calculated via symbolic dynamics, revealing a hitherto unknown richness in how dynamical systems compute. © 2014 Elsevier B.V.

A stochastic nonlinear dynamical system generates information, as measured by its entropy rate. Some-the ephemeral information-is dissipated and some-the bound information-is actively stored and so affects future behavior. We derive analytic expressions for the ephemeral and bound information in the limit of infinitesimal time discretization for two classical systems that exhibit dynamical equilibria: first-order Langevin equations (i) where the drift is the gradient of an analytic potential function and the diffusion matrix is invertible and (ii) with a linear drift term (Ornstein-Uhlenbeck), but a noninvertible diffusion matrix. In both cases, the bound information is sensitive to the drift and diffusion, while the ephemeral information is sensitive only to the diffusion matrix and not to the drift. Notably, this information anatomy changes discontinuously as any of the diffusion coefficients vanishes, indicating that it is very sensitive to the noise structure. We then calculate the information anatomy of the stochastic cusp catastrophe and of particles diffusing in a heat bath in the overdamped limit, both examples of stochastic gradient descent on a potential landscape. Finally, we use our methods to calculate and compare approximations for the time-local predictive information for adaptive agents.

We present two examples of finite-alphabet, infinite excess entropy processes generated by stationary hidden Markov models (HMMs) with countable state sets. The first, simpler example is not ergodic, but the second is. These are the first explicit constructions of processes of this type. © 2014 by the authors.

Understanding the travel behavior of private cars will generate promising solutions on addressing urban problems such as alleviating traffic congestion and improving transport services. In this paper, we focus on investigating the individual travel patterns of private car users based on a large-scale private car trajectory dataset. To achieve this goal, we first analyze the stop-and-wait information from the private car trajectory data and utilize DBSCAN method to implement clustering with the aim at identifying the frequently-visit places (FVPs). After that, we leverage Markov chain to study the spatial-temporal transition characteristics when private cars travel among their FVPs. Finally yet importantly, we design the concept of spatial-temporal entropy rate and conduct a quantitative study for measuring the regularity of each individual private car’s mobility. We validate the proposed methodology based on a real-world dataset including 25,564 private cars driving during one month in China. Extensive experiments demonstrate that the proposed method outperforms the existing methods in terms of the accuracy on measuring the mobility behavior. Moreover, we observe that, on one side, the travel pattern is easier to mine from the private car users with fewer FVPs, on the other side, there are also a small number of users whose FVPs are large, while their mobility are relatively regular. Our work is the first effort to explore individual travel patterns of private car users via studying private car trajectory big data, thereby being able to provide new insight into the research of human travel activities, traffic management and urban planning. [ABSTRACT FROM AUTHOR]

We show that the typical coordinate-wise encoding of multivariate ergodic source into pre-scribed alphabets has the entropy profile close to the convolution of the entropy profile of the source and the modular polymatroid that is determined by the cardinalities of the output alphabets. We show that the proportion of the exceptional encodings that are not close to the convolution goes to zero doubly exponentially. The result holds for a class of multivariate sources that satisfy asymptotic equipartition property described via the mean uctuation of the information functions. This class covers asymptotically mean stationary processes with ergodic mean, ergodic processes, irreducible Markov chains with an arbitrary initial distribution. We also proved that typical encodings yield the asymptotic equipartition property for the output variables. These asymptotic results are based on an explicit lower bound of the proportion of encodings that transform a multivariate random variable into a variable with the entropy profile close to the suitable convolution. [ABSTRACT FROM AUTHOR]

Considering a low power wide area random access system (like LoRaWAN) where the individual transmitters are mobile, we 1) propose a way of quantifying the performance of tracking of the mobile devices, and 2) design a distributed algorithm to achieve a target tracking performance. The insights gained are then used to provide an analysis of a family of target-achieving reinforcement-learning algorithms used in the literature to learn the optimal (Nash Equilibrium) random access probabilities. By mapping the payoff function in the equivalent game to the performance metric, we establish that, under some general conditions of inter-node parameter separability, the algorithm convergence is independent of the payoff function used. The mapping from the desired performance metric to the success probability in random access can be used in the algorithm to achieve a target success probability. [ABSTRACT FROM AUTHOR]

A long-standing question in applications of dynamical systems theory is how to distinguish noisy signals from chaotic steady states. Information-theoretic measures hold promise to resolve this problem. We apply two such measures to numerically computed phase-space trajectories of continuous-state nonlinear oscillators: forecasting or statistical complexity, which quantifies the minimum memory required for the optimal prediction of discrete observables, and the entropy rate, which quantifies their intrinsic unpredictability. We estimate empirical generating partitions to obtain discrete observables faithfully representing continuous-state chaotic time series. We focus on the problem of distinguishing stochastically perturbed periodic orbits from chaotic attractors that exist at nearby parameter values, in a region of the parameter space where a strange invariant set exists. We find that a stochastically perturbed, stable, high-period ( p = 15 ) orbit of a periodically driven Duffing oscillator admits high values of both information measures, making it difficult to distinguish it from chaotic states at adjacent parameters, even with small noise. However, for a low-period ( p = 3 ) orbit, such a distinction becomes easier, as both measures admit considerably lower values compared to a chaotic attractor at a nearby parameter. Furthermore, the forecasting complexity of the selected periodic orbits increases with noise as they undergo a transition to "noise-induced chaos." For sufficiently high noise levels, our ability to distinguish chaos from noise depends on model-order selection when estimating forecasting complexity and also on the exact choice of discrete observables used to encode phase-space trajectories. [ABSTRACT FROM AUTHOR]

Mutual information rate is an extension of the notion of mutual information to pairs of stationary stochastic processes. This quantity is defined as the time-averaged mutual information between corresponding segments of a pair of stationary stochastic processes, as the segment length tends to infinity. The contribution of this paper is to establish fairly mild regularity conditions for the existence of mutual information rate between Gaussian processes, and to rigorously derive an explicit formula to express this quantity in terms of the power and cross-power spectral densities of these Gaussian processes.

We introduce an index based on information theory to quantify the stationarity of a stochastic process. The index compares on the one hand the information contained in the increment at the time scale τ of the process at time t with, on the other hand, the extra information in the variable at time t that is not present at time t−τ. By varying the scale τ, the index can explore a full range of scales. We thus obtain a multi-scale quantity that is not restricted to the first two moments of the density distribution, nor to the covariance, but that probes the complete dependences in the process. This index indeed provides a measure of the regularity of the process at a given scale. Not only is this index able to indicate whether a realization of the process is stationary, but its evolution across scales also indicates how rough and non-stationary it is. We show how the index behaves for various synthetic processes proposed to model fluid turbulence, as well as on experimental fluid turbulence measurements.

Copyright of Journal of Chongqing University of Posts & Telecommunications (Natural Science Edition) is the property of Chongqing University of Posts & Telecommunications and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

We introduce an ambidextrous view of stochastic dynamical systems, comparing their forward-time and reverse-time representations and then integrating them into a single time-symmetric representation. The perspective is useful theoretically, computationally, and conceptually. Mathematically, we prove that the excess entropy—a familiar measure of organization in complex systems—is the mutual information not only between the past and future, but also between the predictive and retrodictive causal states. Practically, we exploit the connection between prediction and retrodiction to directly calculate the excess entropy. Conceptually, these lead one to discover new system measures for stochastic dynamical systems: crypticity (information accessibility) and causal irreversibility. Ultimately, we introduce a time-symmetric representation that unifies all of these quantities, compressing the two directional representations into one. The resulting compression offers a new conception of the amount of information stored in the present.

We study random string-duplication systems, which we call Pólya string models. These are motivated by a class of mutations that are common in most organisms and lead to an abundance of repeated sequences in their genomes. Unlike previous works that study the combinatorial capacity of string-duplication systems, or in a probabilistic setting, various string statistics, this work provides the exact entropy rate or bounds on it, for several probabilistic models. The entropy rate determines the compressibility of the resulting sequences, as well as quantifying the amount of sequence diversity that these mutations can create. In particular, we study the entropy rate of noisy string-duplication systems, including the tandem-duplication, end-duplication, and interspersed-duplication systems, where in all cases we study duplication of length 1 only. Interesting connections are drawn between some systems and the signature of random permutations, as well as to the beta distribution common in population genetics. [ABSTRACT FROM AUTHOR]

The analyticity of the entropy and relative entropy rates of continuous-state hidden Markov models is studied here. Using the analytic continuation principle and the stability properties of the optimal filter, the analyticity of these rates is established for analytically parameterized models. The obtained results hold under relatively mild conditions and cover several useful classes of hidden Markov models. These results are relevant for several theoretically and practically important problems arising in statistical inference, system identification and information theory. [ABSTRACT FROM AUTHOR]

Closed-form expressions for generalized entropy rates of Markov chains are obtained through pertinent averaging. First, the rates are expressed in terms of Perron-Frobenius eigenvalues of perturbations of the transition matrices. This leads to a classification of generalized entropy functionals into five exclusive types. Then, a weighted expression is obtained in which the associated Perron-Frobenius eigenvectors play the same role as the stationary distribution in the well-known weighted expression of Shannon entropy rate. Finally, all terms are shown to bear a meaning in terms of dynamics of an auxiliary absorbing Markov chain through the notion of quasi-limit distribution. Illustration of important properties of the involved spectral elements is provided through application to binary Markov chains. [ABSTRACT FROM AUTHOR]

The challenges from Distributed Denial-of-Service (DDoS) attacks are severe and still increasing significantly. We observe that the existing entropy-based methods only consider the probability distribution of traffic flows that have high false negative rates. On the other hand, sophisticated attack strategies, increasing attack strength and dynamic nature of network traffic patterns make it more difficult to detect the DDoS attacks with high accuracy. In this letter, we present an accurate approach, entropy rate measurement (ERM), to detect DDoS attacks. The proposed approach is based on the differences between the probability distributions and the number of flows. Both theoretical proofs and the results of experiments using real datasets demonstrate that our method has high detection accuracy rate compared to the existing measurements. [ABSTRACT FROM AUTHOR]