Search

The stochastic error distance (SED) introduced by Diebold and Shin () ranks forecast models by divergence between distributions of the errors of the actual and perfect forecast models. The basic SED is defined by the variation distance and provides a representation of the mean absolute error, but by basing ranking on the entire error distribution and divergence, the SED moves beyond the traditional forecast evaluations. First, we establish connections between ranking forecast models by the SED, error entropy and some partial orderings of distributions. Then, we introduce the notion of excess error for forecast errors of magnitudes larger than a tolerance threshold and give the SED representation of the mean excess error (MEE). As a function of the threshold, the MEE is a local risk measure. With the distribution of the absolute error as a prior for the threshold, its Bayes risk is the entropy functional of the survival function, which is a known measure in the information theory and reliability. Notions and results are illustrated using various distributions for the error. The empirical versions of SED, MEE and its Bayes risk are compared with the mean squared error in ranking regression and autoregressive integrated moving average models for forecasting bond risk premia.

This paper introduces an approach to carry the uncertainty about the wind speed through the optimization for the economic dispatching. This approach uses simulated wind speed data points for inputs to economic dispatch models and produces data for estimating the probability distributions of optimal fossil fuel generation outputs, transmission loss, and total cost of power generation. Large samples of simulated data allow statistical analyses of the outputs such as the mean, median, percentiles, and confidence intervals for each output variable, plots of distributions of all output variables, and correlation measures for examining relationships between pairs of the output variables. Through the proposed algorithm, a generation expansion planner can perform useful analyses without executing the algorithm repeatedly. The algorithm and its results are illustrated through an example of an economic dispatch model and a model for the distribution of the wind speed based on a forecast for the planning target time.

Kernel estimates of entropy and mutual information have been studied extensively in statistics and econometrics. Kullback–Leibler divergence has been used in the kernel estimation literature; yet the information characteristic of kernel estimation remains unexplored. We explore kernel estimation as an information transmission operation where the empirical cumulative distribution function is transformed into a smooth estimate. The smooth kernel estimate is a mixture of kernel functions. The Jensen–Shannon (JS) divergence of the mixture distribution provides the information measure of kernel estimation. This measure admits Kullback–Leibler and mutual information representations and provides a lower bound for the entropy of the kernel estimate of the distribution in terms of the Shannon entropy of the kernel function and the bandwidth. The JS divergence provides guidance for kernel choice based on information-theoretic considerations which helps resolve a conundrum, namely that it is legitimate and d...

Research summary: Shareholder activism has become more widespread, yet the role of corporate governance as antecedent to shareholder activism remains equivocal. We propose a new conceptual model that characterizes the stochastic of observable shareholder activism as a compound product of two latent components representing (1) shareholder activists' propensity to target a company and (2) executives' propensity to settle activists' demands privately. Our model explicitly decouples corporate governance expectations for the two latent components embedded in activism process, and thus allows us to relax assumptions of homogenous shareholder interests and constrained managerial discretion where corporate managers are expected to negotiate privately and settle only value-creating activist demands. Bayesian analysis of zero-inflated Poisson regression reveals that corporate governance relationships with activism vary across shareholder demands and private settlements. Managerial summary: Increasing shareholder activism has generated debates as to whether activism promotes managerial accountability and responsibility or instead encourages managerial short-termism. Our research model allows for heterogeneous interests among a company's shareholders. We theorize and empirically investigate a broader role of corporate governance: governance mechanisms need to ensure that executives are not (1) ignoring activists' value-increasing demands or (2) accommodating activists' value-decreasing demands in a private, opaque manner that disenfranchises other shareholders. Our results indicate that corporate governance implications differ for visible shareholder demands in contrast with private activism. A plausible application of our model is that it provides estimates of the probability of the numbers of shareholder demands to be received by a firm and the probability of privately settling a demand. Copyright © 2015 John Wiley & Sons, Ltd.

The Bayes risk of the mean residual (MR) or excess of a random variable above a threshold is defined by the expectation with respect to a prior for the threshold. Recently the MR Bayes risk has been studied for the proportional hazards (PH) model. We provide results without assuming any stricture such as PH for the distribution, further results for the PH case and new reliability applications, and an empirical application. We show that the standard deviation provides a tight upper bound for the MR Bayes risk. Attainment of the bound characterizes the exponential distribution. The ratio of the MR Bayes risk to the standard deviation provides a risk index which reaches the maximum of one if and only if the underlying distribution is exponential. Distributional examples illustrate the tightness of the standard deviation bound and the improvement over the known bound. Another result broadens the notion of MR Bayes risk to functions of the threshold. We provide a result for ordering of PH models by MR Bayes risk. New reliability applications include the MR Bayes risk at the system level and distributions with new better (worse) than used in expectation property. An empirical example illustrates application of the MR and MR Bayes risk for the New York City’s taxi trip times.

This article introduces two new maximum entropy (ME) methods for modeling the distribution of time to an event. One method is within the classical ME framework and provides characterizations of change point models such as the piecewise exponential distribution. The second method uses the entropy of the equilibrium distribution (ED) for the objective function and provides new characterizations of the exponential, Weibull, Pareto, and uniform distributions. With the same moment constraints, the classical ME and the maximum ED entropy algorithms generate different models for the interarrival time. © 2014 Wiley Periodicals, Inc. Naval Research Logistics 61: 427–434, 2014

Which component is most important for a system's survival? We answer this question by ranking the information relationship between a system and its components. The mutual information (M) measures dependence between the operational states of the system and a component for a mission time as well as between their life lengths. This measure ranks each component in terms of its expected utility for predicting the system's survival. We explore some relationships between the ordering of importance of components by M and by Zellner's Maximal Data Information (MDIP) criterion. For many systems the bivariate distribution of the component and system lifetimes does not have a density with respect to the two-dimensional Lebesgue measure. For these systems, M is not defined, so we use a modification of a mutual information index to cover such situations. Our results for ordering dependence are general in terms of binary structures, sum of random variables, and order statistics.

The stochastic error distance (SED) introduced by Diebold & Shin (2017) ranks forecast models by divergence between distributions of the errors of the actual and prefect forecast models. The basic SED is defined by the variation distance and provides a representation of the mean absolute error, but by basing ranking on the entire error distribution and divergence, the SED moves beyond the traditional forecast evaluations. First, we establish connections between ranking forecast models by the SED, error entropy, and some partial orderings of distributions. Then we introduce the notion of excess error for forecast errors of magnitudes larger than a tolerance threshold and give the SED representation of the mean excess error (MEE). As a function of the threshold, the MEE is a local risk measure. With the distribution of the absolute error as a prior for the threshold, its Bayes risk is the entropy functional of the survival function, which is a known measure in the information theory and reliability. Notions and results are illustrated using various distributions for the error. The empirical versions of SED, MEE, and its Bayes risk are compared with the mean squared error in ranking regression and ARIMA models for forecasting bond risk premia.

A framework involving independent competing risks permits observing failures due to a specific cause and failures due to a competing cause, which constitute survival times from the cause of primary interest. Is observing more failures more informative than observing survivals? Intuitively, due to the definitiveness of failures, the answer seems to be the former. However, it has been shown before that this intuition holds when estimating the mean but not the failure rate of the exponential model with a gamma prior distribution for the failure rate. In this article, we address this question at a more general level. We show that for a certain class of distributions failures can be more informative than survivals for prediction of life length and vice versa for some others. We also show that for a large class of lifetime models, failure is less informative than survival for estimating the proportional hazards parameter with gamma, Jeffreys, and uniform priors. We further show that, for this class of lifetime models, on average, failure is more informative than survival for parameter estimation and for prediction. These results imply that the inferential purpose and properties of the lifetime distribution are germane for conducting life tests. © 2013 Wiley Periodicals, Inc. Naval Research Logistics, 2013

Operational processes are usually studied in terms of stochastic processes. The main information measure used for predictability of stochastic processes is the entropy rate, which is asymptotic conditional entropy, thus not suitable for application over a finite horizon. We use the conditional entropy to study the predictability of stochastic processes over the finite horizon. It is well-known that the conditional entropies of stationary processes decrease as the processes evolve, implying that, on average, their pasts become more informative about prediction of their future outcomes. Some important operational processes such as martingale, models for maintenance policies, nonhomogeneous Poisson, and mixed Poisson processes are nonstationary. We show that as a nonstationary process evolves, it may provide more information or less information about the future state of the system. We develop results for comparing the predictability of stochastic processes. © 2011 Wiley Periodicals, Inc. Naval Research Logistics, 2011

We explore properties of information measures of the Dirichlet family and related distributions. Representations of the information measures of the Dirichlet family in terms of the information measures of the gamma family reflect the characterization of Dirichlet distribution in terms of the ratios of independent gamma distributions to their sum. We present measures of information provided by a multinomial vector about the multinomial parameters under the Dirichlet prior and measures of information loss due to aggregation of Dirichlet components and the multinomial categories. New information characterizations of the bivariate Dirichlet distribution in terms of survival and hazard gradient constraints are given. The information analysis of contingency tables is briefly described. An example illustrates applications of Dirichlet prior for inferences about information measures of contingency tables and Kendall's tau for ordinal data. Results that compare solicitation of the mean and mode vectors for the Dirichlet model are given and are illustrated through an example from a study of business executives' environmental uncertainty in the aftermath of 9/11 terrorist attacks. The probability integral transformation is proposed for facilitating implementation of the maximum entropy Dirichlet process prior procedure for inference about the fit of continuous distributions to the data. Copyright © 2011 John Wiley & Sons, Ltd.

Information-theoretic methodologies are increasingly being used in various disciplines. Fre- quently an information measure is adapted for a problem, yet the perspective of information as the unifying notion is overlooked. We set forth this perspective through presenting information- theoretic methodologies for a set of problems in probability and statistics. Our focal measures are Shannon entropy and Kullback-Leibler information. The background topics for these mea- sures include notions of uncertainty and information, their axiomaticfoundation, interpretations, properties, and generalizations. Topics with broad methodological applications include discrep- ancy between distributions, derivation of probability models, dependence between variables, and Bayesian analysis. More specific methodological topics include model selection, limiting distributions, optimal prior distribution and design of experiment, modeling duration variables, order statistics, data disclosure, and relative importance of predictors. Illustrations range from very basic to highly technical ones that draw attention to subtle points.

This article develops information optimal models for the joint distribution based on partial information about the survival function or hazard gradient in terms of inequalities. In the class of all distributions that satisfy the partial information, the optimal model is characterized by well-known information criteria. General results relate these information criteria with the upper orthant and the hazard gradient orderings. Applications include information characterizations of the bivariate Farlie–Gumbel–Morgenstern, bivariate Gumbel, and bivariate generalized Gumbel, for which no other information characterization are available. The generalized bivariate Gumbel model is obtained from partial information about the survival function and hazard gradient in terms of marginal hazard rates. Other examples include dynamic information characterizations of the bivariate Lomax and generalized bivariate Gumbel models having marginals that are transformations of exponential such as Pareto, Weibull, and extreme value. Mixtures of bivariate Gumbel and generalized Gumbel are obtained from partial information given in terms of mixtures of the marginal hazard rates.

This article presents statistical methods for identifying outcomes in a given sample that can be inferred as plausible extreme and whether the extremes on two variables are associated. Applications to CEO pay and performance of 50 top-paid CEOs illustrate these methodologies. Thresholds between extremes and nonextremes are found using high probability intervals under the probability distributions that govern sampling variations of the sample extremes. A Bayesian approach is used to compute odds on the association between the extremes of the two variables. The extreme pay—performance analysis of 50 top-paid CEOs reveals astonishing odds in favor of a company being extreme high only on one of the two versus on both variables. The result is considered decisive evidence for a negative association between extreme on CEO pay and extreme on performance among such top-paid CEOs. By contrast, analysis of the nonextreme CEOs yielded no evidence of any association between CEO pay and performance.

This article defines a quantized entropy and develops Bayes estimates and inference for the entropy and a Kullback–Leibler information index of the model fit. We use a Dirichlet process prior for the unknown data-generating distribution with a maximum entropy candidate model as the expected distribution. This formulation produces prior and posterior distributions for the quantized entropy, the information index of fit, the moments, and the model parameters. The posterior mean of the quantized entropy provides a Bayes estimate of entropy under quadratic loss. The consistency of the Bayes estimates and the information index are shown. The implementation and the performances of the Bayes estimates are illustrated using data simulated from exponential, gamma, and lognormal distributions.

An information framework is proposed for the systematic application, unification, and generalizations of the measures used in the uncertainty and disagreement of economic forecasters literature. The framework uses the mixture model of density forecasts and applies an uncertainty function and an information divergence to individual forecast distributions generated in a random environment and to their mixture distribution. The focal measure is the Jensen-Shannon divergence of the mixture which admits Kullback-Leibler and mutual information representations. Illustrations include exploring the dynamics of the individual and aggregate uncertainty about the US inflation rate using the Survey of Professional Forecasters (SPF). Each quarter the SPF solicits forecast distributions on a set of intervals and the survey design has been changed over time. We show that the normalized entropy index corrects the distortion caused by the changes made in the number of intervals. Bayesian hierarchical models are used to examine the association of the inflation uncertainty with the anticipated inflation and the dispersion of point forecasts. Implementation of the information framework based on variance and Dirichlet model for capturing uncertainty about the probability distribution of the economic variable are briefly discussed.

The objective of this paper is to integrate the generalized gamma (GG) distribution into the information theoretic literature. We study information properties of the GG distribution and provide an assortment of information measures for the GG family, which includes the exponential, gamma, Weibull, and generalized normal distributions as its subfamilies. The measures include entropy representations of the log-likelihood ratio, AIC, and BIC, discriminating information between GG and its subfamilies, a minimum discriminating information function, power transformation information, and a maximum entropy index of fit to histogram. We provide the full parametric Bayesian inference for the discrimination information measures. We also provide Bayesian inference for the fit of GG model to histogram, using a semi-parametric Bayesian procedure, referred to as the maximum entropy Dirichlet (MED). The GG information measures are computed for duration of unemployment and duration of CEO tenure.

In this paper, we introduce the minimum dynamic discrimination information (MDDI) approach to probability modeling. The MDDI model relative to a given distribution G is that which has least Kullback-Leibler information discrepancy relative to G, among all distributions satisfying some information constraints given in terms of residual moment inequalities, residual moment growth inequalities, or hazard rate growth inequalities. Our results lead to MDDI characterizations of many well-known lifetime models and to the development of some new models. Dynamic information constraints that characterize these models are tabulated. A result for characterizing distributions based on dynamic Rényi information divergence is also given.

In many reliability and survival analysis problems the current age of an item under study must be taken into account by information measures of the lifetime distribution. Kullback–Leibler information and Shannon entropy have been considered in this context, which led to information measures that depend on time, and thus are dynamic. This paper develops dynamic information divergence and entropy of order α , also known as Renyi information and entropy, which for α = 1 give the Kullback–Leibler information and Shannon entropy, respectively. We give characterizations of the proportional hazards model, the exponential distribution, and Generalized Pareto distributions in terms of dynamic Renyi information and entropy. It is also shown that dynamic Renyi entropy uniquely determines distributions that have monotone densities. A result that relates dynamic Renyi entropy and hazard rate orderings is given. This result leads to a Maximum Dynamic Entropy of order α formulation and characterizations of some well-known lifetime models. A dynamic entropy hazard rate inequality is developed as an analog of the well-known entropy moment inequality.

A formal approach to produce a model for the data-generating distribution based on partial knowledge is the well-known maximum entropy method. In this approach, partial knowledge about the data-generating distribution is formulated in terms of some information constraints and the model is obtained by maximizing the Shannon entropy under these constraints. Frequently, in reliability analysis the problem of interest is the lifetime beyond an age t. In such cases, the distribution of interest for computing uncertainty and information is the residual distribution. The information functions involving a residual life distribution depend on t, and hence are dynamic. The maximum dynamic entropy (MDE) model is the distribution with the density that maximizes the dynamic entropy for all t. We provide a result that relates the orderings of dynamic entropy and the hazard function for distributions with monotone densities. Applications include dynamic entropy ordering within some parametric families of distributions, orderings of distributions of lifetimes of systems and their components connected in series and parallel, record values, and formulation of constraints for the MDE model in terms of the evolution paths of the hazard function and mean residual lifetime function. In particular, we identify classes of distributions in which some well-known distributions, including the mixture of two exponential distributions and the mixture of two Pareto distributions, are the MDE models.

The unified framework of information theoretic statistics was established by Kullback (1959). Since then numerous information indices have been developed in various contexts. This paper represents many of these indices in a unified context. The unification thread is the discrimination information function: information indices are all logarithmic measures of discrepancy between two probability distributions. First, we present a summary of informational aspects of the basic information functions, a unification of various information-theoretic modeling approaches, and some explication in terms of traditional measures. We then tabulate a unified representation of assortments of information indices developed in the literature for maximum entropy modeling, covariate information, and influence diagnostics. The subjects of these indices include parametric model fitting, nonparametric entropy estimation, categorical data analysis, the linear and exponential family regression, and time series. The coverage however, is not exhaustive. The tabulation includes sampling theory and Bayesian indices, but the focus is on interpretation as descriptive measures and inferential properties are noted tangentially. Finally, applications of some information indices are illustrated through modeling duration data for Sprint's churned customer and choice of long distance provider.

(2000). Principal Information Theoretic Approaches. Journal of the American Statistical Association: Vol. 95, No. 452, pp. 1349-1353.

This paper examines the role of variance and entropy in ordering distributions and random prospects. There is no universal relation between entropy and variance orderings of distributions. But we place their relationship in the context of a stronger ordering relation known as dispersion ordering. Further, some conditions are identified under which variance and entropy order similarly when continuous variables are transformed. We also analyze parametric changes which do not disturb the agreement between these rankings. The results are conveniently tabulated in terms of distribution parameters.

In maximum entropy (ME) modeling, the information discrepancy between two distributions is measured in terms of their entropy difference. In discrimination information statistics the information discrepancy between two distributions is measured in terms of the Kullback-Leibler function (i.e., relative entropy or cross-entropy). This article presents an equivalence between Kullback–Leibler functions and entropy differences involving an ME distribution. Based on this equivalence, the concept of information discrimination (ID) distinguishability is introduced as a unifying framework for the two methods of measuring information discrepancy between distributions. Applications of ID distinguishability as diagnostics for examining robustness of parametric procedures and sensitivity of nonparametric statistics across parametric families of distributions is proposed. The equivalence result facilitates estimation of Kullback-Leibler functions in terms of entropy estimates. Application of the ID distinguish...

Generally, in the choice modeling literature, each individual decision-maker is assumed to be aware of a set of alternatives at the start of the decision process. This initial set defined as the awareness set usually includes a mixture of alternatives, some seriously and some casually under contemplation. The decision-making process leads to a partitioning of the awareness set into two disjoint subsets: consideration set and non-consideration set. The consideration set consists of the more seriously considered alternatives. This paper proposes an information-theoretic algorithm for estimating the consideration set. The proposed algorithm estimates the choice probabilities in the awareness sets according to the maximum entropy principle which by use of a multinomial logit model, computes an individual information index for each set, identifies the weak or unacceptable alternatives for each individual, and reduces each awareness set to a consideration set. The average of the selected alternative probabilities is proposed as a statistic by which the predicmtive quality of various consideration sets can be compared. We empirically illustrate the proposed method by estimating consideration sets for a sample of industrial decision-makers using a real world marketing data set. The analysis leads to an important finding; that the predictive power of the multinomial logit is in identifying the weak (non-considered) alternatives rather than predicting the chosen alternative among the choices in the consideration set. The proposed algorithm enables us to empirically implement choice set reduction for a given data set using an information criterion. Thus, the stage is set for advancing choice theory by investigating the impact of using additional data and/or other choice models on the derived consideration sets.