Search

Methods for summarizing case influence in Bayesian models take essentially two forms: (1) use common divergence measures for calculating distances between the full-data posterior and the case-deleted posterior, and (2) measure the impact of infinitesimal perturbations to the likelihood to study local case influence. Methods based on approach (1) lead naturally to considering the behavior of case-deletion importance sampling weights (the weights used to approximate samples from the case-deleted posterior using samples from the full posterior). Methods based on approach (2) lead naturally to considering the local curvature of the Kullback–Leibler divergence of the full posterior from a geometrically perturbed quasi-posterior. By examining the connections between the two approaches, we establish a rationale for employing low-dimensional summaries of case influence obtained entirely via the variance–covariance matrix of the log importance sampling weights. We illustrate the use of the proposed diagnos...

In this paper, we highlight properties of Bayesian models in which the prior puts positive mass on all Kullback–Leibler neighborhoods of all densities. These properties are concerned with model choice via the Bayes factor, density estimation and the maximization of expected utility for decision problems. In four illustrations we focus on the Bayes factor and show that whatever models are being compared, the [log(Bayes factor)]/[sample size] converges to a non-random number which has a nice interpretation. A parametric versus semiparametric model comparison provides a fifth illustration.

Minimum discrimination information provides a useful generalization of likelihood methodology for classification and clustering of multivariate time series. Discrimination between different classes of multivariate time series that can be characterized by differing covariance or spectral structures is of importance in applications occurring in the analysis of geophysical and medical time series data. For discrimination between such multivariate series, Kullback-Leibler discrimination information and the Chernoff information measure are developed for the multivariate non-Gaussian case. Asymptotic error rates and limiting distributions are given for a generalized spectral disparity measure that includes the foregoing criteria as special cases. Applications to problems of clustering and classifying earthquakes and mining explosions are given.

Using both frequentist and Bayesian techniques, predicting densities are derived for future observations from a multivariate linear model with matrix normal error terms. All the candidates belong to a general location-scale family of predicting densities. An analytic comparison is undertaken, using a Kullback—Leibler loss, by citing an optimal member of a subclass including most of these predicting densities as members. The subclass is based on an invariant Student-t random matrix, and the optimal member is the Bayesian predictive density corresponding to a Jeffreys noninformative prior. Information-based numerical comparisons illustrate the nature of the dominance.

Here we use posterior densities based on relative entropy reference priors for two purposes. The first purpose is to identify data implicit in the use of informative priors. We represent an informative prior as the posterior from an experiment with a known likelihood and a reference prior. Minimizing the relative entropy distance between this posterior and the informative prior over choices of data results in a data set that can be regarded as representative of the information in the informative prior. The second implication from reference priors is obtained by replacing the informative prior with a class of densities from which one might wish to make inferences. For each density in this class, one can obtain a data set that minimizes a relative entropy. The maximum of these sample sizes as the inferential density varies over its class can be used as a guess as to how much data is required for the desired inferences. We bound this sample size above and below by other techniques that permit it to ...

In maximum entropy (ME) modeling, the information discrepancy between two distributions is measured in terms of their entropy difference. In discrimination information statistics the information discrepancy between two distributions is measured in terms of the Kullback-Leibler function (i.e., relative entropy or cross-entropy). This article presents an equivalence between Kullback–Leibler functions and entropy differences involving an ME distribution. Based on this equivalence, the concept of information discrimination (ID) distinguishability is introduced as a unifying framework for the two methods of measuring information discrepancy between distributions. Applications of ID distinguishability as diagnostics for examining robustness of parametric procedures and sensitivity of nonparametric statistics across parametric families of distributions is proposed. The equivalence result facilitates estimation of Kullback-Leibler functions in terms of entropy estimates. Application of the ID distinguish...

Viewing the analysis of designed experiments as a model selection problem, we introduce the use of a predictive Bayesian criterion in this context based on the predictive density of a replicate experiment (PDRE). A calibration of the criterion is provided to assist in the model choice. The relationships of the proposed criterion to other prevalent criteria, such as AIC, BIC, and Mallows's C p , are given. An information theoretic criterion based on the PDRE's of two competing models is also introduced and compared with the usual F statistic for two nested models. Examples are given to illustrate the proposed methodology.

The conditional logit model is a multinomial logit model that permits the inclusion of choice-specific attributes. This article shows that the conditional logit model will maximize entropy given a set of attribute-value preserving constraints. A correspondence between the maximum entropy (ME) and maximum likelihood (ML) estimates for logit probabilities is established. Some easily computable and useful diagnostics for logit analysis are provided, and it is shown that an evaluation of the relative importance of attributes can be made using the ME formulation. The ME formulation is also generalized to accommodate initial choice probabilities into the logit model. An example is given. KEY WORDS: Choice models; Entropy; Kullback-Leibler discrimination information function; Relative importance.

This article develops a general method for assessing the influence of model assumptions in a Bayesian analysis. We assume that model choices are indexed by a hyperparameter with some given initial choice. We use the term “model” to encompass both the sampling model and the prior distribution. We wish to assess the effect of changing the hyperparameter away from the initial choice. We are performing a sensitivity analysis, with the hyperparameter defining our perturbations. We use the Kullback—Leibler divergence to measure the difference between posteriors corresponding to different choices of the hyperparameter. We also measure the change in priors. If small changes in the priors lead to large changes in posteriors, the choice of hyperparameter is influential. The second-order difference in the Kullback—Leibler divergence is expressed by Fisher information matrices. The relative change in posteriors compared with priors may be summarized by the relative eigenvalue of the posterior and prior Fishe...

An explicitly computable, necessary, and sufficient condition for the existence of an adaptive classification procedure is obtained. By definition, an adaptive procedure, which classifies a sample as coming from one of alternative distributions known only up to a finite-valued nuisance parameter, is required to have the same asymptotic behavior of error probability for these families as asymptotically optimal rules for each of the families. We investigate the conditions under which the overall maximum likelihood procedure is adaptive and derive a rule that is adaptive if any procedure is. We study the consistency of these procedures. Several exponential-family examples illustrate their form, and a small-sample study of error probabilities is performed.

There has been a lot of work on measures of dependence or association for bivariate probability distributions or bivariate data. These measures usually assume that the variables are both continuous or both categorical. In comparison, there is very little work on multivariate or conditional measures of dependence. The purpose of this article is to discuss measures of multivariate dependence and measures of conditional dependence based on relative entropies. These measures are conceptually very general, as they can be used for a set of variables that can be a mixture of continuous, ordinal-categorical, and nominal-categorical variables. For continuous or ordinal-categorical variables, a certain transformation of relative entropy to the interval [0, 1] leads to generalizations of the correlation, multiple-correlation, and partial-correlation coefficients. If all variables are nominal categorical, the relative entropies are standardized to take a maximum of 1 and then transformed so that in the bivar...

A prediction density function g* for the normal linear model is derived. This function is shown to dominate three well-known prediction densities by first constructing a specified class of densities that includes these three and then proving that g* is the optimal member of this class in the sense of minimizing a criterion based on the Kullback—Leibler divergence. g* coincides with a Bayesian prediction density assuming diffuse prior.