Search

For many decades now, Bayesian Model Averaging (BMA) has been a popular framework to systematically account for model uncertainty that arises in situations when multiple competing models are available to describe the same or similar physical process. The implementation of this framework, however, comes with a multitude of practical challenges including posterior approximation via Markov chain Monte Carlo and numerical integration. We present a Variational Bayesian Inference approach to BMA as a viable alternative to the standard solutions which avoids many of the aforementioned pitfalls. The proposed method is "black box" in the sense that it can be readily applied to many models with little to no model-specific derivation. We illustrate the utility of our variational approach on a suite of examples and discuss all the necessary implementation details. Fully documented Python code with all the examples is provided as well. [ABSTRACT FROM AUTHOR]

We review the field of data assimilation (DA) from a Bayesian perspective and show that, in addition to its by now common application to state estimation, DA may be used for model selection. An important special case of the latter is the discrimination between a factual model–which corresponds, to the best of the modeller's knowledge, to the situation in the actual world in which a sequence of events has occurred–and a counterfactual model, in which a particular forcing or process might be absent or just quantitatively different from the actual world. Three different ensemble-DA methods are reviewed for this purpose: the ensemble Kalman filter (EnKF), the ensemble four-dimensional variational smoother (En-4D-Var), and the iterative ensemble Kalman smoother (IEnKS). An original contextual formulation of model evidence (CME) is introduced. It is shown how to apply these three methods to compute CME, using the approximated time-dependent probability distribution functions (pdfs) each of them provide in the process of state estimation. The theoretical formulae so derived are applied to two simplified nonlinear and chaotic models: (i) the Lorenz three-variable convection model (L63), and (ii) the Lorenz 40-variable midlatitude atmospheric dynamics model (L95). The numerical results of these three DA-based methods and those of an integration based on importance sampling are compared. It is found that better CME estimates are obtained by using DA, and the IEnKS method appears to be best among the DA methods. Differences among the performance of the three DA-based methods are discussed as a function of model properties. Finally, the methodology is implemented for parameter estimation and for event attribution.

In generative modeling of neuroimaging data, such as dynamic causal modeling (DCM), one typically considers several alternative models, either to determine the most plausible explanation for observed data (Bayesian model selection) or to account for model uncertainty (Bayesian model averaging). Both procedures rest on estimates of the model evidence, a principled trade-off between model accuracy and complexity. In the context of DCM, the log evidence is usually approximated using variational Bayes. Although this approach is highly efficient, it makes distributional assumptions and is vulnerable to local extrema. This paper introduces the use of thermodynamic integration (TI) for Bayesian model selection and averaging in the context of DCM. TI is based on Markov chain Monte Carlo sampling which is asymptotically exact but orders of magnitude slower than variational Bayes. In this paper, we explain the theoretical foundations of TI, covering key concepts such as the free energy and its origins in statistical physics. Our aim is to convey an in-depth understanding of the method starting from its historical origin in statistical physics. In addition, we demonstrate the practical application of TI via a series of examples which serve to guide the user in applying this method. Furthermore, these examples demonstrate that, given an efficient implementation and hardware capable of parallel processing, the challenge of high computational demand can be overcome successfully. The TI implementation presented in this paper is freely available as part of the open source software TAPAS. [ABSTRACT FROM AUTHOR]

The paper obtains analytical results for the asymptotic properties of Model Selection Criteria – widely used in practice – for a general family of hidden Markov models (HMMs), thereby substantially extending the related theory beyond typical 'i.i.d.-like' model structures and filling in an important gap in the relevant literature. In particular, we look at the Bayesian and Akaike Information Criteria (BIC and AIC) and the model evidence. In the setting of nested classes of models, we prove that BIC and the evidence are strongly consistent for HMMs (under regularity conditions), whereas AIC is not weakly consistent. Numerical experiments support our theoretical results. [ABSTRACT FROM AUTHOR]

This article develops a new estimator of the marginal likelihood that requires only a sample of the posterior distribution as the input from the analyst. This sample may come from any sampling scheme, such as Gibbs sampling or Metropolis–Hastings sampling. The presented approach can be implemented generically in almost any application of Bayesian modeling and significantly decreases the computational burdens associated with marginal likelihood estimation compared to existing techniques. The functionality of this method is demonstrated in the context of probit and logit regressions, on two mixtures of normals models, and also on a high-dimensional random intercept probit. Simulation results show that the simple approach presented here achieves excellent stability in low-dimensional models, and also clearly outperforms existing methods when the number of coefficients in the model increases. [ABSTRACT FROM AUTHOR]

Selecting between competing statistical models is a challenging problem especially when the competing models are non-nested. In this paper we offer a simple solution by devising an algorithm which combines MCMC and importance sampling to obtain computationally efficient estimates of the marginal likelihood which can then be used to compare the models. The algorithm is successfully applied to a longitudinal epidemic data set, where calculating the marginal likelihood is made more challenging by the presence of large amounts of missing data. In this context, our importance sampling approach is shown to outperform existing methods for computing the marginal likelihood. [ABSTRACT FROM AUTHOR]

Neuroimaging increasingly exploits machine learning techniques in an attempt to achieve clinically relevant single-subject predictions. An alternative to machine learning, which tries to establish predictive links between features of the observed data and clinical variables, is the deployment of computational models for inferring on the (patho)physiological and cognitive mechanisms that generate behavioural and neuroimaging responses. This paper discusses the rationale behind a computational approach to neuroimaging-based single-subject inference, focusing on its potential for characterising disease mechanisms in individual subjects and mapping these characterisations to clinical predictions. Following an overview of two main approaches – Bayesian model selection and generative embedding – which can link computational models to individual predictions, we review how these methods accommodate heterogeneity in psychiatric and neurological spectrum disorders, help avoid erroneous interpretations of neuroimaging data, and establish a link between a mechanistic, model-based approach and the statistical perspectives afforded by machine learning. [ABSTRACT FROM AUTHOR]

A key quantity in the analysis of structured populations is the parameter K, which describes the number of subpopulations that make up the total population. Inference of K ideally proceeds via the model evidence, which is equivalent to the likelihood of the model. However, the evidence in favor of a particular value of K cannot usually be computed exactly, and instead programs such as Structure make use of heuristic estimators to approximate this quantity. We show—using simulated data sets small enough that the true evidence can be computed exactly—that these heuristics often fail to estimate the true evidence and that this can lead to incorrect conclusions about K. Our proposed solution is to use thermodynamic integration (TI) to estimate the model evidence. After outlining the TI methodology we demonstrate the effectiveness of this approach, using a range of simulated data sets. We find that TI can be used to obtain estimates of the model evidence that are more accurate and precise than those based on heuristics. Furthermore, estimates of K based on these values are found to be more reliable than those based on a suite of model comparison statistics. Finally, we test our solution in a reanalysis of a white-footed mouse data set. The TI methodology is implemented for models both with and without admixture in the software MavericK1.0. [ABSTRACT FROM AUTHOR]

Approximation of the model evidence is well known to be challenging. One promising approach is based on thermodynamic integration, but a key concern is that the thermodynamic integral can suffer from high variability in many applications. This article considers the reduction of variance that can be achieved by exploiting control variates in this setting. Our methodology applies whenever the gradient of both the log-likelihood and the log-prior with respect to the parameters can be efficiently evaluated. Results obtained on regression models and popular benchmark datasets demonstrate a significant and sometimes dramatic reduction in estimator variance and provide insight into the wider applicability of control variates to evidence estimation. Supplementary materials for this article are available online. [ABSTRACT FROM PUBLISHER]

Background Dynamic causal modeling (DCM) for fMRI is an established method for Bayesian system identification and inference on effective brain connectivity. DCM relies on a biophysical model that links hidden neuronal activity to measurable BOLD signals. Currently, biophysical simulations from DCM constitute a serious computational hindrance. Here, we present Massively Parallel Dynamic Causal Modeling ( mpdcm) , a toolbox designed to address this bottleneck. New method mpdcm delegates the generation of simulations from DCM's biophysical model to graphical processing units (GPUs). Simulations are generated in parallel by implementing a low storage explicit Runge–Kutta's scheme on a GPU architecture. mpdcm is publicly available under the GPLv3 license. Results We found that mpdcm efficiently generates large number of simulations without compromising their accuracy. As applications of mpdcm , we suggest two computationally expensive sampling algorithms: thermodynamic integration and parallel tempering. Comparison with existing method(s) mpdcm is up to two orders of magnitude more efficient than the standard implementation in the software package SPM. Parallel tempering increases the mixing properties of the traditional Metropolis–Hastings algorithm at low computational cost given efficient, parallel simulations of a model. Conclusions Future applications of DCM will likely require increasingly large computational resources, for example, when the likelihood landscape of a model is multimodal, or when implementing sampling methods for multi-subject analysis. Due to the wide availability of GPUs, algorithmic advances can be readily available in the absence of access to large computer grids, or when there is a lack of expertise to implement algorithms in such grids. [ABSTRACT FROM AUTHOR]

Abstract: Lohmann et al. (this issue) make three unremarkable observations about model selection and use them to critique dynamic causal modelling—a Bayesian model selection procedure based on causal models of dynamical systems (Marreiros et al., 2010). In this response, we unpack their misconceptions and try to answer their questions. [Copyright &y& Elsevier]

Abstract: To construct an adequate regression model one has to fulfill the set of measured features with their generated derivatives. Often the number of these features exceeds the number of the samples in the data set. After a feature generation process the problem of feature selection from a set of highly correlated features arises. The proposed algorithm uses an evidence maximization procedure to select a model as a subset of generated features. During the selection process it rejects multicollinear features. A problem of European option volatility modeling illustrates the algorithm. Its performance is compared with the performances of similar well-known algorithms. [Copyright &y& Elsevier]

For nearly any challenging scientific problem evaluation of the likelihood is problematic if not impossible. Approximate Bayesian computation (ABC) allows us to employ the whole Bayesian formalism to problems where we can use simulations from a model, but cannot evaluate the likelihood directly. When summary statistics of real and simulated data are compared-rather than the data directly-information is lost, unless the summary statistics are sufficient. Sufficient statistics are, however, not common but without them statistical inference in ABC inferences are to be considered with caution. Previously other authors have attempted to combine different statistics in order to construct (approximately) sufficient statistics using search and information heuristics. Here we employ an information-theoretical framework that can be used to construct appropriate (approximately sufficient) statistics by combining different statistics until the loss of information is minimized. We start from a potentially large number of different statistics and choose the smallest set that captures (nearly) the same information as the complete set. We then demonstrate that such sets of statistics can be constructed for both parameter estimation and model selection problems, and we apply our approach to a range of illustrative and real-world model selection problems. [ABSTRACT FROM AUTHOR]

Abstract: Twenty years ago, the discovery of the blood oxygen level dependent (BOLD) contrast and invention of functional magnetic resonance imaging (MRI) not only allowed for enhanced analyses of regional brain activity, but also laid the foundation for novel approaches to studying effective connectivity, which is essential for mechanistically interpretable accounts of neuronal systems. Dynamic causal modeling (DCM) and Granger causality (G-causality) modeling have since become the most frequently used techniques for inferring effective connectivity from fMRI data. In this paper, we provide a short historical overview of these approaches, describing milestones of their development from our subjective perspectives. [Copyright &y& Elsevier]

Abstract: Dynamic causal modelling (DCM) was originally proposed as a hypothesis driven procedure in which a small number of neurobiologically motivated models are compared. Model comparison in this context usually proceeds by individually fitting each model to data and then approximating the corresponding model evidence with a free energy bound. However, a recent trend has emerged for comparing very large numbers of models in a more exploratory manner. This led to propose a post-hoc approximation to the model evidence, which is computed by optimising only the largest (full) model of a set of models. The evidence for any (reduced) submodel is then obtained using a generalisation of the Savage-Dickey density ratio (). The benefit of this post-hoc approach is a huge reduction in the computational time required for model fitting. This is because only a single model is fitted to data, allowing a potentially huge model space to be searched relatively quickly. In this paper, we explore the relationship between the free energy bound and post-hoc approximations to the model evidence in the context of deterministic (bilinear) dynamic causal models (DCMs) for functional magnetic resonance imaging data. [Copyright &y& Elsevier]

Abstract: Dynamic causal modeling (DCM) is a generic Bayesian framework for inferring hidden neuronal states from measurements of brain activity. It provides posterior estimates of neurobiologically interpretable quantities such as the effective strength of synaptic connections among neuronal populations and their context-dependent modulation. DCM is increasingly used in the analysis of a wide range of neuroimaging and electrophysiological data. Given the relative complexity of DCM, compared to conventional analysis techniques, a good knowledge of its theoretical foundations is needed to avoid pitfalls in its application and interpretation of results. By providing good practice recommendations for DCM, in the form of ten simple rules, we hope that this article serves as a helpful tutorial for the growing community of DCM users. [Copyright &y& Elsevier]

Abstract: Functional integration in the brain rests on anatomical connectivity (the presence of axonal connections) and effective connectivity (the causal influences mediated by these connections). The deployment of anatomical connections provides important constraints on effective connectivity, but does not fully determine it, because synaptic connections can be expressed functionally in a dynamic and context-dependent fashion. Although it is generally assumed that anatomical connectivity data is important to guide the construction of neurobiologically realistic models of effective connectivity; the degree to which these models actually profit from anatomical constraints has not yet been formally investigated. Here, we use diffusion weighted imaging and probabilistic tractography to specify anatomically informed priors for dynamic causal models (DCMs) of fMRI data. We constructed 64 alternative DCMs, which embodied different mappings between the probability of an anatomical connection and the prior variance of the corresponding of effective connectivity, and fitted them to empirical fMRI data from 12 healthy subjects. Using Bayesian model selection, we show that the best model is one in which anatomical probability increases the prior variance of effective connectivity parameters in a nonlinear and monotonic (sigmoidal) fashion. This means that the higher the likelihood that a given connection exists anatomically, the larger one should set the prior variance of the corresponding coupling parameter; hence making it easier for the parameter to deviate from zero and represent a strong effective connection. To our knowledge, this study provides the first formal evidence that probabilistic knowledge of anatomical connectivity can improve models of functional integration. [Copyright &y& Elsevier]

Abstract: Bayesian model selection (BMS) is a powerful method for determining the most likely among a set of competing hypotheses about the mechanisms that generated observed data. BMS has recently found widespread application in neuroimaging, particularly in the context of dynamic causal modelling (DCM). However, so far, combining BMS results from several subjects has relied on simple (fixed effects) metrics, e.g. the group Bayes factor (GBF), that do not account for group heterogeneity or outliers. In this paper, we compare the GBF with two random effects methods for BMS at the between-subject or group level. These methods provide inference on model-space using a classical and Bayesian perspective respectively. First, a classical (frequentist) approach uses the log model evidence as a subject-specific summary statistic. This enables one to use analysis of variance to test for differences in log-evidences over models, relative to inter-subject differences. We then consider the same problem in Bayesian terms and describe a novel hierarchical model, which is optimised to furnish a probability density on the models themselves. This new variational Bayes method rests on treating the model as a random variable and estimating the parameters of a Dirichlet distribution which describes the probabilities for all models considered. These probabilities then define a multinomial distribution over model space, allowing one to compute how likely it is that a specific model generated the data of a randomly chosen subject as well as the exceedance probability of one model being more likely than any other model. Using empirical and synthetic data, we show that optimising a conditional density of the model probabilities, given the log-evidences for each model over subjects, is more informative and appropriate than both the GBF and frequentist tests of the log-evidences. In particular, we found that the hierarchical Bayesian approach is considerably more robust than either of the other approaches in the presence of outliers. We expect that this new random effects method will prove useful for a wide range of group studies, not only in the context of DCM, but also for other modelling endeavours, e.g. comparing different source reconstruction methods for EEG/MEG or selecting among competing computational models of learning and decision-making. [Copyright &y& Elsevier]

Recently, the issue of parameter identifiability has been highlighted in Bayesian inference of building energy models. Parameter identifiability addresses the correspondence between the observed data and the model parameters, and this analysis verifies whether the model parameters can be estimated using observed data. The authors propose a model selection process for Bayesian inference involving the unidentifiable parameters based on the model evidence. The model evidence is a component of the Bayes theorem, which corresponds to the observation probability (i.e., the goodness of fit) of the observed data for a given model. In this study, nested sampling was used to estimate the model evidence. A case study with the reference office building developed by the US Department of Energy shows the model selection process that uses the model evidence as the evaluation index for the unidentifiable parameter. For comparison with the existing approach of model evaluation, the authors present the results of a comparative analysis between the proposed process and that based on model prediction error (CV(RMSE)). As a result, it was observed that the higher the model evidence in the hypothesis for the unidentifiable parameters, the more the posteriors tend to be similar to the true values. In addition, no significant relationship was observed between the model prediction error and the accuracy of the posterior inference. This indicates that the model evidence can be used as an objective evaluation index for selecting the optimal hypothesis of unidentifiable parameters in the Bayesian inference of building energy models. [ABSTRACT FROM AUTHOR]

We show a link between Bayesian inference and information theory that is useful for model selection, assessment of information entropy and experimental design. We align Bayesian model evidence (BME) with relative entropy and cross entropy in order to simplify computations using prior-based (Monte Carlo) or posterior-based (Markov chain Monte Carlo) BME estimates. On the one hand, we demonstrate how Bayesian model selection can profit from information theory to estimate BME values via posterior-based techniques. Hence, we use various assumptions including relations to several information criteria. On the other hand, we demonstrate how relative entropy can profit from BME to assess information entropy during Bayesian updating and to assess utility in Bayesian experimental design. Specifically, we emphasize that relative entropy can be computed avoiding unnecessary multidimensional integration from both prior and posterior-based sampling techniques. Prior-based computation does not require any assumptions, however posterior-based estimates require at least one assumption. We illustrate the performance of the discussed estimates of BME, information entropy and experiment utility using a transparent, non-linear example. The multivariate Gaussian posterior estimate includes least assumptions and shows the best performance for BME estimation, information entropy and experiment utility from posterior-based sampling. [ABSTRACT FROM AUTHOR]