Search

Generalized linear latent variable models (GLLVMs) are a class of methods for analyzing multi-response data which has gained considerable popularity in recent years, e.g., in the analysis of multivariate abundance data in ecology. One of the main features of GLLVMs is their capacity to handle a variety of responses types, such as (overdispersed) counts, binomial and (semi-)continuous responses, and proportions data. On the other hand, the inclusion of unobserved latent variables poses a major computational challenge, as the resulting marginal likelihood function involves an intractable integral for non-normally distributed responses. This has spurred research into a number of approximation methods to overcome this integral, with a recent and particularly computationally scalable one being that of variational approximations (VA). However, research into the use of VA for GLLVMs has been hampered by the fact that fully closed-form variational lower bounds have only been obtained for certain combinations of response distributions and link functions. In this article, we propose an extended variational approximations (EVA) approach which widens the set of VA-applicable GLLVMs dramatically. EVA draws inspiration from the underlying idea behind the Laplace approximation: by replacing the complete-data likelihood function with its second order Taylor approximation about the mean of the variational distribution, we can obtain a fully closed-form approximation to the marginal likelihood of the GLLVM for any response type and link function. Through simulation studies and an application to a species community of testate amoebae, we demonstrate how EVA results in a “universal” approach to fitting GLLVMs, which remains competitive in terms of estimation and inferential performance relative to both standard VA (where any intractable integrals are either overcome through reparametrization or quadrature) and a Laplace approximation approach, while being computationally more scalable than both methods in practice.

We propose a unified framework for establishing existence of nonparametric $M$-estimators, computing the corresponding estimates, and proving their strong consistency when the class of functions is exceptionally rich. In particular, the framework addresses situations where the class of functions is complex involving information and assumptions about shape, pointwise bounds, location of modes, height at modes, location of level-sets, values of moments, size of subgradients, continuity, distance to a “prior” function, multivariate total positivity and any combination of the above. The class might be engineered to perform well in a specific setting even in the presence of little data. The framework views the class of functions as a subset of a particular metric space of upper semicontinuous functions under the Attouch–Wets distance. In addition to allowing a systematic treatment of numerous $M$-estimators, the framework yields consistency of plug-in estimators of modes of densities, maximizers of regression functions, level-sets of classifiers and related quantities, and also enables computation by means of approximating parametric classes. We establish consistency through a one-sided law of large numbers, here extended to sieves, that relaxes assumptions of uniform laws, while ensuring global approximations even under model misspecification.

The logistic regression model constitutes a natural and simple tool to understand how covariates (when available) contribute to explain the topology of a binary network. After estimating the logistic parameters, one of the main questions which arises in practice is to assess the goodness of fit of the corresponding model. To address this problem, we add a general term, related to the graphon function of W-graph models, to the logistic function. Such an extra term aims at characterizing the residual structure of the network, that is not explained by the covariates. We approximate this new generic logistic model using a class of models with blockwise constant residual structure. This framework allows to derive a Bayesian procedure from a model based selection context using goodness-of-fit criteria. All these criteria depend on marginal likelihood terms for which we do provide estimates relying on two series of variational approximations. Experiments on toy data are carried out to assess the inference procedure. Finally, six networks from social sciences and ecology are studied to illustrate the proposed methodology. Logistic regression is a natural and simple tool to understand how covariates contribute to explain the topology of a binary network. Once the model is fitted, the practitioner is interested in the goodness-of-fit of the regression in order to check if the covariates are sufficient to explain the whole topology of the network and, if they are not, to analyze the residual structure. To address this problem, we introduce a generic model that combines logistic regression with a network-oriented residual term. This residual term takes the form of the graphon function of a W-graph. Using a variational Bayes framework, we infer the residual graphon by averaging over a series of blockwise constant functions. This approach allows us to define a generic goodness-of-fit criterion, which corresponds to the posterior probability for the residual graphon to be constant. Experiments on toy data are carried out to assess the accuracy of the procedure. Several networks from social sciences and ecology are studied to illustrate the proposed methodology.

to appear; International audience; Probabilistic graphical models offer a powerful framework to account for the dependence structure between variables, which is represented as a graph. However, the dependence between variables may render inference tasks intractable. In this paper, we review techniques exploiting the graph structure for exact inference, borrowed from optimisation and computer science. They are built on the principle of variable elimination whose complexity is dictated in an intricate way by the order in which variables are eliminated. The so‐called treewidth of the graph characterises this algorithmic complexity: low‐treewidth graphs can be processed efficiently. The first point that we illustrate is therefore the idea that for inference in graphical models, the number of variables is not the limiting factor, and it is worth checking the width of several tree decompositions of the graph before resorting to the approximate method. We show how algorithms providing an upper bound of the treewidth can be exploited to derive a ‘good' elimination order enabling to realise exact inference. The second point is that when the treewidth is too large, algorithms for approximate inference linked to the principle of variable elimination, such as loopy belief propagation and variational approaches, can lead to accurate results while being much less time consuming than Monte‐Carlo approaches. We illustrate the techniques reviewed in this article on benchmarks of inference problems in genetic linkage analysis and computer vision, as well as on hidden variables restoration in coupled Hidden Markov Models.

Approximate Bayesian inference methods offer methodology for fitting Bayesian models as fast alternatives to Markov Chain Monte Carlo methods that sometimes have only a slight loss of accuracy. In this thesis, we consider variable selection for linear models, and zero inflated mixed models. Variable selection for linear regression models are ubiquitous in applied statistics. We use the popular g-prior (Zellner, 1986) for model selection of linear models with normal priors where g is a prior hyperparameter. We derive exact expressions for the model selection Bayes Factors in terms of special functions depending on the sample size, number of covariates and R-squared of the model. We show that these expressions are accurate, fast to evaluate, and numerically stable. An R package blma for doing Bayesian linear model averaging using these exact expressions has been released on GitHub. We extend the Particle EM method of (Rockova, 2017) using Particle Variational Approximation and the exact posterior marginal likelihood expressions to derive a computationally efficient algorithm for model selection on data sets with many covariates. Our algorithm performs well relative to existing algorithms, completing in 8 seconds on a model selection problem with a sample size of 600 and 7200 covariates. We consider zero-inflated models that have many applications in areas such as manufacturing and public health, but pose numerical issues when fitting them to data. We apply a variational approximation to zero-inflated Poisson mixed models with Gaussian distributed random effects using a combination of VB and the Gaussian Variational Approximation (GVA). We also incorporate a novel parameterisation of the covariance of the GVA using the Cholesky factor of the precision matrix, similar to Tan and Nott (2018) to resolve associated numerical difficulties.

Variational approximations are approximate inference techniques for complex statisticalmodels providing fast, deterministic alternatives to conventional methods that,however accurate, take much longer to run. We extend recent work concerning variationalapproximations developing and assessing some variational tools for likelihoodbased and Bayesian inference. In particular, the first part of this thesis employs a Gaussian variational approximation strategy to handle frequentist generalized linear mixedmodels with general design random effects matrices such as those including spline basisfunctions. This method involves approximation to the distributions of random effectsvectors, conditional on the responses, via a Gaussian density. The second thread isconcerned with a particular class of variational approximations, known as mean fieldvariational Bayes, which is based upon a nonparametric product density restriction on the approximating density. Algorithms for inference and fitting for models with elaborateresponses and structures are developed adopting the variational message passingperspective. The modularity of variational message passing is such that extensions tomodels with more involved likelihood structures and scalability to big datasets are relatively simple. We also derive algorithms for models containing higher level randomeffects and non-normal responses, which are streamlined in support of computationalefficiency. Numerical studies and illustrations are provided, including comparisons witha Markov chain Monte Carlo benchmark.

Mixture models are widely used in Bayesian statistics and machine learning, in particular in computational biology, natural language processing and many other fields. Variational inference, a technique for approximating intractable posteriors thanks to optimization algorithms, is extremely popular in practice when dealing with complex models such as mixtures. The contribution of this paper is two-fold. First, we study the concentration of variational approximations of posteriors, which is still an open problem for general mixtures, and we derive consistency and rates of convergence. We also tackle the problem of model selection for the number of components: we study the approach already used in practice, which consists in maximizing a numerical criterion (the Evidence Lower Bound). We prove that this strategy indeed leads to strong oracle inequalities. We illustrate our theoretical results by applications to Gaussian and multinomial mixtures.

We propose a framework for nonparametric maximum likelihood estimation of densities in situations where the sample is supplemented by information and assumptions about shape, support, continuity, slope, location of modes, density values, and other conditions that, individually or in combination, restrict the family of densities under consideration. We establish existence of estimators and their cluster points, strong consistency under mild assumptions, and robustness in the presence of model misspecification. The results are achieved by means of viewing densities as elements of spaces of semicontinuous functions with the hypo-distance metric. This metric emerges as natural and convenient when considering broad classes of side conditions. It also has the exceptional property that convergence of densities in this metric implies convergence of modes, near-modes, height of modes, and high-likelihood events. Thus, we automatically achieve strong consistency of a rich class of plug-in estimators for modes and related quantities. Relying on almost sure epi-convergence of criterion functions, we avoid the strong assumptions associated with uniform laws of large numbers and instead leverage a less demanding law, for which we provide a new proof. Specific examples illustrate the framework including an estimator simultaneously subject to bounds on density values and its (sub)gradients, restriction to concavity, penalization that encourages lower modes, and imprecise information about the expected value.

University of Technology Sydney. Faculty of Science. Generalised linear mixed models are the cornerstone of longitudinal and multilevel data analysis. However, exact inference for Bayesian mixed models with semiparametric extensions is typically intractable, requiring approximate inference methods for use in practice. Markov chain Monte Carlo or MCMC is one of the most commonly used approximate inference methods in this setting, but can be computationally intensive and often suffers from poor convergence in complex models. A faster, deterministic alternative to MCMC is variational approximations, a class of deterministic algorithms that is based on reformulating the problem of computing the posterior distribution as an optimisation problem, simplifying that problem and finding solutions to the perturbed problem. In this thesis, we work with a particular class of variational approximations, known as the mean field variational Bayes (MFVB). In essence, MFVB approximations are based upon optimising the Kullback-Leibler divergence with respect to the so-called approximating distribution. We derive MFVB algorithms for a wide variety of Bayesian semiparametric mixed models with Gaussian, Student-t, Bernoulli and Poisson responses. In order to overcome the computational cost of the direct naïve approach to the underlying MFVB calculations for models, we introduce a novel, streamlined approach that involves matrix permutation and block decomposition. Through a series of numerical studies, we demonstrate that the MFVB algorithms achieve a good level of accuracy compared to a MCMC benchmark (our gold standard). Furthermore, our developed streamlined algorithms are shown to have a complexity that is linear in the number of groups at each level, representing a two orders of magnitude improvement over the naïve approach. More importantly, the modularity of MFVB allows relatively simple extensions to more complicated scenarios, including higher-level random effects, measurement error and/or missing data problems, models with group-specific curves and real-time or online data processing. Illustrations from various real data examples are provided.

Probabilistic graphical models offer a powerful framework to account for the dependence structure between variables, which is represented as a graph. However, the dependence between variables may render inference tasks intractable. In this paper we review techniques exploiting the graph structure for exact inference, borrowed from optimisation and computer science. They are built on the principle of variable elimination whose complexity is dictated in an intricate way by the order in which variables are eliminated. The so-called treewidth of the graph characterises this algorithmic complexity: low-treewidth graphs can be processed efficiently. The first message that we illustrate is therefore the idea that for inference in graphical model, the number of variables is not the limiting factor, and it is worth checking for the treewidth before turning to approximate methods. We show how algorithms providing an upper bound of the treewidth can be exploited to derive a 'good' elimination order enabling to perform exact inference. The second message is that when the treewidth is too large, algorithms for approximate inference linked to the principle of variable elimination, such as loopy belief propagation and variational approaches, can lead to accurate results while being much less time consuming than Monte-Carlo approaches. We illustrate the techniques reviewed in this article on benchmarks of inference problems in genetic linkage analysis and computer vision, as well as on hidden variables restoration in coupled Hidden Markov Models., Comment: 47 pages, 3 tables, 12 figures

Functional magnetic resonance imaging (fMRI) is a noninvasive technique allowing the study of brain activity via the measurement of hemodynamic changes. Recently, a joint detection-estimation (JDE) framework was developed and relies on both (1) the brain activity detection and (2) the hemodynamic response function estimation, two steps that are generally addressed in a separate way. The JDE approach is a parcel-based model that alternates (1) and (2) on each parcel successively. The JDE analysis assumes that all delivered stimuli (e.g. visual, auditory, etc.) possibly generate a response everywhere in the brain although activation is likely to be induced by only some of them in specific brain areas. Inclusion of irrelevant events may degrade the results. Since the relevant conditions or stimulus types can change between different brain areas, a model selection procedure will be computationally expensive. Furthermore, criteria are not always available to select the relevant conditions prior to activation detection, especially in pathological cases. The goal of this work is to develop a JDE extension allowing an automatic selection of the relevant conditions according to the brain activity they elicit. This condition selection is done simultaneously to the analysis and adaptively through the different brain areas. Analysis on simulated and real datasets illustrate the ability of our model to select the relevant conditions and its interest compare to the standard JDE analysis., L'imagerie par résonance magnétique fonctionnelle (IRMf) est une technique non invasive permettant l'étude de l'activité cérébrale au travers des changements hémodynamiques associés. Récemment, une technique de détection-estimation conjointe (DEC) a été développée permettant d'alterner (1) la détection de l'activité cérébrale induite par une stimulation ainsi que (2) l'estimation de la fonction de réponse hémodynamique caractérisant la dynamique vasculaire; deux problèmes qui sont généralement traités indépendamment. Cette approche considère une parcellisation a priori du cerveau en zones fonctionnellement homogènes et alterne (1) et (2) sur chacune d'entre elles séparément. De manière standard, l'analyse DEC suppose que le cerveau entier peut être activé par tous les types de stimuli (visuel, auditif, etc.). Cependant la spécialisation fonctionnelle des régions cérébrales montre que l'activité d'une région n'est due qu'à certains types de stimuli. La prise en compte de stimuli non pertinents dans l'analyse, peut dégrader les résultats. La sous-famille des types de stimuli pertinents n'étant pas la même à travers le cerveau une procédure de sélection de modèles serait très coûteuse en temps de calcul. De plus, une telle sélection a priori n'est pas toujours possible surtout dans les cas pathologiques. Ce travail de thèse propose une extension de l'approche DEC permettant la sélection automatique des conditions (types de stimuli) pertinentes selon l'activité cérébrale qu'elles suscitent, cela simultanément à l'analyse et adaptativement à travers les régions cérébrales. Des exemples d'analyses sur des jeux de données simulés et réels, illustrent la capacité de l'approche DEC parcimonieuse proposée à sélectionner les conditions pertinentes ainsi que son intérêt par rapport à l'approche DEC standard.

The thesis considers non-linear probabilistic models for natural language parsing, and it primarily focuses on the class of models which do not impose strict constraints on the structure of statistical dependencies. The main contribution is the demonstration that such models are appropriate for natural language parsing tasks and provide advantages over the use of standard 'linear' methods. We demonstrate this, first, by showing that though exact inference is intractable for the studied class of models, there exist accurate and tractable approximations. Second, we show that using non-linear representations results in powerful feature induction methods simplifying construction of parsers for new problems and domains, and leading to the state-of-the-art performance. Also, we demonstrate that the latent space induced by the model can be exploited in discriminative rerankers, and that this results in the significant improvement both in the standard parsing settings and in domain adaptation.

We study a novel variational approach to solve the dynamics of Ising-like discrete spin systems. The equations are derived under mean-field approximations based on the cluster variational method. Comparison with the actual Glauber dynamics of models defined on Erdos-R\'enyi random graphs show that some of these approximations are extremely accurate, both at equilibrium and in the off-equilibrium regime, providing the same result of the Monte Carlo simulation in a much shorter time. The models studied are the ferromagnetic kinetic Ising model (both with symmetric and partially asymmetric interactions), the random field Ising model and the Viana-Bray model. Only for the latter model we find some small discrepancies in the very low temperature phase, probably due to the existence of a large number of metastable states. QC 20160905

Milestones in sparse signal reconstruction and compressive sensing can be understood in a probabilistic Bayesian context, fusing underdetermined measurements with knowledge about low level signal properties in the posterior distribution, which is maximized for point estimation. We review recent progress to advance beyond this setting. If the posterior is used as distribution to be integrated over instead of merely an optimization criterion, sparse estimators with better properties may be obtained, and applications beyond point reconstruction from fixed data can be served. We describe novel variational relaxations of Bayesian integration, characterized as well as posterior maximization, which can be solved robustly for very large models by algorithms unifying convex reconstruction and Bayesian graphical model technology. They excel in difficult real-world imaging problems where posterior maximization performance is often unsatisfactory.