Search

Score-based generative models (SGMs) are a powerful class of generative models that exhibit remarkable empirical performance. Score-based generative modelling (SGM) consists of a ``noising'' stage, whereby a diffusion is used to gradually add Gaussian noise to data, and a generative model, which entails a ``denoising'' process defined by approximating the time-reversal of the diffusion. Existing SGMs assume that data is supported on a Euclidean space, i.e. a manifold with flat geometry. In many domains such as robotics, geoscience or protein modelling, data is often naturally described by distributions living on Riemannian manifolds and current SGM techniques are not appropriate. We introduce here Riemannian Score-based Generative Models (RSGMs), a class of generative models extending SGMs to Riemannian manifolds. We demonstrate our approach on a variety of manifolds, and in particular with earth and climate science spherical data., Neurips 2022 camera ready

Generative models are typically trained on grid-like data such as images. As a result, the size of these models usually scales directly with the underlying grid resolution. In this paper, we abandon discretized grids and instead parameterize individual data points by continuous functions. We then build generative models by learning distributions over such functions. By treating data points as functions, we can abstract away from the specific type of data we train on and construct models that are agnostic to discretization. To train our model, we use an adversarial approach with a discriminator that acts on continuous signals. Through experiments on a wide variety of data modalities including images, 3D shapes and climate data, we demonstrate that our model can learn rich distributions of functions independently of data type and resolution., Comment: AISTATS 2022 Oral camera ready. Incorporated reviewer feedback

Neural information processing systems foundation. All rights reserved. Most deep reinforcement learning algorithms are data inefficient in complex and rich environments, limiting their applicability to many scenarios. One direction for improving data efficiency is multitask learning with shared neural network parameters, where efficiency may be improved through transfer across related tasks. In practice, however, this is not usually observed, because gradients from different tasks can interfere negatively, making learning unstable and sometimes even less data efficient. Another issue is the different reward schemes between tasks, which can easily lead to one task dominating the learning of a shared model. We propose a new approach for joint training of multiple tasks, which we refer to as Distral (distill & transfer learning). Instead of sharing parameters between the different workers, we propose to share a "distilled" policy that captures common behaviour across tasks. Each worker is trained to solve its own task while constrained to stay close to the shared policy, while the shared policy is trained by distillation to be the centroid of all task policies. Both aspects of the learning process are derived by optimizing a joint objective function. We show that our approach supports efficient transfer on complex 3D environments, outperforming several related methods. Moreover, the proposed learning process is more robust to hyperparameter settings and more stable - attributes that are critical in deep reinforcement learning.

Objects are composed of a set of geometrically organized parts. We introduce an unsupervised capsule autoencoder (SCAE), which explicitly uses geometric relationships between parts to reason about objects. Since these relationships do not depend on the viewpoint, our model is robust to viewpoint changes. SCAE consists of two stages. In the first stage, the model predicts presences and poses of part templates directly from the image and tries to reconstruct the image by appropriately arranging the templates. In the second stage, the SCAE predicts parameters of a few object capsules, which are then used to reconstruct part poses. Inference in this model is amortized and performed by off-the-shelf neural encoders, unlike in previous capsule networks. We find that object capsule presences are highly informative of the object class, which leads to state-of-the-art results for unsupervised classification on SVHN (55%) and MNIST (98.7%).

The Variational Auto-Encoder (VAE) is a popular method for learning a generative model and embeddings of the data. Many real datasets are hierarchically structured. However, traditional VAEs map data in a Euclidean latent space which cannot efficiently embed tree-like structures. Hyperbolic spaces with negative curvature can. We therefore endow VAEs with a Poincaré ball model of hyperbolic geometry as a latent space and rigorously derive the necessary methods to work with two main Gaussian generalisations on that space. We empirically show better generalisation to unseen data than the Euclidean counterpart, and can qualitatively and quantitatively better recover hierarchical structures.

We develop a functional encoder-decoder approach to supervised meta-learning, where labeled data is encoded into an infinite-dimensional functional representation rather than a finite-dimensional one. Furthermore, rather than directly producing the representation, we learn a neural update rule resembling functional gradient descent which iteratively improves the representation. The final representation is used to condition the decoder to make predictions on unlabeled data. Our approach is the first to demonstrates the success of encoder-decoder style meta-learning methods like conditional neural processes on large-scale few-shot classification benchmarks such as miniImageNet and tieredImageNet, where it achieves state-of-the-art performance.

Naive approaches to amortized inference in probabilistic programs with unbounded loops can produce estimators with infinite variance. This is particularly true of importance sampling inference in programs that explicitly include rejection sampling as part of the user-programmed generative procedure. In this paper we develop a new and efficient amortized importance sampling estimator. We prove finite variance of our estimator and empirically demonstrate our method's correctness and efficiency compared to existing alternatives on generative programs containing rejection sampling loops and discuss how to implement our method in a generic probabilistic programming framework., Comment: AISTATS 2022 camera ready

Many real world tasks exhibit rich structure that is repeated across different parts of the state space or in time. In this work we study the possibility of leveraging such repeated structure to speed up and regularize learning. We start from the KL regularized expected reward objective which introduces an additional component, a default policy. Instead of relying on a fixed default policy, we learn it from data. But crucially, we restrict the amount of information the default policy receives, forcing it to learn reusable behaviors that help the policy learn faster. We formalize this strategy and discuss connections to information bottleneck approaches and to the variational EM algorithm. We present empirical results in both discrete and continuous action domains and demonstrate that, for certain tasks, learning a default policy alongside the policy can significantly speed up and improve learning., Accepted as a conference paper at ICLR 2019

Treating neural network inputs and outputs as random variables, we characterize the structure of neural networks that can be used to model data that are invariant or equivariant under the action of a compact group. Much recent research has been devoted to encoding invariance under symmetry transformations into neural network architectures, in an effort to improve the performance of deep neural networks in data-scarce, non-i.i.d., or unsupervised settings. By considering group invariance from the perspective of probabilistic symmetry, we establish a link between functional and probabilistic symmetry, and obtain generative functional representations of probability distributions that are invariant or equivariant under the action of a compact group. Our representations completely characterize the structure of neural networks that can be used to model such distributions and yield a general program for constructing invariant stochastic or deterministic neural networks. We demonstrate that examples from the recent literature are special cases, and develop the details of the general program for exchangeable sequences and arrays., Revised structure for clarity; fixed minor mistakes; incorporated reviewer feedback for publication

Current meta-learning approaches focus on learning functional representations of relationships between variables, i.e. on estimating conditional expectations in regression. In many applications, however, we are faced with conditional distributions which cannot be meaningfully summarized using expectation only (due to e.g. multimodality). Hence, we consider the problem of conditional density estimation in the meta-learning setting. We introduce a novel technique for meta-learning which combines neural representation and noise-contrastive estimation with the established literature of conditional mean embeddings into reproducing kernel Hilbert spaces. The method is validated on synthetic and real-world problems, demonstrating the utility of sharing learned representations across multiple conditional density estimation tasks.

Universal probabilistic programming systems (PPSs) provide a powerful framework for specifying rich probabilistic models. They further attempt to automate the process of drawing inferences from these models, but doing this successfully is severely hampered by the wide range of non--standard models they can express. As a result, although one can specify complex models in a universal PPS, the provided inference engines often fall far short of what is required. In particular, we show that they produce surprisingly unsatisfactory performance for models where the support varies between executions, often doing no better than importance sampling from the prior. To address this, we introduce a new inference framework: Divide, Conquer, and Combine, which remains efficient for such models, and show how it can be implemented as an automated and generic PPS inference engine. We empirically demonstrate substantial performance improvements over existing approaches on three examples., Comment: Published at the 37th International Conference on Machine Learning (ICML 2020)

Continual learning aims to improve the ability of modern learning systems to deal with non-stationary distributions, typically by attempting to learn a series of tasks sequentially. Prior art in the field has largely considered supervised or reinforcement learning tasks, and often assumes full knowledge of task labels and boundaries. In this work, we propose an approach (CURL) to tackle a more general problem that we will refer to as unsupervised continual learning. The focus is on learning representations without any knowledge about task identity, and we explore scenarios when there are abrupt changes between tasks, smooth transitions from one task to another, or even when the data is shuffled. The proposed approach performs task inference directly within the model, is able to dynamically expand to capture new concepts over its lifetime, and incorporates additional rehearsal-based techniques to deal with catastrophic forgetting. We demonstrate the efficacy of CURL in an unsupervised learning setting with MNIST and Omniglot, where the lack of labels ensures no information is leaked about the task. Further, we demonstrate strong performance compared to prior art in an i.i.d setting, or when adapting the technique to supervised tasks such as incremental class learning., Comment: NeurIPS 2019

Many machine learning tasks such as multiple instance learning, 3D shape recognition, and few-shot image classification are defined on sets of instances. Since solutions to such problems do not depend on the order of elements of the set, models used to address them should be permutation invariant. We present an attention-based neural network module, the Set Transformer, specifically designed to model interactions among elements in the input set. The model consists of an encoder and a decoder, both of which rely on attention mechanisms. In an effort to reduce computational complexity, we introduce an attention scheme inspired by inducing point methods from sparse Gaussian process literature. It reduces the computation time of self-attention from quadratic to linear in the number of elements in the set. We show that our model is theoretically attractive and we evaluate it on a range of tasks, demonstrating the state-of-the-art performance compared to recent methods for set-structured data., Comment: ICML 2019

This paper makes two contributions to Bayesian machine learning algorithms. Firstly, we propose stochastic natural gradient expectation propagation (SNEP), a novel alternative to expectation propagation (EP), a popular variational inference algorithm. SNEP is a black box variational algorithm, in that it does not require any simplifying assumptions on the distribution of interest, beyond the existence of some Monte Carlo sampler for estimating the moments of the EP tilted distributions. Further, as opposed to EP which has no guarantee of convergence, SNEP can be shown to be convergent, even when using Monte Carlo moment estimates. Secondly, we propose a novel architecture for distributed Bayesian learning which we call the posterior server. The posterior server allows scalable and robust Bayesian learning in cases where a data set is stored in a distributed manner across a cluster, with each compute node containing a disjoint subset of data. An independent Monte Carlo sampler is run on each compute node, with direct access only to the local data subset, but which targets an approximation to the global posterior distribution given all data across the whole cluster. This is achieved by using a distributed asynchronous implementation of SNEP to pass messages across the cluster. We demonstrate SNEP and the posterior server on distributed Bayesian learning of logistic regression and neural networks.

We introduce a new probability distribution over a potentially infinite number of binary Markov chains which we call the Markov Indian buffet process. This process extends the IBP to allow temporal dependencies in the hidden variables. We use this stochastic process to build a nonparametric extension of the factorial hidden Markov model. After constructing an inference scheme which combines slice sampling and dynamic programming we demonstrate how the infinite factorial hidden Markov model can be used for blind source separation.

In this paper we propose a new incremental spike sorting model that automatically eliminates refractory period violations, accounts for action potential waveform drift, and can handle "appearance" and "disappearance" of neurons. Our approach is to augment a known time-varying Dirichlet process that ties together a sequence of infinite Gaussian mixture models, one per action potential waveform observation, with an interspike-interval-dependent likelihood that prohibits refractory period violations. We demonstrate this model by showing results from sorting two publicly available neural data recordings for which a partial ground truth labeling is known.

A wide variety of Dirichlet-multinomial 'topic' models have found interesting applications in recent years. While Gibbs sampling remains an important method of inference in such models, variational techniques have certain advantages such as easy assessment of convergence, easy optimization without the need to maintain detailed balance, a bound on the marginal likelihood, and side-stepping of issues with topic-identifiability. The most accurate variational technique thus far, namely collapsed variational latent Dirichlet allocation, did not deal with model selection nor did it include inference for hyperparameters. We address both issues by generalizing the technique, obtaining the first variational algorithm to deal with the hierarchical Dirichlet process and to deal with hyperparameters of Dirichlet variables. Experiments show a significant improvement in accuracy.

We propose a novel class of Bayesian nonparametric models for sequential data called fragmentation-coagulation processes (FCPs). FCPs model a set of sequences using a partition-valued Markov process which evolves by splitting and merging clusters. An FCP is exchangeable, projective, stationary and reversible, and its equilibrium distributions are given by the Chinese restaurant process. As opposed to hidden Markov models, FCPs allow for flexible modelling of the number of clusters, and they avoid label switching non-identifiability problems. We develop an efficient Gibbs sampler for FCPs which uses uniformization and the forward-backward algorithm. Our development of FCPs is motivated by applications in population genetics, and we demonstrate the utility of FCPs on problems of genotype imputation with phased and unphased SNP data.

The sequence memoizer is a model for sequence data with state-of-the-art performance on language modeling and compression. We propose a number of improvements to the model and inference algorithm, including an enlarged range of hyperparameters, a memory-efficient representation, and inference algorithms operating on the new representation. Our derivations are based on precise definitions of the various processes that will also allow us to provide an elementary proof of the "mysterious" coagulation and fragmentation properties used in the original paper on the sequence memoizer by Wood et al. (2009). We present some experimental results supporting our improvements.

This paper presents a novel approach for exploiting the global context for the task of word sense disambiguation (WSD). This is done by using topic features constructed using the latent dirichlet allocation (LDA) algorithm on unlabeled data. The features are incorporated into a modified näive Bayes network alongside other features such as part-of-speech of neighboring words, single words in the surrounding context, local collocations, and syntactic patterns. In both the English all-words task and the English lexical sample task, the method achieved significant improvement over the simple näive Bayes classifier and higher accuracy than the best official scores on Senseval-3 for both task. © 2007 Association for Computational Linguistics.

The Indian buffet process (IBP) is an exchangeable distribution over binary matrices used in Bayesian nonparametric featural models. In this paper we propose a three-parameter generalization of the IBP exhibiting power-law behavior. We achieve this by generalizing the beta process (the de Finetti measure of the IBP) to the stable-beta process and deriving the IBP corresponding to it. We find interesting relationships between the stable-beta process and the Pitman-Yor process (another stochastic process used in Bayesian nonparametric models with interesting power-law properties). We derive a stick-breaking construction for the stable-beta process, and find that our power-law IBP is a good model for word occurrences in document corpora.

In this paper we propose a new framework for learning from large scale datasets based on iterative learning from small mini-batches. By adding the right amount of noise to a standard stochastic gradient optimization algorithm we show that the iterates will converge to samples from the true posterior distribution as we anneal the stepsize. This seamless transition between optimization and Bayesian posterior sampling provides an inbuilt protection against overfitting. We also propose a practical method for Monte Carlo estimates of posterior statistics which monitors a "sampling threshold" and collects samples after it has been surpassed. We apply the method to three models: a mixture of Gaussians, logistic regression and ICA with natural gradients. Copyright 2011 by the author(s)/owner(s).

Additive regression trees are flexible non-parametric models and popular off-the-shelf tools for real-world non-linear regression. In application domains, such as bioinformatics, where there is also demand for probabilistic predictions with measures of uncertainty, the Bayesian additive regression trees (BART) model, introduced by Chipman et al. (2010), is increasingly popular. As data sets have grown in size, however, the standard Metropolis–Hastings algorithms used to per- form inference in BART are proving inadequate. In particular, these Markov chains make local changes to the trees and suffer from slow mixing when the data are high- dimensional or the best-fitting trees are more than a few layers deep. We present a novel sampler for BART based on the Particle Gibbs (PG) algorithm (Andrieu et al., 2010) and a top-down particle filtering algorithm for Bayesian decision trees (Lakshminarayanan et al., 2013). Rather than making local changes to individual trees, the PG sampler proposes a complete tree to fit the residual. Experiments show that the PG sampler outperforms existing samplers in many settings.

Markov jump processes (or continuous-time Markov chains) are a simple and important class of continuous-time dynamical systems. In this paper, we tackle the problem of simulating from the posterior distribution over paths in these models, given partial and noisy observations. Our approach is an auxiliary variable Gibbs sampler, and is based on the idea of uniformization. This sets up a Markov chain over paths by alternately sampling a finite set of virtual jump times given the current path and then sampling a new path given the set of extant and virtual jump times using a standard hidden Markov model forward filtering-backward sampling algorithm. Our method is exact and does not involve approximations like time-discretization. We demonstrate how our sampler extends naturally to MJP-based models like Markov-modulated Poisson processes and continuous-time Bayesian networks and show significant computational benefits over state-of-the-art MCMC samplers for these models., Accepted at the Journal of Machine Learning Research (JMLR)

GBP and EP are two successful algorithms for approximate probabilistic inference, which are based on different approximation strategies. An open problem in both algorithms has been how to choose an appropriate approximation structure. We introduce 'structured region graphs', a formalism which marries these two strategies, reveals a deep connection between them, and suggests how to choose good approximation structures. In this formalism, each region has an internal structure which defines an exponential family, whose sufficient statistics must be matched by the parent region. Reduction operators on these structures allow conversion between EP and GBP free energies. Thus it is revealed that all EP approximations on discrete variables are special cases of GBP, and conversely that some wellknown GBP approximations, such as overlapping squares, are special cases of EP. Furthermore, region graphs derived from EP have a number of good structural properties, including maxent-normality and overall counting number of one. The result is a convenient framework for producing high-quality approximations with a user-adjustable level of complexity, Appears in Proceedings of the Twenty-First Conference on Uncertainty in Artificial Intelligence (UAI2005)

The Indian buffet process (IBP) is a Bayesian nonparametric distribution whereby objects are modelled using an unbounded number of latent features. In this paper we derive a stick-breaking representation for the IBP. Based on this new representation, we develop slice samplers for the IBP that are efficient, easy to implement and are more generally applicable than the currently available Gibbs sampler. This representation, along with the work of Thibaux and Jordan [17], also illuminates interesting theoretical connections between the IBP, Chinese restaurant processes, Beta processes and Dirichlet processes.

We propose a new hierarchical Bayesian n-gram model of natural languages. Our model makes use of a generalization of the commonly used Dirichlet distributions called Pitman-Yor processes which produce power-law distributions more closely resembling those in natural languages. We show that an approximation to the hierarchical Pitman-Yor language model recovers the exact formulation of interpolated Kneser-Ney, one of the best smoothing methods for n-gram language models. Experiments verify that our model gives cross entropy results superior to interpolated Kneser-Ney and comparable to modified Kneser-Ney. © 2006 Association for Computational Linguistics.

Kernel methods have been an essential instrument in machine learning over the years due to their ability to map data into high dimensional spaces efficiently. Nowadays, especially in machine learning, multimodality in the data is a common phenomenon and hence developing tools that allow us to capture these structures is crucial. Being able to reliably represent these datasets without many distributional assumptions is at the core of the main methodology used in this thesis i.e. Kernel Mean Embeddings. This search for good representations of the datasets become even more challenging when we are only presented with little data. There are two particular areas in machine learning that this thesis aims to contribute to using the kernel mean embeddings, which are 1) Meta learning and 2) Causality. Meta learning has become a widely popular area of research in machine learning due to its ability to leverage statistical dependencies between similar datasets also known as tasks. By taking advantage of task similarities, meta learning is able to learn and adapt quickly to new unseen problems. This can have many applications, where the data collection or labelling is expensive, such as in computational chemistry or health care. In this thesis, we will tackle limitations of current meta learning methods to multimodal data with the help of kernel mean embeddings and proposed efficient algorithms that do not make explicit assumptions on the distribution of the data. In chapter 2 we consider the problem of conditional density estimation and proposed MetaCDE. MetaCDE is able to accurately determine the density even in multimodal setting, whereas standard methods fail. Furthermore, in chapter 3, we also propose MetaCGNN for causal discovery. MetaCGNN is a meta learning algorithm which leverages the power of kernel mean embeddings in the little data setting and allows us to determine the causal direction in the bivariate case, i.e. determining whether X → Y or vice versa. Here again, we show that we are able to significantly improve upon existing methods. This leads us to the problem of causality. Causality is a fundamental problem in machine learning, as it allows us to reason about the cause and effects of our actions. Having dived into the realm of causality through causal discovery in chapter 3, chapter 4 will be tackling a problem on the causal inference with the help of kernel mean embeddings. Specifically, we introduce BayesIMP, and we demonstrate how to draw causal conclusions, while also accounting for the uncertainty in the data and the model. We achieve this by combining several literature on kernel methods, Gaussian processes and Causal inference. This thesis proposes several algorithms that aim to tackle some core issues in machine learning with the help of kernel mean embeddings. Given that the problem of small datasets and lack of causal awareness is likely to remain a concern in the area, we believe that this work can be a guide for future research and development.

Deep learning provides us with ever-more-sophisticated neural networks that can be tuned via gradient ascent to maximise some objective. Bayesian statistics provides us with a principled and unified approach to specify statistical models and to perform inference. One productive way to pair these two methodologies results in Deep Generative Models (DGMs), where the mappings between the statistical parameters in a probabilistic model are themselves parameterised using neural networks. In this thesis we study both the ways in which this approach can be brought to bear on various problems in machine learning and the properties of the resulting models. There are three recurrent themes in this thesis, robustness, structure and hierarchy, that reemerge throughout. We begin by studying how we can structure a deep generative model to learn in a novel learning regime we propose called semi-unsupervised learning. It is an extreme case of semi-supervised learning, where for some classes of data there are no labelled examples given whatsoever. In learning to partition the data into distinct components, different ground truth classes, the model must be able to cluster over the unlabelled classes and perform semi-supervised learning over the classes for which some labelled examples are given. We show how this can be done for a range of standard datasets. From handling one discrete latent, cluster assignment, we then study models with a hierarchy of discrete latent variables. We propose a novel way to parameterise latent variables in models of this type, relaxed responsibility vector quantisation that enables training of very deep hierarchies of layer of latent variables. This method achieves state-of-the-art results at maximising a lower bound on the evidence of the data (train and test set) for hierarchical discrete DGMs trained end-to-end, on a range of standard datasets. In doing so, these models help to close the gap between hierarchical DGMs with discrete latents and those with continuous latents, as well as providing extremely stable training. We then swap to a different problem, how to structure a model to effectively learn statistically independent latent representations from high dimensional data. We propose a hierarchical approach, where we use a bijective function, a flow, to produce an intermediate representation that a highly-constrained linear Independent Component Analysis (ICA) model then acts on. This leads to superior performance on various toy and real datasets compared to other approaches. We then study the hitherto-unconsidered problem of how to render DGMs robust to adversarial attack. We demonstrate that regularising the latent space of these models can reliably induce robustness, and obtain even more robust models by applying this regularisation to hierarchical DGMs. Finally, we then study the problem of DGM robustness from a theoretical standpoint. We define $r$-robustness, a novel criterion for DGM robustness, and then derive a margin on that criterion within which a model can be said to be robust. Combined with new theory about the optimal model for a variety of DGM where the latent space is regularised, the form of this margin sheds insight into how this regularisation increases robustness. The work presented in this thesis shows how productive the combination of deep learning and Bayesian statistics can be, as well as providing insights into the nature of the models made by their combination. This opens up new research in both of these directions—for new models that build on the work presented and also new avenues for theoretical work studying deep generative models.

Deep generative models have de facto emerged as state of the art when it comes to density estimation and sampling high-dimensional and multi-modal data. They combine the abstract yet practical mathematical description of probabilistic modelling with the flexibility and scalability brought by neural networks. They have been successfully applied to a wide spectrum of problems ranging from computer vision and natural language, to the realms of physical sciences. Probabilistic modelling, and in particular Bayesian statistics, have long enabled scientists and practitioners to include prior knowledge that they may have about the data into models. Building well-specified models is beneficial as it leads to improved generalisation capacity, data efficiency and interpretability of the model. In contrast, principled methods to encode such inductive biases in deep generative models are still under development. This thesis presents three pieces of work aimed at addressing this problem. First, we propose a principled approach to introduce inductive biases in variational auto-encoders. Taking a Bayesian perspective, we show that encoding a desired structure into the prior distribution, and applying a proper regularisation, can lead to the desired decomposition in the learnt encodings. We demonstrate this approach on a variety of computer vision datasets and successfully learn representations with sparsity, clustering, and even intricate hierarchical dependency relationships. Next, we introduce an extension of variational auto-encoders to model data with underlying hierarchical structure. As hyperbolic spaces are perfectly suited to embed tree-like data, in contrast to Euclidean geometry, we endow the latent space with hyperbolic geometry. We do so by deriving the necessary methods to work with two main Gaussian generalisations and geometry-aware architectures for the encoder and decoder networks. Finally, we leverage the formalism of Riemannian geometry to define flexible distributions for data that are assumed to live on a given manifold. We do so by extending continuous normalising flows and parametrising manifold-valued diffeomorphisms as solutions of ordinary differential equations.

Experimentation is key to learning about our world, but careful design of experiments is critical to ensure resources are used efficiently to conduct discerning investigations. Bayesian experimental design (BED) is an elegant framework that provides a mathematical definition of the expected information gain (EIG) of running a certain experiment. Finding the design with the maximal EIG will, in expectation, give experimental outcomes that are most informative about the underlying phenomenon. BED promises to launch a revolution in science and machine learning, but it is only beginning to realise its potential due to numerous unsolved computational problems. One fundamental computational issue is the estimation of EIG, where a naïve approach necessitates nested calculation of Bayesian posteriors. Further computational challenges concern the optimisation of the EIG across design space, and the design of adaptive experiments that use data that has been already observed to find the optimal design of the next experiment. In this thesis, we ask whether the machinery of modern machine learning can be brought to bear on these computational challenges, demonstrating that significant advances are possible when modern ML is combined with a deep understanding of BED. We begin by examining the EIG estimation problem, being the first to apply variational inference and inference amortisation to the problem. We then turn to optimisation of the EIG over a continuous design space, showing that stochastic gradient methods, which have not been widely adopted in BED, combine with simultaneous optimisation of variational parameters to great effect. Continuing on this theme, we show that it is possible to obtain unbiased gradients of EIG using Multi-level Monte Carlo. For the adaptive design problem, a key limitation of most methods is that they require substantial computation at each iteration of the experiment. We ask whether this process itself cannot be amortised, ultimately deriving an algorithm that trains a design policy network offline to be deployed with lightning-fast design decisions during a live experiment. Finally, we show how this policy-driven approach extends to implicit models. Together, these contributions move the field of Bayesian experimental design forward significantly in terms of what is computationally possible. Our hope is that practitioners will be able to apply these ideas to advance human understanding in many scientific disciplines.

Bayesian Statistics provide us with a powerful approach to model real-world phenomena and quantify the uncertainty therein by treating unknown factors as random. The Bayesian paradigm prescribes quantifying the prior knowledge about some state of the world, and after having obtained new information updating that knowledge in order to update the prior beliefs and propose posterior knowledge. A central approach to Bayesian Statistics is modelling, i.e. to represent a data generating process using statistical models equipped with some parameters which are to be estimated via Bayesian inference. Bayesian Nonparametric modelling comes with great flexibility as it provides infnitely many parameters. Bayesian Nonparametric models are data adaptive because when given a finite set of data, the size of the finite set of parameters to be used, adapts to the complexity of the observed data. Bayesian Nonparametric models have been used in many applications of machine learning such as density estimation, clustering, latent feature models, survival analysis and function approximation. One of the contributions of this thesis to Bayesian Nonparametrics is the formulation of priors on random graphs to model networks. Our motivation for networks sources from the fact that networks are found in numerous areas of modern society reflecting the patterns of connection across a wide range of physical and social phenomena. Our research focus is on modelling two types of networks; undirected static networks and directed networks with temporal connections. In both cases our objective is to overcome the limitations of the current literature and propose network methodology that captures real-world network properties. Precisely, the property of graph sparsity is particularly important. However, traditional Bayesian network models that assume the desirable property of exchangeability are necessarily not sparse. The first random graph to allow sparsity as well as exchangeability was recently introduced by Caron and Fox [2017] whose framework we follow. Our objective is to propose models with communities in their latent structure, and we therefore propose a generalisation of existing undirected network models with community structure to the sparse regime. Regarding temporal interaction network data, our goal is to capture reciprocation in interactions and therefore define a family of network models that generalises existing classes of reciprocating models to the sparse graph case. The key building blocks of our models as well as for various other popular Bayesian Nonparametric constructions, are infnite-activity completely random measures. The use of these random measures offers flexibility, but is also accompanied by computational complexity. The third objective of this thesis is a methodological and theoretical contribution to Bayesian Nonparametrics consisting of a novel framework to form approximations of infinite-activity completely random measures. By providing finite approximations to these infinite data structures we offer practicality and efficiency. Our constructions can be used to develop efficient posterior inference algorithms and shed light on current Bayesian Nonparametric issues of computational complexity.

Probabilistic programming is a paradigm that enables us to efficiently write probabilistic models as program code that we can sample, infer underlying parameters and predict outcomes based on complete or incomplete observations. Naturally, stochastic simulators, a special sub-class of simulators containing random variables, internal inference procedures, and the simulation of observations, are structurally rich probabilistic models. However, most simulators are not written in the probabilistic programming paradigm, as they are written in arbitrary programming code. This means that it is challenging to automatically update the variables in these simulators to account for observations from conducted experiments, which limits the simulators' use. Furthermore, there are two components to a probabilistic programming system i) the language and compilation procedure, ii) the inference procedures. These components can limit our ability to compile particular classes of probabilistic models, such as models that contain mixtures of parameter types, due to restrictions in the expressiveness of the language. Restrictions in the expressivity of the language can also inhibit our ability to generate efficient inferences, as this naturally influences the design of the probabilistic programming system and the set of available inference backends. Creating probabilistic programming systems that are expressive enough for different probabilistic models leads to the creation of many different probabilistic programming systems, which is inefficient - it would be more efficient if we could repurpose existing probabilistic programming systems. In this thesis, we develop three pieces of original work through four papers. The first piece of work describes how to extend differentiable first-order probabilistic programming systems to perform statistically correct and computationally efficient inference on models with mixtures of continuous and non-continuous parameters, without having to modify the underlying language, or develop an entirely new probabilistic programming system. The second describes how to translate real-world stochastic simulators written in arbitrary program languages to probabilistic programming systems. And finally, in the third piece of work, we develop two new Bayesian inference schemes to make inference more computationally and statistically efficient in nested models, models that contain probabilistic programs, within probabilistic programs, which arise in many real-world stochastic simulators.

Probabilistic models enable us to infer the underlying relationships within data and make decisions based on this information. Certain models are more commonly used not because they are more appropriate to imitate a particular system, but because they are simple enough to analyze given the current computational resources and available inference algorithms. However, there are a broad range of non-standard models that we could use to describe real-world tasks better; they may have complex dependency structures, variables following non-common distributions, and/or a stochastic number of variables. But we typically avoid using them as these features substantially complicate the inference procedure such that many conventional inference algorithms would fail when dealing with these models. To alleviate the computational concerns of data scientists or domain experts and free them to develop non-standard, complex models as needed, more sophisticated inference methods need to be more easily accessible. Additionally, because it is usually difficult to extract helpful information of non-standard models ahead of inference, like where the modes might be, such inference techniques should be expected to work reasonably well without too much such information. In an ideal world, domain experts would be able to specify any model of interest, and no longer need to worry about the technical details of inference algorithms or how to implement them. Moreover, these algorithms are provided as generic engines and can be used to reason the models in an automated manner. Achieving so is the ambitious, long-term goal of the emerging field called probabilistic programming. The aim of this thesis is to make inroads to this ultimate goal: we develop and automate advanced inference methods that are applicable for a broad range of non-standard probabilistic models. In particular, we introduce a novel class of adaptive inference algorithms which can be implemented in a black-box manner. They are especially useful for the models with multiple, separated modes where many off-the-shelf options struggle. Furthermore, we investigate both a restricted class of probabilistic programming systems (PPSs) that impose strong constraints on the model class to ensure inference efficiency, and universal PPSs which are designed around the long-term goal and aim to support any possible model but substantially complicate the inference procedures. For the former, we propose a principled way to extend them to incorporate a broader range of models and inference engines that are not supported before. For the latter, we develop a general framework to handle one class of the most challenging models where the support of a model can be stochastic.

The success of machine learning is due in part to the effectiveness of scalable computational methods, like stochastic gradient descent or Monte Carlo methods, that undergird learning algorithms. This thesis contributes four new scalable methods for distinct problems that arise in machine learning. It introduces a new method for gradient estimation in discrete variable models, a new objective for maximum likelihood learning in the presence of latent variables, and two new gradient-based differentiable optimization methods. Although quite different, these contributions address distinct, critical parts of a typical machine learning workflow. Furthermore, each contribution is inspired by an interplay between the numerical problems of optimization and integration, an interplay that forms the central theme of this thesis.

Bayesian machine learning has gained tremendous attention in the machine learning community over the past few years. Bayesian methods offer a coherent reasoning for quantifying uncertainties in the decision making procedure, based on the Bayes rule. One of the core advantages of Bayesian methods is the separation of modelling and inference. In other words, the likelihood models are completely independent of the computation of the posterior distribution of the parameters. There are many Bayesian models that are widely used in the machine learning community. For example, non-parametric models such as Gaussian Processes and Dirichlet Processes are flexible models which are able to capture and learn the structure of the data. Bayesian deep learning models, which are based on neural networks, are another example of flexible Bayesian models that are rich enough to represent non-linear structures in the data. The process of inferring the posterior lies at the center of Bayesian inference. When computing the posterior distribution exactly is not feasible, due to intractability of the posterior and the computational or memory constraints, approximate Bayesian inference comes to play. In this PhD thesis, I develop and investigate various Bayesian modelling and inference techniques and apply them to multiple interesting domains and tasks. We begin with Tucker Gaussian Processes(TGP), a class of flexible non-parametric models based on Gaussian Processes (GP). We apply the method to 1) regression problems on structured input data, and 2) collaborative filtering problems where TGP offers an elegant way of incorporating side information. We demonstrate superior results compared with benchmarks on a number of examples across different domains. A closely related line of research based on GPs is Bayesian Optimization (BO). It is a black-box optimizer where one optimizes an objective function through subsequent queries about next input locations to be evaluated at. However, this method does not work well when the input space is non-Euclidean or combinatorial. We alleviate the problem by learning a low dimensional Euclidean representation of the combinatorial input space with variational inference, using Variational Auto-encoder (VAE). The optimization can then be conducted on the low dimensional embedding instead. We apply our method to Automatic Statistician and natural scene understanding, which give promising results. For approximate Bayesian inference, we first propose an algorithm called Relativistic Hamiltonian Monte Carlo (RHMC) which is a variant of MCMC. In particular, we replace Newton’s kinetic energy in the Hamiltonian with Einstein’s relativistic kinetic energy, which makes the algorithm more robust. There are several extensions to RHMC, including a stochastic gradient version for scalability, a thermostat version based on the temperature of the physical system and a resulting optimization algorithm which gives comparable performance compared with the state-of-the-art. Finally, we propose another sampling based inference method called the Adaptive Importance Sampling with Exploration and Exploitation (Daisee), where we look into the problem of exploration-exploitation in adaptive importance sampling through establishing a natural connection between importance sampling and multi-armed bandit problem. In particular, through a finite-time regret analysis we show that the regret of the proposed algorithm grows sublinearly with time. Further, we propose a hierarchical extension of Daisee to encourage exploration in the region with high uncertainty. The new models proposed in this thesis help to allow for more flexible Bayesian modelling and the inference techniques introduced can open new research directions for efficient and accurate posterior inference. These contribute to Bayesian inference and probabilistic machine learning.

International audience; We consider structure discovery of undirected graphical models from observational data. Inferring likely structures from few examples is a complex task often requiring the formulation of priors and sophisticated inference procedures. Popular methods rely on estimating a penalized maximum likelihood of the precision matrix. However, in these approaches structure recovery is an indirect consequence of the data-fit term, the penalty can be difficult to adapt for domain-specific knowledge, and the inference is computationally demanding. By contrast, it may be easier to generate training samples of data that arise from graphs with the desired structure properties. We propose here to leverage this latter source of information as training data to learn a function, parametrized by a neural network that maps empirical covariance matrices to estimated graph structures. Learning this function brings two benefits: it implicitly models the desired structure or sparsity properties to form suitable priors, and it can be tailored to the specific problem of edge structure discovery, rather than maximizing data likelihood. Applying this framework, we find our learnable graph-discovery method trained on synthetic data generalizes well: identifying relevant edges in both synthetic and real data, completely unknown at training time. We find that on genetics, brain imaging, and simulation data we obtain performance generally superior to analytical methods.