Search

Neural networks (NN) can be divided into two broad categories, recurrent and non-recurrent. Both types of neural networks are popular and extensively studied, but they are often treated as distinct families of machine learning algorithms. In this position paper, we argue that there is a closer relationship between these two types of neural networks than is normally appreciated. We show that many common neural network models, such as Recurrent Neural Networks (RNN), Multi-Layer Perceptrons (MLP), and even deep multi-layer transformers, can all be represented as iterative maps. The close relationship between RNNs and other types of NNs should not be surprising. In particular, RNNs are known to be Turing complete, and therefore capable of representing any computable function (such as any other types of NNs), but herein we argue that the relationship runs deeper and is more practical than this. For example, RNNs are often thought to be more difficult to train than other types of NNs, with RNNs being plagued by issues such as vanishing or exploding gradients. However, as we demonstrate in this paper, MLPs, RNNs, and many other NNs lie on a continuum, and this perspective leads to several insights that illuminate both theoretical and practical aspects of NNs.

Most ordinary differential equation (ODE) models used to describe biological or physical systems must be solved approximately using numerical methods. Perniciously, even those solvers which seem sufficiently accurate for the forward problem, i.e., for obtaining an accurate simulation, may not be sufficiently accurate for the inverse problem, i.e., for inferring the model parameters from data. We show that for both fixed step and adaptive step ODE solvers, solving the forward problem with insufficient accuracy can distort likelihood surfaces, which may become jagged, causing inference algorithms to get stuck in local "phantom" optima. We demonstrate that biases in inference arising from numerical approximation of ODEs are potentially most severe in systems involving low noise and rapid nonlinear dynamics. We reanalyze an ODE changepoint model previously fit to the COVID-19 outbreak in Germany and show the effect of the step size on simulation and inference results. We then fit a more complicated rainfall-runoff model to hydrological data and illustrate the importance of tuning solver tolerances to avoid distorted likelihood surfaces. Our results indicate that when performing inference for ODE model parameters, adaptive step size solver tolerances must be set cautiously and likelihood surfaces should be inspected for characteristic signs of numerical issues.

Ordinary differential equation models are used to describe dynamic processes across biology. To perform likelihood-based parameter inference on these models, it is necessary to specify a statistical process representing the contribution of factors not explicitly included in the mathematical model. For this, independent Gaussian noise is commonly chosen, with its use so widespread that researchers typically provide no explicit justification for this choice. This noise model assumes `random' latent factors affect the system in ephemeral fashion resulting in unsystematic deviation of observables from their modelled counterparts. However, like the deterministically modelled parts of a system, these latent factors can have persistent effects on observables. Here, we use experimental data from dynamical systems drawn from cardiac physiology and electrochemistry to demonstrate that highly persistent differences between observations and modelled quantities can occur. Considering the case when persistent noise arises due only to measurement imperfections, we use the Fisher information matrix to quantify how uncertainty in parameter estimates is artificially reduced when erroneously assuming independent noise. We present a workflow to diagnose persistent noise from model fits and describe how to remodel accounting for correlated errors.

We develop an \textit{a posteriori} error analysis for a numerical estimate of the time at which a functional of the solution to a partial differential equation (PDE) first achieves a threshold value on a given time interval. This quantity of interest (QoI) differs from classical QoIs which are modeled as bounded linear (or nonlinear) functionals {of the solution}. Taylor's theorem and an adjoint-based \textit{a posteriori} analysis is used to derive computable and accurate error estimates in the case of semi-linear parabolic and hyperbolic PDEs. The accuracy of the error estimates is demonstrated through numerical solutions of the one-dimensional heat equation and linearized shallow water equations (SWE), representing parabolic and hyperbolic cases, respectively.

We construct a space-time parallel method for solving parabolic partial differential equations by coupling the Parareal algorithm in time with overlapping domain decomposition in space. The goal is to obtain a discretization consisting of "local" problems that can be solved on parallel computers efficiently. However, this introduces significant sources of error that must be evaluated. Reformulating the original Parareal algorithm as a variational method and implementing a finite element discretization in space enables an adjoint-based a posteriori error analysis to be performed. Through an appropriate choice of adjoint problems and residuals the error analysis distinguishes between errors arising due to the temporal and spatial discretizations, as well as between the errors arising due to incomplete Parareal iterations and incomplete iterations of the domain decomposition solver. We first develop an error analysis for the Parareal method applied to parabolic partial differential equations, and then refine this analysis to the case where the associated spatial problems are solved using overlapping domain decomposition. These constitute our Time Parallel Algorithm (TPA) and Space-Time Parallel Algorithm (STPA) respectively. Numerical experiments demonstrate the accuracy of the estimator for both algorithms and the iterations between distinct components of the error.

Classical a posteriori error analysis for differential equations quantifies the error in a Quantity of Interest (QoI) which is represented as a bounded linear functional of the solution. In this work we consider a posteriori error estimates of a quantity of interest that cannot be represented in this fashion, namely the time at which a threshold is crossed for the first time. We derive two representations for such errors and use an adjoint-based a posteriori approach to estimate unknown terms that appear in our representation. The first representation is based on linearizations using Taylor's Theorem. The second representation is obtained by implementing standard root-finding techniques. We provide several examples which demonstrate the accuracy of the methods. We then embed these error estimates within a framework to provide error bounds on a cumulative distribution function when parameters of the differential equations are uncertain.

Domain decomposition methods are widely used for the numerical solution of partial differential equations on high performance computers. We develop an adjoint-based a posteriori error analysis for both multiplicative and additive overlapping Schwarz domain decomposition methods. The numerical error in a user-specified functional of the solution (quantity of interest) is decomposed into contributions that arise as a result of the finite iteration between the subdomains and from the spatial discretization. The spatial discretization contribution is further decomposed into contributions arising from each subdomain. This decomposition of the numerical error is used to construct a two stage solution strategy that efficiently reduces the error in the quantity of interest by adjusting the relative contributions to the error.

Here we develop a lung ventilation model, based a continuum poroelastic representation of lung parenchyma and a 0D airway tree flow model. For the poroelastic approximation we design and implement a lowest order stabilised finite element method. This component is strongly coupled to the 0D airway tree model. The framework is applied to a realistic lung anatomical model derived from computed tomography data and an artificially generated airway tree to model the conducting airway region. Numerical simulations produce physiologically realistic solutions, and demonstrate the effect of airway constriction and reduced tissue elasticity on ventilation, tissue stress and alveolar pressure distribution. The key advantage of the model is the ability to provide insight into the mutual dependence between ventilation and deformation. This is essential when studying lung diseases, such as chronic obstructive pulmonary disease and pulmonary fibrosis. Thus the model can be used to form a better understanding of integrated lung mechanics in both the healthy and diseased states.

We compute approximate solutions to inverse problems for determining parameters in differential equation models with stochastic data on output quantities. The formulation of the problem and modeling framework define a solution as a probability measure on the parameter domain for a given $\sigma-$algebra. In the case where the number of output quantities is less than the number of parameters, the inverse of the map from parameters to data defines a type of generalized contour map. The approximate contour maps define a geometric structure on events in the $\sigma-$algebra for the parameter domain. We develop and analyze an inherently non-intrusive method of sampling the parameter domain and events in the given $\sigma-$algebra to approximate the probability measure. We use results from stochastic geometry for point processes to prove convergence of a random sample based approximation method. We define a numerical $\sigma-$algebra on which we compute probabilities and derive computable estimates for the error in the probability measure. We present numerical results to illustrate the various sources of error for a model of fluid flow past a cylinder., Comment: 26 pages, 24 figures

A stabilized conforming mixed finite element method for the three-field (displacement, fluid flux and pressure) poroelasticity problem is developed and analyzed. We use the lowest possible approximation order, namely piecewise constant approximation for the pressure and piecewise linear continuous elements for the displacements and fluid flux. By applying a local pressure jump stabilization term to the mass conservation equation we ensure stability and avoid pressure oscillations. Importantly, the discretization leads to a symmetric linear system. For the fully discretized problem we prove existence and uniqueness, an energy estimate and an optimal a-priori error estimate, including an error estimate for the divergence of the fluid flux. Numerical experiments in 2D and 3D illustrate the convergence of the method, show the effectiveness of the method to overcome spurious pressure oscillations, and evaluate the added mass effect of the stabilization term., Comment: 25 pages