Search

Data assimilation aims to estimate the states of a dynamical system by optimally combining sparse and noisy observations of the physical system with uncertain forecasts produced by a computational model. The states of many dynamical systems of interest obey nonlinear physical constraints, and the corresponding dynamics is confined to a certain sub-manifold of the state space. Standard data assimilation techniques applied to such systems yield posterior states lying outside the manifold, violating the physical constraints. This work focuses on particle flow filters which use stochastic differential equations to evolve state samples from a prior distribution to samples from an observation-informed posterior distribution. The variational Fokker-Planck (VFP) -- a generic particle flow filtering framework -- is extended to incorporate non-linear, equality state constraints in the analysis. To this end, two algorithmic approaches that modify the VFP stochastic differential equation are discussed: (i) VFPSTAB, to inexactly preserve constraints with the addition of a stabilizing drift term, and (ii) VFPDAE, to exactly preserve constraints by treating the VFP dynamics as a stochastic differential-algebraic equation (SDAE). Additionally, an implicit-explicit time integrator is developed to evolve the VFPDAE dynamics. The strength of the proposed approach for constraint preservation in data assimilation is demonstrated on three test problems: the double pendulum, Korteweg-de-Vries, and the incompressible Navier-Stokes equations.

The Adam optimizer, often used in Machine Learning for neural network training, corresponds to an underlying ordinary differential equation (ODE) in the limit of very small learning rates. This work shows that the classical Adam algorithm is a first order implicit-explicit (IMEX) Euler discretization of the underlying ODE. Employing the time discretization point of view, we propose new extensions of the Adam scheme obtained by using higher order IMEX methods to solve the ODE. Based on this approach, we derive a new optimization algorithm for neural network training that performs better than classical Adam on several regression and classification problems.

Real-world multibody systems are often subject to phenomena like friction, joint clearances, and external events. These phenomena can significantly impact the optimal design of the system and its controller. This work addresses the gradient-based optimization methodology for multibody dynamic systems with joint friction using a direct sensitivity approach for gradient computation. After a thorough review of various friction models developed over the years, the Brown McPhee model has been found to be the most suitable for the study due to its accuracy for dynamic simulation and its compatibility with sensitivity analysis. The methodology supports co-design of the system and its controller, which is especially relevant for applications like robotics and servo-mechanical systems where the actuation and the design are highly dependent on each other. Numerical results are obtained using a new implementation of the MBSVT (Multi-Body Systems at Virginia Tech) software package; MBSVT 2.0 is reprogrammed in Julia for ease of implementation while maintaining high computational efficiency. Three case studies are provided to demonstrate the attractive properties of simultaneous optimal design and control approach for certain applications.

Deep neural networks (DNN) have found wide applicability in numerous fields due to their ability to accurately learn very complex input-output relations. Despite their accuracy and extensive use, DNNs are highly susceptible to adversarial attacks due to limited generalizability. For future progress in the field, it is essential to build DNNs that are robust to any kind of perturbations to the data points. In the past, many techniques have been proposed to robustify DNNs using first-order derivative information of the network. This paper proposes a new robustification approach based on control theory. A neural network architecture that incorporates feedback control, named Feedback Neural Networks, is proposed. The controller is itself a neural network, which is trained using regular and adversarial data such as to stabilize the system outputs. The novel adversarial training approach based on the feedback control architecture is called Feedback Looped Adversarial Training (FLAT). Numerical results on standard test problems empirically show that our FLAT method is more effective than the state-of-the-art to guard against adversarial attacks.

The generalized additive Runge-Kutta (GARK) framework provides a powerful approach for solving additively partitioned ordinary differential equations. This work combines the ideas of symplectic GARK schemes and multirate GARK schemes to efficiently solve additively partitioned Hamiltonian systems with multiple time scales. Order conditions, as well as conditions for symplecticity and time-reversibility, are derived in the general setting of non-separable Hamiltonian systems. Investigations of the special case of separable Hamiltonian systems are also carried out. We show that particular partitions may introduce stability issues, and discuss partitions that enable an implicit-explicit integration leading to improved stability properties. Higher-order symplectic multirate GARK schemes based on advanced composition techniques are discussed. The performance of the schemes is demonstrated by means of the Fermi-Pasta-Ulam problem., Comment: 29 pages, 6 figures

Regularization techniques such as $\mathcal{L}_1$ and $\mathcal{L}_2$ regularizers are effective in sparsifying neural networks (NNs). However, to remove a certain neuron or channel in NNs, all weight elements related to that neuron or channel need to be prunable, which is not guaranteed by traditional regularization. This paper proposes a simple new approach named "Guided Regularization" that prioritizes the weights of certain NN units more than others during training, which renders some of the units less important and thus, prunable. This is different from the scattered sparsification of $\mathcal{L}_1$ and $\mathcal{L}_2$ regularizers where the the components of a weight matrix that are zeroed out can be located anywhere. The proposed approach offers a natural reduction of NN in the sense that a model is being trained while also neutralizing unnecessary units. We empirically demonstrate that our proposed method is effective in pruning NNs while maintaining performance.

We propose, analyze, and investigate numerically a novel two-level Galerkin reduced order model (2L-ROM) for the efficient and accurate numerical simulation of the steady Navier-Stokes equations. In the first step of the 2L-ROM, a relatively low-dimensional nonlinear system is solved. In the second step, the Navier-Stokes equations are linearized around the solution found in the first step, and a higher-dimensional system for the linearized problem is solved. We prove an error bound for the new 2L-ROM and compare it to the standard one level ROM (1L-ROM) in the numerical simulation of the steady Burgers equation. The 2L-ROM significantly decreases (by a factor of $2$ and even $3$) the 1L-ROM computational cost, without compromising its numerical accuracy.

Traditional data assimilation uses information obtained from the propagation of one physics-driven model and combines it with information derived from real-world observations in order to obtain a better estimate of the truth of some natural process. However, in many situations multiple simulation models that describe the same physical phenomenon are available. Such models can have different sources. On one hand there are theory-guided models are constructed from first physical principles, while on the other there are data-driven models that are constructed from snapshots of high fidelity information. In this work we provide a possible way to make use of this collection of models in data assimilation by generalizing the idea of model hierarchies into model forests -- collections of high fidelity and low fidelity models organized in a groping of model trees such as to capture various relationships between different models. We generalize the multifidelity ensemble Kalman filter that previously operated on model hierarchies into the model forest ensemble Kalman filter through a generalized theory of linear control variates. This new filter allows for much more freedom when treading the line between accuracy and speed. Numerical experiments with a high fidelity quasi-geostrophic model and two of its low fidelity reduced order models validate the accuracy of our approach.

A rapidly growing area of research is the use of machine learning approaches such as autoencoders for dimensionality reduction of data and models in scientific applications. We show that the canonical formulation of autoencoders suffers from several deficiencies that can hinder their performance. Using a meta-learning approach, we reformulate the autoencoder problem as a bi-level optimization procedure that explicitly solves the dimensionality reduction task. We prove that the new formulation corrects the identified deficiencies with canonical autoencoders, provide a practical way to solve it, and showcase the strength of this formulation with a simple numerical illustration.

A fundamental problem in science and engineering is designing optimal control policies that steer a given system towards a desired outcome. This work proposes Control Physics-Informed Neural Networks (Control PINNs) that simultaneously solve for a given system state, and for the optimal control signal, in a one-stage framework that conforms to the underlying physical laws. Prior approaches use a two-stage framework that first models and then controls a system in sequential order. In contrast, a Control PINN incorporates the required optimality conditions in its architecture and in its loss function. The success of Control PINNs is demonstrated by solving the following open-loop optimal control problems: (i) an analytical problem, (ii) a one-dimensional heat equation, and (iii) a two-dimensional predator-prey problem.

When applied to stiff, linear differential equations with time-dependent forcing, Runge-Kutta methods can exhibit convergence rates lower than predicted by the classical order condition theory. Commonly, this order reduction phenomenon is addressed by using an expensive, fully implicit Runge-Kutta method with high stage order or a specialized scheme satisfying additional order conditions. This work develops a flexible approach of augmenting an arbitrary Runge-Kutta method with a fully implicit method used to treat the forcing such as to maintain the classical order of the base scheme. Our methods and analyses are based on the general-structure additive Runge-Kutta framework. Numerical experiments using diagonally implicit, fully implicit, and even explicit Runge-Kutta methods confirm that the new approach eliminates order reduction for the class of problems under consideration, and the base methods achieve their theoretical orders of convergence.

Particle flow filters solve Bayesian inference problems by smoothly transforming a set of particles into samples from the posterior distribution. Particles move in state space under the flow of an McKean-Vlasov-Ito process. This work introduces the Variational Fokker-Planck (VFP) framework for data assimilation, a general approach that includes previously known particle flow filters as special cases. The McKean-Vlasov-Ito process that transforms particles is defined via an optimal drift that depends on the selected diffusion term. It is established that the underlying probability density - sampled by the ensemble of particles - converges to the Bayesian posterior probability density. For a finite number of particles the optimal drift contains a regularization term that nudges particles toward becoming independent random variables. Based on this analysis, we derive computationally-feasible approximate regularization approaches that penalize the mutual information between pairs of particles, and avoid particle collapse. Moreover, the diffusion plays a role akin to a particle rejuvenation approach that aims to alleviate particle collapse. The VFP framework is very flexible. Different assumptions on prior and intermediate probability distributions can be used to implement the optimal drift, and localization and covariance shrinkage can be applied to alleviate the curse of dimensionality. A robust implicit-explicit method is discussed for the efficient integration of stiff McKean-Vlasov-Ito processes. The effectiveness of the VFP framework is demonstrated on three progressively more challenging test problems, namely the Lorenz '63, Lorenz '96 and the quasi-geostrophic equations.

Data assimilation is the process of fusing information from imperfect computer simulations with noisy, sparse measurements of reality to obtain improved estimates of the state or parameters of a dynamical system of interest. The data assimilation procedures used in many geoscience applications, such as numerical weather forecasting, are variants of the our-dimensional variational (4D-Var) algorithm. The cost of solving the underlying 4D-Var optimization problem is dominated by the cost of repeated forward and adjoint model runs. This motivates substituting the evaluations of the physical model and its adjoint by fast, approximate surrogate models. Neural networks offer a promising approach for the data-driven creation of surrogate models. The accuracy of the surrogate 4D-Var solution depends on the accuracy with each the surrogate captures both the forward and the adjoint model dynamics. We formulate and analyze several approaches to incorporate adjoint information into the construction of neural network surrogates. The resulting networks are tested on unseen data and in a sequential data assimilation problem using the Lorenz-63 system. Surrogates constructed using adjoint information demonstrate superior performance on the 4D-Var data assimilation problem compared to a standard neural network surrogate that uses only forward dynamics information.

Rejuvenation in particle filters is necessary to prevent the collapse of the weights when the number of particles is insufficient to sample the high probability regions of the state space. Rejuvenation is often implemented in a heuristic manner by the addition of stochastic samples that widen the support of the ensemble. This work aims at improving canonical rejuvenation methodology by the introduction of additional prior information obtained from climatological samples; the dynamical particles used for importance sampling are augmented with samples obtained from stochastic covariance shrinkage. The ensemble transport particle filter, and its second order variant, are extended with the proposed rejuvenation approach. Numerical experiments show that modified filters significantly improve the analyses for low dynamical ensemble sizes.

Reduced order modeling (ROM) is a field of techniques that approximates complex physics-based models of real-world processes by inexpensive surrogates that capture important dynamical characteristics with a smaller number of degrees of freedom. Traditional ROM techniques such as proper orthogonal decomposition (POD) focus on linear projections of the dynamics onto a set of spectral features. In this paper we explore the construction of ROM using autoencoders (AE) that perform nonlinear projections of the system dynamics onto a low dimensional manifold learned from data. The approach uses convolutional neural networks (CNN) to learn spatial features as opposed to spectral, and utilize a physics informed (PI) cost function in order to capture temporal features as well. Our investigation using the quasi-geostrophic equations reveals that while the PI cost function helps with spatial reconstruction, spatial features are less powerful than spectral features, and that construction of ROMs through machine learning-based methods requires significant investigation into novel non-standard methodologies.

Generalized Additive Runge-Kutta schemes have shown to be a suitable tool for solving ordinary differential equations with additively partitioned right-hand sides. This work develops symplectic GARK schemes for additively partitioned Hamiltonian systems. In a general setting, we derive conditions for symplecticness, as well as symmetry and time-reversibility. We show how symplectic and symmetric schemes can be constructed based on schemes which are only symplectic, or only symmetric. Special attention is given to the special case of partitioned schemes for Hamiltonians split into multiple potential and kinetic energies. Finally we show how symplectic GARK schemes can leverage different time scales and evaluation costs for different potentials, and provide efficient numerical solutions by using different order for these parts.

Splitting-based time integration approaches such as fractional steps, alternating direction implicit, operator splitting, and locally one-dimensional methods partition the system of interest into components and solve individual components implicitly in a cost-effective way. This work proposes a unified formulation of splitting time integration schemes in the framework of general-structure additive Runge-Kutta (GARK) methods. Specifically, we develop implicit-implicit (IMIM) GARK schemes, provide the order conditions and stability analysis for this class, and explain their application to partitioned systems of ordinary differential equations. We show that classical splitting methods belong to the IMIM GARK family, and therefore can be studied in this unified framework. New IMIM-GARK splitting methods are developed and tested using parabolic systems.

Data assimilation is a Bayesian inference process that obtains an enhanced understanding of a physical system of interest by fusing information from an inexact physics-based model, and from noisy sparse observations of reality. The multifidelity ensemble Kalman filter (MFEnKF) recently developed by the authors combines a full-order physical model and a hierarchy of reduced order surrogate models in order to increase the computational efficiency of data assimilation. The standard MFEnKF uses linear couplings between models, and is statistically optimal in case of Gaussian probability densities. This work extends MFEnKF to work with non-linear couplings between the models. Optimal nonlinear projection and interpolation operators are obtained by appropriately trained physics-informed autoencoders, and this approach allows to construct reduced order surrogate models with less error than conventional linear methods. Numerical experiments with the canonical Lorenz '96 model illustrate that nonlinear surrogates perform better than linear projection-based ones in the context of multifidelity filtering.

Many complex applications require the solution of initial-value problems where some components change fast, while others vary slowly. Multirate schemes apply different step sizes to resolve different components of the system, according to their dynamics, in order to achieve increased computational efficiency. The stiff components of the system, fast or slow, are best discretized with implicit base methods in order to ensure numerical stability. To this end, linearly implicit methods are particularly attractive as they solve only linear systems of equations at each step. This paper develops the Multirate GARK-ROS/ROW (MR-GARK-ROS/ROW) framework for linearly-implicit multirate time integration. The order conditions theory considers both exact and approximative Jacobians. The effectiveness of implicit multirate methods depends on the coupling between the slow and fast computations; an array of efficient coupling strategies and the resulting numerical schemes are analyzed. Multirate infinitesimal step linearly-implicit methods, that allow arbitrarily small micro-steps and offer extreme computational flexibility, are constructed. The new unifying framework includes existing multirate Rosenbrock(-W) methods as particular cases, and opens the possibility to develop new classes of highly effective linearly implicit multirate integrators.

Time integration methods for solving initial value problems are an important component of many scientific and engineering simulations. Implicit time integrators are desirable for their stability properties, significantly relaxing restrictions on timestep size. However, implicit methods require solutions to one or more systems of nonlinear equations at each timestep, which for large simulations can be prohibitively expensive. This paper introduces a new family of linearly implicit multistep methods (LIMM), which only requires the solution of one linear system per timestep. Order conditions and stability theory for these methods are presented, as well as design and implementation considerations. Practical methods of order up to five are developed that have similar error coefficients, but improved stability regions, when compared to the widely used BDF methods. Numerical testing of a self-starting variable stepsize and variable order implementation of the new LIMM methods shows measurable performance improvement over a similar BDF implementation., Comment: 36 pages, 5 figures, submitted to SISC in May 2020

Simulation of complex dynamical systems arising in many applications is computationally challenging due to their size and complexity. Model order reduction, machine learning, and other types of surrogate modeling techniques offer cheaper and simpler ways to describe the dynamics of these systems but are inexact and introduce additional approximation errors. In order to overcome the computational difficulties of the full complex models, on one hand, and the limitations of surrogate models, on the other, this work proposes a new accelerated time-stepping strategy that combines information from both. This approach is based on the multirate infinitesimal general-structure additive Runge-Kutta (MRI-GARK) framework. The inexpensive surrogate model is integrated with a small timestep to guide the solution trajectory, and the full model is treated with a large timestep to occasionally correct for the surrogate model error and ensure convergence. We provide a theoretical error analysis, and several numerical experiments, to show that this approach can be significantly more efficient than using only the full or only the surrogate model for the integration.

Systems driven by multiple physical processes are central to many areas of science and engineering. Time discretization of multiphysics systems is challenging, since different processes have different levels of stiffness and characteristic time scales. The multimethod approach discretizes each physical process with an appropriate numerical method; the methods are coupled appropriately such that the overall solution has the desired accuracy and stability properties. The authors developed the general-structure additive Runge-Kutta (GARK) framework, which constructs multimethods based on Runge-Kutta schemes. This paper constructs the new GARK-ROS/GARK-ROW families of multimethods based on linearly implicit Rosenbrock/Rosenbrock-W schemes. For ordinary differential equation models, we develop a general order condition theory for linearly implicit methods with any number of partitions, using exact or approximate Jacobians. We generalize the order condition theory to two-way partitioned index-1 differential-algebraic equations. Applications of the framework include decoupled linearly implicit, linearly implicit/explicit, and linearly implicit/implicit methods. Practical GARK-ROS and GARK-ROW schemes of order up to four are constructed.

This work develops a new multifidelity ensemble Kalman filter (MFEnKF) algorithm based on linear control variate framework. The approach allows for rigorous multifidelity extensions of the EnKF, where the uncertainty in coarser fidelities in the hierarchy of models represent control variates for the uncertainty in finer fidelities. Small ensembles of high fidelity model runs are complemented by larger ensembles of cheaper, lower fidelity runs, to obtain much improved analyses at only small additional computational costs. We investigate the use of reduced order models as coarse fidelity control variates in the MFEnKF, and provide analyses to quantify the improvements over the traditional ensemble Kalman filters. We apply these ideas to perform data assimilation with a quasi-geostrophic test problem, using direct numerical simulation and a corresponding POD-Galerkin reduced order model. Numerical results show that the two-fidelity MFEnKF provides better analyses than existing EnKF algorithms at comparable or reduced computational costs.

This paper studies fixed-step convergence of implicit-explicit general linear methods. We focus on a subclass of schemes that is internally consistent, has high stage order, and favorable stability properties. Classical, index-1 differential algebraic equation, and singular perturbation convergence analyses results are given. For all these problems IMEX GLMs from the class of interest converge with the full theoretical orders under general assumptions. The convergence results require the time steps to be sufficiently small, with upper bounds that are independent on the stiffness of the problem.

This paper uses a probabilistic approach to analyze the converge of an ensemble Kalman filter solution to an exact Kalman filter solution in the simplest possible setting, the scalar case, as it allows us to build upon a rich literature of scalar probability distributions and non-elementary functions. To this end we introduce the bare-bones Scalar Pedagogical Ensemble Kalman Filter (SPEnKF). We show that in the asymptotic case of ensemble size, the expected value of both the analysis mean and variance estimate of the SPEnKF converges to that of the true Kalman filter, and that the variances of both tend towards zero, at each time moment. We also show that the ensemble converges in probability in the complementary case, when the ensemble is finite, and time is taken to infinity. Moreover, we show that in the finite-ensemble, finite-time case, variance inflation and mean correction can be leveraged to coerce the SPEnKF converge to its scalar Kalman filter counterpart. We then apply this framework to analyze perturbed observations and explain why perturbed observations ensemble Kalman filters underperform their deterministic counterparts.

The Ensemble Kalman Filters (EnKF) employ a Monte-Carlo approach to represent covariance information, and are affected by sampling errors in operational settings where the number of model realizations is much smaller than the model state dimension. To alleviate the effects of these errors EnKF relies on model-specific heuristics such as covariance localization, which takes advantage of the spatial locality of correlations among the model variables. This work proposes an approach to alleviate sampling errors that utilizes a locally averaged-in-time dynamics of the model, described in terms of a climatological covariance of the dynamical system. We use this covariance as the target matrix in covariance shrinkage methods, and develop a stochastic covariance shrinkage approach where synthetic ensemble members are drawn to enrich both the ensemble subspace and the ensemble transformation. We additionally provide for a way in which this methodology can be localized similar to the state-of-the-art LETKF method, and that for a certain model setup, our methodology significantly outperforms it.

High-order discretizations of partial differential equations (PDEs) necessitate high-order time integration schemes capable of handling both stiff and nonstiff operators in an efficient manner. Implicit-explicit (IMEX) integration based on general linear methods (GLMs) offers an attractive solution due to their high stage and method order, as well as excellent stability properties. The IMEX characteristic allows stiff terms to be treated implicitly and nonstiff terms to be efficiently integrated explicitly. This work develops two systematic approaches for the development of IMEX GLMs of arbitrary order with stages that can be solved in parallel. The first approach is based on diagonally implicit multistage integration methods (DIMSIMs) of types 3 and 4. The second is a parallel generalization of IMEX Euler and has the interesting feature that the linear stability is independent of the order of accuracy. Numerical experiments confirm the theoretical rates of convergence and reveal that the new schemes are more efficient than serial IMEX GLMs and IMEX Runge-Kutta methods.

This paper develops a general methodology for a posteriori error estimation in time-dependent multiphysics numerical simulations. The methodology builds upon the generalized-structure additive Runge--Kutta (GARK) approach to time integration. GARK provides a unified formulation of multimethods that simulate complex systems by applying different discretization formulas and/or different time steps to individual components of the system. We derive discrete GARK adjoints and analyze their time accuracy. Based on the adjoint method, we establish computable a posteriori identities for the impacts of both temporal and spatial discretization errors on a given goal function. Numerical examples with reaction-diffusion systems illustrate the accuracy of the derived error measures. Local error decompositions are used to illustrate the power of this framework in adaptive refinements of both temporal and spatial meshes., Comment: 25 pages, 7 figures

Multiphysics systems are driven by multiple processes acting simultaneously, and their simulation leads to partitioned systems of differential equations. This paper studies the solution of partitioned systems of differential equations using exponential Runge-Kutta methods. We propose specific multiphysics implementations of exponential Runge-Kutta methods satisfying stiff order conditions that were developed in [Hochbruck et al., SISC, 1998] and [Luan and Osterman, JCAM, 2014]. We reformulate stiffly--accurate exponential Runge--Kutta methods in a way that naturally allows of the structure of multiphysics systems, and discuss their application to both component and additively partitioned systems. The resulting partitioned exponential methods only compute matrix functions of the Jacobians of individual components, rather than the Jacobian of the full, coupled system. We derive modified formulations of particular methods of order two, three and four, and apply them to solve a partitioned reaction-diffusion problem. The proposed methods retain full order for several partitionings of the discretized problem, including by components and by physical processes.

This work considers multirate generalized-structure additively partitioned Runge-Kutta (MrGARK) methods for solving stiff systems of ordinary differential equations (ODEs) with multiple time scales. These methods treat different partitions of the system with different timesteps for a more targeted and efficient solution compared to monolithic single rate approaches. With implicit methods used across all partitions, methods must find a balance between stability and the cost of solving nonlinear equations for the stages. In order to characterize this important trade-off, we explore multirate coupling strategies, problems for assessing linear stability, and techniques to efficiently implement Newton iterations for stage equations. Unlike much of the existing multirate stability analysis which is limited in scope to particular methods, we present general statements on stability and describe fundamental limitations for certain types of multirate schemes. New implicit multirate methods up to fourth order are derived, and their accuracy and efficiency properties are verified with numerical tests.

The Rosenbrock-Krylov family of time integration schemes is an extension of Rosenbrock-W methods that employs a specific Krylov based approximation of the linear system solutions arising within each stage of the integrator. This work proposes an extension of Rosenbrock-Krylov methods to address stability questions which arise for methods making use of inexact linear system solution strategies. Two approaches for improving the stability and efficiency of Rosenbrock-Krylov methods are proposed, one through direct control of linear system residuals and the second through a novel extension of the underlying Krylov space to include stage right hand side vectors. Rosenbrock-Krylov methods employing the new approaches show a substantial improvement in computational efficiency relative to prior implementations.

Many scientific applications require the solution of large initial-value problems, such as those produced by the method of lines after semi-discretization in space of partial differential equations. The computational cost of implicit time discretizations is dominated by the solution of nonlinear systems of equations at each time step. In order to decrease this cost, the recently developed Rosenbrock-Krylov (ROK) time integration methods extend the classical linearly-implicit Rosenbrock(-W) methods, and make use of a Krylov subspace approximation to the Jacobian computed via an Arnoldi process. Since the ROK order conditions rely on the construction of a single Krylov space, no restarting of the Arnoldi process is allowed, and the iterations quickly become expensive with increasing subspace dimensions. This work extends the ROK framework to make use of the Lanczos biorthogonalization procedure for constructing Jacobian approximations. The resulting new family of methods is named biorthogonal ROK (BOROK). The Lanczos procedure's short two-term recurrence allows BOROK methods to utilize larger subspaces for the Jacobian approximation, resulting in increased numerical stability of the time integration at a reduced computational cost. Adaptive subspace size selection and basis extension procedures are also developed for the new schemes. Numerical experiments show that for stiff problems, where a large subspace used to approximate the Jacobian is required for stability, the BOROK methods outperform the original ROK methods.

Multiphysics problems involving two or more coupled physical phenomena are ubiquitous in science and engineering. This work develops a new partitioned exponential approach for the time integration of multiphysics problems. After a possible semi-discretization in space, the class of problems under consideration is modeled by a system of ordinary differential equations where the right-hand side is a summation of two component functions, each corresponding to a given set of physical processes. The partitioned-exponential methods proposed herein evolve each component of the system via an exponential integrator, and information between partitions is exchanged via coupling terms. The traditional approach to constructing exponential methods, based on the variation-of-constants formula, is not directly applicable to partitioned systems. Rather, our approach to developing new partitioned-exponential families is based on a general-structure additive formulation of the schemes. Two method formulations are considered, one based on a linear-nonlinear splitting of the right hand component functions, and another based on approximate Jacobians. The paper develops classical (non-stiff) order conditions theory for partitioned exponential schemes based on particular families of T-trees and B-series theory. Several practical methods of third order are constructed that extend the Rosenbrock-type and EPIRK families of exponential integrators. Several implementation optimizations specific to the application of these methods to reaction-diffusion systems are also discussed. Numerical experiments reveal that the new partitioned-exponential methods can perform better than traditional unpartitioned exponential methods on some problems., Comment: Fixed a definition and other minor typos. Results remain unchanged

Alternating Directions Implicit (ADI) integration is an operator splitting approach to solve parabolic and elliptic partial differential equations in multiple dimensions based on solving sequentially a set of related one-dimensional equations. Classical ADI methods have order at most two, due to the splitting errors. Moreover, when the time discretization of stiff one-dimensional problems is based on Runge-Kutta schemes, additional order reduction may occur. This work proposes a new ADI approach based on the partitioned General Linear Methods framework. This approach allows the construction of high order ADI methods. Due to their high stage order, the proposed methods can alleviate the order reduction phenomenon seen with other schemes. Numerical experiments are shown to provide further insight into the accuracy, stability, and applicability of these new methods.

ODE Test Problems (OTP) is an object-oriented MATLAB package offering a broad range of initial value problems which can be used to test numerical methods such as time integration methods and data assimilation (DA) methods. It includes problems that are linear and nonlinear, homogeneous and nonhomogeneous, autonomous and nonautonomous, scalar and high-dimensional, stiff and nonstiff, and chaotic and nonchaotic. Many are real-world problems from fields such as chemistry, astrophysics, meteorology, and electrical engineering. OTP also supports partitioned ODEs for testing IMEX methods, multirate methods, and other multimethods. Functions for plotting solutions and creating movies are available for all problems, and exact solutions are provided when available. OTP is desgined for ease of use-meaning that working with and modifying problems is simple and intuitive.

Traditional time discretization methods use a single timestep for the entire system of interest and can perform poorly when the dynamics of the system exhibits a wide range of time scales. Multirate infinitesimal step (MIS) methods (Knoth and Wolke, 1998) offer an elegant and flexible approach to efficiently integrate such systems. The slow components are discretized by a Runge-Kutta method, and the fast components are resolved by solving modified fast differential equations. Sandu (2018) developed the Multirate Infinitesimal General-structure Additive Runge-Kutta (MRI-GARK) family of methods that includes traditional MIS schemes as a subset. The MRI-GARK framework allowed the construction of the first fourth order MIS schemes. This framework also enabled the introduction of implicit methods, which are decoupled in the sense that any implicitness lies entirely within the fast or slow integrations. It was shown by Sandu that the stability of decoupled implicit MRI-GARK methods has limitations when both the fast and slow components are stiff and interact strongly. This work extends the MRI-GARK framework by introducing coupled implicit methods to solve stiff multiscale systems. The coupled approach has the potential to considerably improve the overall stability of the scheme, at the price of requiring implicit stage calculations over the entire system. Two coupling strategies are considered. The first computes coupled Runge-Kutta stages before solving a single differential equation to refine the fast solution. The second alternates between computing coupled Runge-Kutta stages and solving fast differential equations. We derive order conditions and perform the stability analysis for both strategies. The new coupled methods offer improved stability compared to the decoupled MRI-GARK schemes. The theoretical properties of the new methods are validated with numerical experiments., Comment: arXiv admin note: substantial text overlap with arXiv:1808.02759

Ever since its inception, the Ensemble Kalman Filter has elicited many heuristic methods that sought to correct it. One such method is localization---the thought that `nearby' variables should be highly correlated with `far away' variable not. Recognizing that correlation is a time-dependent property, adaptive localization is a natural extension to these heuristics. We propose a Bayesian approach to adaptive Schur-product localization for the DEnKF, and extend it to support multiple radii of influence. We test both the empirical validity of (multivariate) adaptive localization, and of our approach. We test a simple toy problem (Lorenz'96), extending it to a multivariate model, and a more realistic geophysical problem (1.5 Layer Quasi-Geostrophic). We show that the multivariate approach has great promise on the toy problem, and that the univariate approach leads to improved filter performance for the realistic geophysical problem.

Differential equations arising in many practical applications are characterized by multiple time scales. Multirate time integration seeks to solve them efficiently by discretizing each scale with a different, appropriate time step, while ensuring the overall accuracy and stability of the numerical solution. In a seminal paper Knoth and Wolke (APNUM, 1998) proposed a hybrid solution approach: discretize the slow component with an explicit Runge-Kutta method, and advance the fast component via a modified fast differential equation. The idea led to the development of multirate infinitesimal step (MIS) methods by Wensch et al. (BIT, 2009.)G\"{u}nther and Sandu (BIT, 2016) explained MIS schemes as a particular case of multirate General-structure Additive Runge-Kutta (MR-GARK) methods. The hybrid approach offers extreme flexibility in the choice of the numerical solution process for the fast component. This work constructs a family of multirate infinitesimal GARK schemes (MRI-GARK) that extends the hybrid dynamics approachin multiple ways. Order conditions theory and stability analyses are developed, and practical explicit and implicit methods of up to order four are constructed. Numerical results confirm the theoretical findings. We expect the new MRI-GARK family to be most useful for systems of equations with widely disparate time scales, where the fast process is dispersive, and where the influence of the fast component on the slow dynamics is weak.

Multirate time integration methods apply different step sizes to resolve different components of the system based on the local activity levels. This local selection of step sizes allows increased computational efficiency while achieving the desired solution accuracy. While the multirate idea is elegant and has been around for decades, multirate methods are not yet widely used in applications. This is due, in part, to the difficulties raised by the construction of high order multirate schemes. Seeking to overcome these challenges, this work focuses on the design of practical high-order multirate methods using the theoretical framework of generalized additive Runge-Kutta (MrGARK) methods, which provides the generic order conditions and the linear and nonlinear stability analyses. A set of design criteria for practical multirate methods is defined herein: method coefficients should be generic in the step size ratio, but should not depend strongly on this ratio; unnecessary coupling between the fast and the slow components should be avoided; and the step size controllers should adjust both the micro- and the macro-steps. Using these criteria, we develop MrGARK schemes of up to order four that are explicit-explicit (both the fast and slow component are treated explicitly), implicit-explicit (implicit in the fast component and explicit in the slow one), and explicit-implicit (explicit in the fast component and implicit in the slow one). Numerical experiments illustrate the performance of these new schemes.

Complex numerical weather prediction models incorporate a variety of physical processes, each described by multiple alternative physical schemes with specific parameters. The selection of the physical schemes and the choice of the corresponding physical parameters during model configuration can significantly impact the accuracy of model forecasts. There is no combination of physical schemes that works best for all times, at all locations, and under all conditions. It is therefore of considerable interest to understand the interplay between the choice of physics and the accuracy of the resulting forecasts under different conditions. This paper demonstrates the use of machine learning techniques to study the uncertainty in numerical weather prediction models due to the interaction of multiple physical processes. The first problem addressed herein is the estimation of systematic model errors in output quantities of interest at future times, and the use of this information to improve the model forecasts. The second problem considered is the identification of those specific physical processes that contribute most to the forecast uncertainty in the quantity of interest under specified meteorological conditions. The discrepancies between model results and observations at past times are used to learn the relationships between the choice of physical processes and the resulting forecast errors. Numerical experiments are carried out with the Weather Research and Forecasting (WRF) model. The output quantity of interest is the model precipitation, a variable that is both extremely important and very challenging to forecast. The physical processes under consideration include various micro-physics schemes, cumulus parameterizations, short wave, and long wave radiation schemes. The experiments demonstrate the strong potential of machine learning approaches to aid the study of model errors., Comment: 23 pages, 5 figures, 4 tables

Sensitivity analysis of multibody systems computes the derivatives of general cost functions that depend on the system solution with respect to parameters or initial conditions. This work develops adjoint sensitivity analysis for hybrid multibody dynamic systems. Hybrid systems are characterized by trajectories that are piecewise continuous in time, with finitely-many discontinuities being caused by events such as elastic/inelastic impacts or sudden changes in constraints. The corresponding direct and adjoint sensitivity variables are also discontinuous at the time of events. The framework discussed herein uses a jump sensitivity matrix to relate the jump conditions for the direct and adjoint sensitivities before and after the time event, and provides analytical jump equations for the adjoint variables. The theoretical framework for sensitivities for hybrid systems is validated on a five-bar mechanism with non-smooth contacts., Comment: arXiv admin note: text overlap with arXiv:1710.04292

Data assimilation plays a key role in large-scale atmospheric weather forecasting, where the state of the physical system is estimated from model outputs and observations, and is then used as initial condition to produce accurate future forecasts. The Ensemble Kalman Filter (EnKF) provides a practical implementation of the statistical solution of the data assimilation problem and has gained wide popularity as. This success can be attributed to its simple formulation and ease of implementation. EnKF is a Monte-Carlo algorithm that solves the data assimilation problem by sampling the probability distributions involved in Bayes theorem. Because of this, all flavors of EnKF are fundamentally prone to sampling errors when the ensemble size is small. In typical weather forecasting applications, the model state space has dimension $10^{9}-10^{12}$, while the ensemble size typically ranges between $30-100$ members. Sampling errors manifest themselves as long-range spurious correlations and have been shown to cause filter divergence. To alleviate this effect covariance localization dampens spurious correlations between state variables located at a large distance in the physical space, via an empirical distance-dependent function. The quality of the resulting analysis and forecast is greatly influenced by the choice of the localization function parameters, e.g., the radius of influence. The localization radius is generally tuned empirically to yield desirable results.This work, proposes two adaptive algorithms for covariance localization in the EnKF framework, both based on a machine learning approach. The first algorithm adapts the localization radius in time, while the second algorithm tunes the localization radius in both time and space. Numerical experiments carried out with the Lorenz-96 model, and a quasi-geostrophic model, reveal the potential of the proposed machine learning approaches., Comment: 23 pages, 12 figures

This paper provides an analytical methodology to compute the sensitivities with respect to system parameters for any second order hybrid Ordinary Differential Equation (ODE) system. The hybrid ODE system is characterized by discontinuities in the velocity state variables due to an impulsive jump caused by an instantaneous impact in the motion of the system. The analytical methodology that solves this problem is structured based on jumping conditions for both the state variables and the sensitivities matrix. The proposed analytical approach is of the benchmarked against a numerical method., Comment: Direct sensitivity analysis, Switching sensitivities, Constrained systems, Impulsive systems