Search

Due to the complex behavior arising from non-uniqueness, symmetry, and bifurcations in the solution space, solving inverse problems of nonlinear differential equations (DEs) with multiple solutions is a challenging task. To address this, we propose homotopy physics-informed neural networks (HomPINNs), a novel framework that leverages homotopy continuation and neural networks (NNs) to solve inverse problems. The proposed framework begins with the use of NNs to simultaneously approximate unlabeled observations across diverse solutions while adhering to DE constraints. Through homotopy continuation, the proposed method solves the inverse problem by tracing the observations and identifying multiple solutions. The experiments involve testing the performance of the proposed method on one-dimensional DEs and applying it to solve a two-dimensional Gray-Scott simulation. Our findings demonstrate that the proposed method is scalable and adaptable, providing an effective solution for solving DEs with multiple solutions and unknown parameters. Moreover, it has significant potential for various applications in scientific computing, such as modeling complex systems and solving inverse problems in physics, chemistry, biology, etc. • Physics-informed neural networks are used to estimate unknown parameters with physical constraints. • HomPINNs helps to identify multiple solutions and solve inverse problems. • Numerical examples demonstrate its scalability and adaptability. • HomPINNs facilitates the discovery of previously unknown solutions. [ABSTRACT FROM AUTHOR]

Deep neural networks have been shown to provide accurate function approximations in high dimensions. However, fitting network parameters requires informative training data that are often challenging to collect in science and engineering applications. This work proposes Neural Galerkin schemes based on deep learning that generate training data with active learning for numerically solving high-dimensional partial differential equations. Neural Galerkin schemes build on the Dirac-Frenkel variational principle to train networks by minimizing the residual sequentially over time, which enables adaptively collecting new training data in a self-informed manner that is guided by the dynamics described by the partial differential equations. This is in contrast to other machine learning methods that aim to fit network parameters globally in time without taking into account training data acquisition. Our finding is that the active form of gathering training data of the proposed Neural Galerkin schemes is key for numerically realizing the expressive power of networks in high dimensions. Numerical experiments demonstrate that Neural Galerkin schemes have the potential to enable simulating phenomena and processes with many variables for which traditional and other deep-learning-based solvers fail, especially when features of the solutions evolve locally such as in high-dimensional wave propagation problems and interacting particle systems described by Fokker-Planck and kinetic equations. • Coupling training of deep networks with adaptive data acquisition to solve high-dimensional evolution equations. • Enables predicting local features in high dimensions for which training with classical, uniform data acquisition fails. • Sequentially collects data over time and so avoids requirement of having rich initial data set over the time-space domain. • Suited for transport problems, for which collocation-based deep-network methods with uniform data sampling are inefficient. [ABSTRACT FROM AUTHOR]

Machine learning has been successfully applied to various fields of scientific computing in recent years. For problems with multiscale features such as flows in porous media and mechanical properties of composite materials, however, it remains difficult for machine learning methods to resolve such multiscale features, especially when the small scale information is present. In this work, we propose a sparse radial basis function neural network method to solve elliptic partial differential equations (PDEs) with multiscale coefficients. Inspired by the deep mixed residual method, we rewrite the second-order problem into a first-order system and employ multiple radial basis function neural networks (RBFNNs) to approximate unknown functions in the system. To avoid the overfitting and improve the performance of RBFNN, an additional regularization is introduced in the loss function. Thus the loss function contains two parts: the L 2 loss for the residual of the first-order system and boundary conditions, and the ℓ 1 regularization term for the weights of radial basis functions (RBFs). An algorithm for optimizing the specific loss function is introduced to accelerate the training process. The accuracy and effectiveness of the proposed method are demonstrated through a collection of multiscale problems with scale separation, discontinuity and multiple scales from one to three dimensions. Notably, the ℓ 1 regularization can achieve the goal of representing the solution by fewer RBFs. As a consequence, the total number of RBFs scales like O (ε − n τ) , where ε is the smallest scale, n is the dimensionality, and τ is typically smaller than 1. It is worth mentioning that the proposed method not only has the numerical convergence and thus provides a reliable numerical solution in three dimensions when a classical method is typically not affordable, but also outperforms most other available machine learning methods in terms of accuracy and robustness. Executable files are available at https://github.com/wangzhiwensuda/SRBFNN-for-multiscale-elliptic-problems. The program was implemented in PyTorch and was run on Linux. • A sparse radial basis function neural network (SRBFNN) is proposed to solve multiscale elliptic problems. • SRBFNN achieves higher accuracy in derivatives and solution with fewer basis functions. • The number of basis functions scales as O (ε − τ n) , where ε is the smallest scale, n is the dimensionality, and τ < 1. [ABSTRACT FROM AUTHOR]

We present a deep learning framework for quantifying and propagating uncertainty in systems governed by non-linear differential equations using physics-informed neural networks. Specifically, we employ latent variable models to construct probabilistic representations for the system states, and put forth an adversarial inference procedure for training them on data, while constraining their predictions to satisfy given physical laws expressed by partial differential equations. Such physics-informed constraints provide a regularization mechanism for effectively training deep generative models as surrogates of physical systems in which the cost of data acquisition is high, and training data-sets are typically small. This provides a flexible framework for characterizing uncertainty in the outputs of physical systems due to randomness in their inputs or noise in their observations that entirely bypasses the need for repeatedly sampling expensive experiments or numerical simulators. We demonstrate the effectiveness of our approach through a series of examples involving uncertainty propagation in non-linear conservation laws, and the discovery of constitutive laws for flow through porous media directly from noisy data. • We developed self-supervised deep learning methods for modeling stochastic systems. • Regularized adversarial inference enables learning generative models from small data. • This yields a flexible workflow for characterizing uncertainty in physical systems. • This method bypasses the need for sampling costly simulators for tackling UQ tasks. [ABSTRACT FROM AUTHOR]

We consider the inverse problem of estimating parameters of a driven diffusion (e.g., the underlying fluid flow, diffusion coefficient, or source terms) from point measurements of a passive scalar (e.g., the concentration of a pollutant). We present two particle methods that leverage the structure of the inverse problem to enable efficient computation of the forward map, one for time evolution problems and one for Dirichlet boundary-value problems. The methods scale in a natural fashion to modern computational architectures, enabling substantial speedup for applications involving sparse observations and high-dimensional unknowns. Numerical examples of applications to Bayesian inference and numerical optimization are provided. • Particle algorithms are presented for PDE-constrained inverse problems. • The methods leverage problem structure to enable fast computation of the forward map. • The methods scale in a natural fashion to modern computational architectures. • Computational complexity for these and traditional two-step algorithms is provided. • Applications to Bayesian inference and numerical optimization are provided. [ABSTRACT FROM AUTHOR]

While deep learning algorithms demonstrate a great potential in scientific computing, its application to multi-scale problems remains to be a big challenge. This is manifested by the "frequency principle" that neural networks tend to learn low frequency components first. Novel architectures such as multi-scale deep neural network (MscaleDNN) were proposed to alleviate this problem to some extent. In this paper, we construct a subspace decomposition based DNN (dubbed SD2NN) architecture for a class of multi-scale problems by combining MscaleDNN algorithms with traditional numerical analysis ideas. The proposed architecture includes one low frequency normal DNN submodule, and one (or a few) high frequency MscaleDNN submodule(s), which are designed to capture the smooth part and the oscillatory part of the multi-scale solutions simultaneously. We demonstrate that the SD2NN outperforms existing models such as MscaleDNN, through several benchmark multi-scale problems in regular or perforated domains. • We propose SD2NN, a new architecture, combining subspace decomposition from numerical analysis and multiscale DNN solvers. • SD2NN has low/high frequency submodules, simultaneously capturing smooth/oscillatory parts of multi-scale solutions. • The new method outperforms state-of-the-art DNN solvers for 1D-3D multiscale PDEs on regular and perforated domains. [ABSTRACT FROM AUTHOR]

We propose an anomaly detection method for multi-variate scientific data based on analysis of high-order joint moments. Using kurtosis as a reliable measure of outliers, we suggest that principal kurtosis vectors, by analogy to principal component analysis (PCA) vectors, signify the principal directions along which outliers appear. The inception of an anomaly, then, manifests as a change in the principal values and vectors of kurtosis. Obtaining the principal kurtosis vectors requires decomposing a fourth order joint cumulant tensor for which we use a simple, computationally less expensive approach that involves performing a singular value decomposition (SVD) over the matricized tensor. We demonstrate the efficacy of this approach on synthetic data, and develop an algorithm to identify the occurrence of a spatial and/or temporal anomalous event in scientific phenomena. The algorithm decomposes the data into several spatial sub-domains and time steps to identify regions with such events. Feature moment metrics, based on the alignments of the principal kurtosis vectors, are computed at each sub-domain and time step for all features to quantify their relative importance towards the overall kurtosis in the data. Accordingly, spatial and temporal anomaly metrics for each sub-domain are proposed using the Hellinger distance of the feature moment metric distribution from a suitable nominal distribution. We apply the algorithm to two turbulent auto-ignition combustion cases and demonstrate that the anomaly metrics reliably capture the occurrence of auto-ignition in relevant spatial sub-domains at the right time steps. • A novel unsupervised anomaly detection method for multi-scale, multi-variate data. • Hypothesis: anomalous events have a signature in the higher order joint moments. • Extends the concept of PCA to higher order moment, specifically kurtosis. • The method identifies principal vectors of kurtosis as directions of anomaly. • The algorithm is designed for distributed large-scale HPC simulations and data. [ABSTRACT FROM AUTHOR]

Abstract We introduce physics-informed neural networks – neural networks that are trained to solve supervised learning tasks while respecting any given laws of physics described by general nonlinear partial differential equations. In this work, we present our developments in the context of solving two main classes of problems: data-driven solution and data-driven discovery of partial differential equations. Depending on the nature and arrangement of the available data, we devise two distinct types of algorithms, namely continuous time and discrete time models. The first type of models forms a new family of data-efficient spatio-temporal function approximators, while the latter type allows the use of arbitrarily accurate implicit Runge–Kutta time stepping schemes with unlimited number of stages. The effectiveness of the proposed framework is demonstrated through a collection of classical problems in fluids, quantum mechanics, reaction–diffusion systems, and the propagation of nonlinear shallow-water waves. Highlights • We put forth a deep learning framework that enables the synergistic combination of mathematical models and data. • We introduce an effective mechanism for regularizing the training of deep neural networks in small data regimes. • The proposed methods enable scientific prediction and discovery from incomplete models and incomplete data. [ABSTRACT FROM AUTHOR]

We introduce a data-driven approach to building reduced dynamical models through manifold learning; the reduced latent space is discovered using Diffusion Maps (a manifold learning technique) on time series data. A second round of Diffusion Maps on those latent coordinates allows the approximation of the reduced dynamical models. This second round enables mapping the latent space coordinates back to the full ambient space (what is called lifting); it also enables the approximation of full state functions of interest in terms of the reduced coordinates. In our work, we develop and test three different reduced numerical simulation methodologies, either through pre-tabulation in the latent space and integration on the fly or by going back and forth between the ambient space and the latent space. The data-driven latent space simulation results, based on the three different approaches, are validated through (a) the latent space observation of the full simulation through the Nyström Extension formula, or through (b) lifting the reduced trajectory back to the full ambient space, via Latent Harmonics. Latent space modeling often involves additional regularization to favor certain properties of the space over others, and the mapping back to the ambient space is then constructed mostly independently from these properties; here, we use the same data-driven approach to construct the latent space and then map back to the ambient space. • Construction of reduced data-driven dynamical models. • Latent Space modeling through manifold learning. • Latent Harmonics to approximate functions defined in Latent Space (here Diffusion Maps) coordinates. • Latent-Space assisted Scientific Computing. [ABSTRACT FROM AUTHOR]

In this paper, we study a physics-informed algorithm for Wasserstein Generative Adversarial Networks (WGANs) for uncertainty quantification in solutions of partial differential equations. By using groupsort activation functions in adversarial network discriminators, network generators are utilized to learn the uncertainty in solutions of partial differential equations observed from the initial/boundary data. Under mild assumptions, we show that the generalization error of the computed generator converges to the approximation error of the network with high probability, when the number of samples are sufficiently taken. According to our established error bound, we also find that our physics-informed WGANs have higher requirement for the capacity of discriminators than that of generators. Numerical results on synthetic examples of partial differential equations are reported to validate our theoretical results and demonstrate how uncertainty quantification can be obtained for solutions of partial differential equations and the distributions of initial/boundary data. However, the quality or the accuracy of the uncertainty quantification theory in all the points in the interior is still the theoretical vacancy, and required for further research. • Wasserstein generative adversarial networks need little priors for uncertainty quantifications. • Physics-informed regularization term enforces the satisfaction of governing PDEs. • The proposed probabilistic model (WGAN-PINNs) is able to deal with non-Gaussian and spatially dependent noise. • WGAN-PINNs have good convergence and generalization properties. • Theoretically we have higher requirements on discriminators than generators. [ABSTRACT FROM AUTHOR]

• An approach for training neural networks combined with partial differential equations. • Combines a finite element framework with a machine learning library. • Learns missing physics in the partial differential equation. We present a methodology combining neural networks with physical principle constraints in the form of partial differential equations (PDEs). The approach allows to train neural networks while respecting the PDEs as a strong constraint in the optimisation as apposed to making them part of the loss function. The resulting models are discretised in space by the finite element method (FEM). The method applies to both stationary and transient as well as linear/nonlinear PDEs. We describe implementation of the approach as an extension of the existing FEM framework FEniCS and its algorithmic differentiation tool dolfin-adjoint. Through series of examples we demonstrate capabilities of the approach to recover coefficients and missing PDE operators from observations. Further, the proposed method is compared with alternative methodologies, namely, physics informed neural networks and standard PDE-constrained optimisation. Finally, we demonstrate the method on a complex cardiac cell model problem using deep neural networks. [ABSTRACT FROM AUTHOR]

This paper proves that shadowing solutions can be almost surely nonphysical. This finding invalidates the argument that small perturbations in a chaotic system can only have a small impact on its statistical behavior. This theoretical finding has implications for many applications in which chaotic dynamics plays an important role. It suggests, for example, that we can control the climate through subtle perturbations. It also suggests that numerical simulations of chaotic dynamics, such as turbulent flows and global atmosphere and ocean circulation, may fail to predict the true long-term or statistical behavior. • Shadowing orbits may not be typical orbits or physical solutions. • This atypicality implies shadowing-based sensitivity can be biased. • Numerical solutions of chaotic dynamical systems can be statistically inconsistent. • Minor parameter changes can significantly alter the statistics of a chaotic system. [ABSTRACT FROM AUTHOR]

• This numerical scheme is devised by the primal-dual weak Galerkin(PDWG) framework. • The PDWG method can be interpreted as a constraint optimization process. • The PDWG scheme conserves mass locally on each element. • Optimal order error estimates are derived for the PDWG solution. This article devises a new numerical method for first-order transport problems by using the primal-dual weak Galerkin (PD-WG) finite element method recently developed in scientific computing. The PD-WG method is based on a variational formulation of the modeling equation for which the differential operator is applied to the test function so that low regularity for the exact solution of the original equation is sufficient for computation. The PD-WG finite element method indeed yields a symmetric system involving both the original equation for the primal variable and its dual for the dual variable (also known as Lagrangian multiplier). For the linear transport problem, it is shown that the PD-WG method offers numerical solutions that conserve mass locally on each element. Optimal order error estimates in various norms are derived for the numerical solutions arising from the PD-WG method with weak regularity assumptions on the modelling equations. A variety of numerical results are presented to demonstrate the accuracy and stability of the new method. [ABSTRACT FROM AUTHOR]

This work presents a data-driven approach to the identification of spatial and temporal truncation errors for linear and nonlinear discretization schemes of Partial Differential Equations (PDEs). Motivated by the central role of truncation errors, for example in the creation of implicit Large Eddy schemes, we introduce the Sparse Identification of Truncation Errors (SITE) framework to automatically identify the terms of the modified differential equation from simulation data. We build on recent advances in the field of data-driven discovery and control of complex systems and combine it with classical work on modified differential equation analysis of Warming, Hyett, Lerat and Peyret. We augment a sparse regression-rooted approach with appropriate preconditioning routines to aid in the identification of the individual modified differential equation terms. The construction of such a custom algorithm pipeline allows attenuating of multicollinearity effects as well as automatic tuning of the sparse regression hyperparameters using the Bayesian information criterion (BIC). As proof of concept, we constrain the analysis to finite difference schemes and leave other numerical schemes open for future inquiry. Test cases include the linear advection equation with a forward-time, backward-space discretization, the Burgers' equation with a MacCormack predictor-corrector scheme and the Korteweg-de Vries equation with a Zabusky and Kruska discretization scheme. Based on variation studies, we derive guidelines for the selection of discretization parameters, preconditioning approaches and sparse regression algorithms. The results showcase highly accurate predictions underlining the promise of SITE for the analysis and optimization of discretization schemes, where analytic derivation of modified differential equations is infeasible. • Sparse regression based framework to automatically identify truncation error terms. • Proving modified differential equations can be identified from simulation data. • Preconditioning at multiple stages attenuates multicollinearity effects. • Highly accurate results for test cases with main limit being machine precision. [ABSTRACT FROM AUTHOR]