Search

• We propose a new strategy for efficient data-driven heliospheric solar wind modeling. • The proposed method generalizes to a wide class of advection-dominated phenomena. • This work extends the operator inference method to exploit translational symmetries. • We showcase the performance of our method on 2D and 3D magnetohydrodynamic models. Solar wind conditions are predominantly predicted via three-dimensional numerical magnetohydrodynamic (MHD) models. Despite their ability to produce highly accurate predictions, MHD models require computationally intensive high-dimensional simulations. This renders them inadequate for making time-sensitive predictions and for large-ensemble analysis required in uncertainty quantification. This paper presents a new data-driven reduced-order model (ROM) capability for forecasting heliospheric solar wind speeds. Traditional model reduction methods based on Galerkin projection have difficulties with advection-dominated systems—such as solar winds—since they require a large number of basis functions and can become unstable. A core contribution of this work addresses this challenge by extending the non-intrusive operator inference ROM framework to exploit the translational symmetries present in the solar wind caused by the Sun's rotation. The numerical results show that our method can adequately emulate the MHD simulations and is more accurate than a reduced-physics surrogate model, the Heliospheric Upwind Extrapolation model. [ABSTRACT FROM AUTHOR]

Deep operator networks (DeepONets) are powerful and flexible architectures that are attracting attention in multiple fields due to their utility for fast and accurate emulation of complex dynamics. As their remarkable generalization capabilities are primarily enabled by their projection-based attribute, in this paper, we investigate connections with low-rank techniques derived from the singular value decomposition (SVD). We demonstrate that some of the concepts behind proper orthogonal decomposition (POD)-neural networks can improve DeepONet's design and training phases. These ideas lead us to a methodology extension that we name SVD-DeepONet. Moreover, through multiple SVD analyses of scenario- and time-aggregated snapshots matrices, we find that DeepONet inherits from its projection-based attribute strong inefficiencies in representing dynamics characterized by symmetries. Inspired by the work on shifted-POD, we develop flexDeepONet, an architecture enhancement that relies on a pre-transformation network for generating a moving reference frame and isolating the rigid components of the dynamics. In this way, the physics can be represented on a latent space free from rotations, translations, and stretches, and an accurate projection can be performed to a low-dimensional basis. In addition to improving DeepONet's flexibility and interpretability, the proposed perspectives increase its generalization capabilities and computational efficiencies. For instance, we show flexDeepONet can accurately surrogate the dynamics of 19 thermodynamic variables in a combustion chemistry application by relying on 95% fewer trainable parameters than that of the 'vanilla' architecture. As stressed in the paper, we argue that DeepONet and SVD-based methods can reciprocally benefit from each other. In particular, the flexibility of the former in leveraging multiple data sources and multifidelity knowledge in the form of both unstructured data and physics-informed constraints has the potential to greatly extend the applicability of methodologies such as POD and principal component analysis (PCA). [ABSTRACT FROM AUTHOR]

• A probabilistic physics-informed neural network for the P S N curve is proposed. • Three forms of physical knowledge are considered. • A loss function is constructed to encode the derivatives with physical constraints. • The model performance is evaluated by the goodness of fit and physical consistency. Machine learning has drawn growing attention from the areas of fatigue, fracture, and structural integrity. However, most current studies are fully data-driven and may contradict the underpinning physical knowledge. To address this issue, we propose a physically consistent framework for fatigue life prediction that uses a probabilistic physics-informed neural network (PINN) to incorporate the physics underpinning the fatigue mechanism. Particularly, we consider the scatter of the fatigue life using a probabilistic neural network with the output to parametrize the fatigue life distribution. Then use neural networks' inherent backpropagation capabilities to automatically compute the derivatives that represent the physical knowledge. Finally, construct a composite loss function to encode the derivatives with certain physical constraints and uses a negative log-likelihood function to consider both failure data and run-out data. This enforces the network training process to learn a continuous function that describes the stress-life relationship satisfying both experimental data and physical knowledge. We demonstrate the proposed framework with sensitivity analysis and a comparison to the fully data-driven neural networks and the conventional statistical methods using the fatigue test data of three different materials. The results show that the proposed framework has a robust performance to effectively reflect the underlying physical knowledge and prevent overfitting issues. The findings provide a better understanding of neural networks' application to fatigue life prediction and suggest that one should be cautious when using a fully data-driven approach in scientific applications. [ABSTRACT FROM AUTHOR]

The computer simulation of organ-scale biomechanistic models of cancer personalized via routinely collected clinical and imaging data enables to obtain patient-specific predictions of tumor growth and treatment response over the anatomy of the patient's affected organ. These patient-specific computational forecasts have been regarded as a promising approach to personalize the clinical management of cancer and derive optimal treatment plans for individual patients, which constitute timely and critical needs in clinical oncology. However, the computer simulation of the underlying spatiotemporal models can entail a prohibitive computational cost, which constitutes a barrier to the successful development of clinically-actionable computational technologies for personalized tumor forecasting. To address this issue, here we propose to utilize dynamic-mode decomposition (DMD) to construct a low-dimensional representation of cancer models and accelerate their simulation. DMD is an unsupervised machine learning method based on the singular value decomposition that has proven useful in many applications as both a predictive and a diagnostic tool. We show that DMD may be applied to Fisher--Kolmogorov models, which constitute an established formulation to represent untreated solid tumor growth that can further accommodate other relevant cancer phenomena (e.g., therapeutic effects, mechanical deformation). Our results show that a DMD implementation of this model over a clinically relevant parameter space can yield promising predictions, with short to medium-term errors remaining under 1% and long-term errors remaining under 20%, despite very short training periods. In particular, we have found that, for moderate to high tumor cell diffusivity and low to moderate tumor cell proliferation rate, DMD reconstructions provide accurate, bounded-error reconstructions for all tested training periods. Additionally, we also show that the three-dimensional DMD reconstruction of the tumor field can be leveraged to accurately reconstruct the displacement fields of the tumor-induced deformation of the host tissue. Thus, we posit the proposed data-driven approach has the potential to greatly reduce the computational overhead of personalized simulations of cancer models, thereby facilitating tumor forecasting, parameter identification, uncertainty quantification, and treatment optimization. [ABSTRACT FROM AUTHOR]

One central challenge in understanding phonon thermal transport is a lack of experimental tools to investigate frequency-resolved phonon transport. Although recent advances in computation lead to frequency-resolved information, it is hindered by unknown defects in bulk regions and at interfaces. Here, a framework that can uncover microscopic phonon transport information in heterostructures is presented, integrating state-of-the-art ultrafast electron diffraction (UED) with advanced scientific machine learning (SciML). Taking advantage of the dual temporal and reciprocal-space resolution in UED, and the ability of SciML to solve inverse problems involving O ( 10 3 ) $\mathcal{O}({10^3})$ coupled Boltzmann transport equations, the frequency-dependent interfacial transmittance and frequency-dependent relaxation times of the heterostructure from the diffraction patterns are reliably recovered. The framework is applied to experimental Au/Si UED data, and a transport pattern beyond the diffuse mismatch model is revealed, which further enables a direct reconstruction of real-space, real-time, frequency-resolved phonon dynamics across the interface. The work provides a new pathway to probe interfacial phonon transport mechanisms with unprecedented details., (© 2022 The Authors. Advanced Materials published by Wiley-VCH GmbH.)

We present a method for solving linear and nonlinear partial differential equations (PDE) based on the variable projection framework and artificial neural networks. For linear PDEs, enforcing the boundary/initial value problem on the collocation points gives rise to a separable nonlinear least squares problem about the network coefficients. We reformulate this problem by the variable projection approach to eliminate the linear output-layer coefficients, leading to a reduced problem about the hidden-layer coefficients only. The reduced problem is solved first by the nonlinear least squares method to determine the hidden-layer coefficients, and then the output-layer coefficients are computed by the linear least squares method. For nonlinear PDEs, enforcing the boundary/initial value problem on the collocation points gives rise to a nonlinear least squares problem that is not separable, which precludes the variable projection strategy for such problems. To enable the variable projection approach for nonlinear PDEs, we first linearize the problem with a Newton iteration, using a particular linearization formulated in terms of the updated approximation field. The linearized system is solved by the variable projection framework together with artificial neural networks. Upon convergence of the Newton iteration, the neural-network coefficients provide the representation of the solution field to the original nonlinear problem. We present ample numerical examples to demonstrate the performance of the method developed herein. For smooth field solutions, the errors of the current method decrease exponentially as the number of collocation points or the number of output-layer coefficients increases. We compare extensively the current method with the extreme learning machine (ELM) method from a previous work. Under identical conditions and configurations, the current method exhibits an accuracy significantly superior to the ELM method. • Method solves PDEs by variable projection and artificial neural networks. • For linear PDEs method solves reduced problem first for hidden-layer parameters, then computes output-layer parameters. • For nonlinear PDEs method combines Newton iteration with variable projection. • Method exhibits exponential convergence for smooth solutions. • Method is significantly superior to ELM in accuracy. [ABSTRACT FROM AUTHOR]

[Display omitted] • A multi-fidelity physics-informed neural network is proposed for multiphysics data. • The accuracy of CFD solver with low-fidelity settings is improved with PINN. • The low-fidelity data accelerate and improve PINN convergence. • Fluid flow, heat transfer, and porous media transport problems are considered. High-fidelity models of multiphysics fluid flow processes are often computationally expensive. On the other hand, less accurate low-fidelity models could be efficiently executed to provide an approximation to the solution. Multi-fidelity approaches combine high-fidelity and low-fidelity data and/or models to obtain a desirable balance between computational efficiency and accuracy. In this manuscript, we propose a multi-fidelity approach where we combine data generated by a low-fidelity computational fluid dynamics (CFD) solution strategy (solver settings and resolution) and data-free physics-informed neural networks (PINN) to obtain improved accuracy. Specifically, transfer learning based on low-fidelity CFD data is used to initialize PINN. Subsequently, PINN with this physics-guided initialization is used to obtain the final results without any high-fidelity training data. The accuracy of the final results relies on the governing equations encoded in PINN together with the low-fidelity CFD data initialization. To investigate the accuracy of this approach, several partial differential equations were solved to predict velocity and temperature in different fluid flow, heat transfer, and porous media transport problems. Comparison with reference high-fidelity CFD data revealed that the proposed approach not only significantly improves the accuracy of low-fidelity CFD data but also improves the convergence speed and accuracy of PINN. [ABSTRACT FROM AUTHOR]

With the capability of accurately representing a functional relationship between the inputs of a physical system's model and output quantities of interest, neural networks have become popular for surrogate modeling in scientific applications. However, as these networks are over-parameterized, their training often requires a large amount of data. To prevent overfitting and improve generalization error, regularization based on, e.g., ℓ 1 - and ℓ 2 -norms of the parameters is applied. Similarly, multiple connections of the network may be pruned to increase sparsity in the network parameters. In this paper, we explore the effects of sparsity promoting ℓ 1 -regularization on training neural networks when only a small training dataset from a high-fidelity model is available. As opposed to standard ℓ 1 -regularization that is known to be inadequate, we consider two variants of ℓ 1 -regularization informed by the parameters of an identical network trained using data from lower-fidelity models of the problem at hand. These bi-fidelity strategies are generalizations of transfer learning of neural networks that uses the parameters learned from a large low-fidelity dataset to efficiently train networks for a small high-fidelity dataset. We also compare the bi-fidelity strategies with two ℓ 1 -regularization methods that only use the high-fidelity dataset but with similar computational cost for generating the training datasets. Three numerical examples for propagating uncertainty through physical systems are used to show that the proposed bi-fidelity ℓ 1 -regularization strategies produce errors that are one order of magnitude smaller than those of networks trained only using datasets from the high-fidelity models. • l1-regularization to train neural network surrogates for uncertainty propagation. • Two bi-fidelity and a single fidelity regularization strategies are proposed. • The bi-fidelity strategies provide one order of magnitude improvements in generalization error. [ABSTRACT FROM AUTHOR]

We propose the use of physics-informed neural networks for solving the shallow-water equations on the sphere in the meteorological context. Physics-informed neural networks are trained to satisfy the differential equations along with the prescribed initial and boundary data, and thus can be seen as an alternative approach to solving differential equations compared to traditional numerical approaches such as finite difference, finite volume or spectral methods. We discuss the training difficulties of physics-informed neural networks for the shallow-water equations on the sphere and propose a simple multi-model approach to tackle test cases of comparatively long time intervals. Here we train a sequence of neural networks instead of a single neural network for the entire integration interval. We also avoid the use of a boundary value loss by encoding the boundary conditions in a custom neural network layer. We illustrate the abilities of the method by solving the most prominent test cases proposed by Williamson et al. (1992) [53]. • PINNs are trained for the shallow-water equations on the sphere. • The method is of low computational complexity and easily parallelizable. • Benchmarks for the shallow-water equations on the sphere are successfully reproduced. • The proposed methodology is completely meshless and hence avoids grid imprinting. [ABSTRACT FROM AUTHOR]

• A parameterized exterior calculus is introduced for learning physics. • Analysis provides guarantees for learning well-posed nonlinear elliptic operators. • Exact treatment of conservation and characterization of nontrivial null-spaces. • Novel training strategy guarantees exact physics independent of available data. As traditional machine learning tools are increasingly applied to science and engineering applications, physics-informed methods have emerged as effective tools for endowing inferences with properties essential for physical realizability. While promising, these methods generally enforce physics weakly via penalization. To enforce physics strongly, we turn to the exterior calculus framework underpinning combinatorial Hodge theory and physics-compatible discretization of partial differential equations (PDEs). Historically, these two fields have remained largely distinct, as graphs are strictly topological objects lacking the metric information fundamental to PDE discretization. We present an approach where this missing metric information may be learned from data, using graphs as coarse-grained mesh surrogates that inherit desirable conservation and exact sequence structure from the combinatorial Hodge theory. The resulting data-driven exterior calculus (DDEC) may be used to extract structure-preserving surrogate models with mathematical guarantees of well-posedness. The approach admits a PDE-constrained optimization training strategy which guarantees machine-learned models enforce physics to machine precision, even for poorly trained models or small data regimes. We provide analysis of the method for a class of models designed to reproduce nonlinear perturbations of elliptic problems and provide examples of learning H (d i v) / H (c u r l) systems representative of subsurface flows and electromagnetics. [ABSTRACT FROM AUTHOR]

Neural operators can learn nonlinear mappings between function spaces and offer a new simulation paradigm for real-time prediction of complex dynamics for realistic diverse applications as well as for system identification in science and engineering. Herein, we investigate the performance of two neural operators, which have shown promising results so far, and we develop new practical extensions that will make them more accurate and robust and importantly more suitable for industrial-complexity applications. The first neural operator, DeepONet , was published in 2019 (Lu et al., 2019), and its original architecture was based on the universal approximation theorem of Chen & Chen (1995). The second one, named Fourier Neural Operator or FNO , was published in 2020 (Li et al., 2020), and it is based on parameterizing the integral kernel in the Fourier space. DeepONet is represented by a summation of products of neural networks (NNs), corresponding to the branch NN for the input function and the trunk NN for the output function; both NNs are general architectures, e.g., the branch NN can be replaced with a CNN or a ResNet. According to Kovachki et al. (2021), FNO in its continuous form can be viewed conceptually as a DeepONet with a specific architecture of the branch NN and a trunk NN represented by a trigonometric basis. In order to compare FNO with DeepONet computationally for realistic setups, we develop several extensions of FNO that can deal with complex geometric domains as well as mappings where the input and output function spaces are of different dimensions. We also develop an extended DeepONet with special features that provide inductive bias and accelerate training, and we present a faster implementation of DeepONet with cost comparable to the computational cost of FNO, which is based on the Fast Fourier Transform. We consider 16 different benchmarks to demonstrate the relative performance of the two neural operators, including instability wave analysis in hypersonic boundary layers, prediction of the vorticity field of a flapping airfoil, porous media simulations in complex-geometry domains, etc. We follow the guiding principles of FAIR (Findability, Accessibility, Interoperability, and Reusability) for scientific data management and stewardship. The performance of DeepONet and FNO is comparable for relatively simple settings, but for complex geometries the performance of FNO deteriorates greatly. We also compare theoretically the two neural operators and obtain similar error estimates for DeepONet and FNO under the same regularity assumptions. • We develop new practical extensions of DeepONet and FNO that make them more accurate and robust. • We consider 16 different benchmarks to demonstrate the performance of DeepONet and FNO. • We obtain theoretical error estimates of DeepONet and FNO. [ABSTRACT FROM AUTHOR]

Dynamic Mode Decomposition (DMD) is an unsupervised machine learning method that has attracted considerable attention in recent years owing to its equation-free structure, ability to easily identify coherent spatio-temporal structures in data, and effectiveness in providing reasonably accurate predictions for certain problems, particularly over short-to-medium time frames. Despite these successes, the application of DMD to certain problems featuring highly nonlinear transient dynamics remains challenging. In such cases, DMD may not only fail to provide acceptable predictions but may indeed fail to recreate the data in which it was trained, restricting its application to diagnostic purposes (i.e., feature identification and extraction). For many such problems in the biological and physical sciences, the structure of the system obeys a compartmental framework, in which the transfer of mass, energy, or some other quantity of interest within the system moves across states. In these cases, the behavior of the system may not be accurately recreated by applying DMD to a single quantity within the system, as proper knowledge of the system dynamics, even for a single compartment, requires that the behavior of other compartments is taken into account in the DMD process. In the present work, we demonstrate, theoretically and numerically, that, when performing DMD on a fully coupled PDE system with compartmental structure, one may recover useful predictive behavior, even when DMD performs poorly when acting compartment-wise. We also establish that important physical quantities, such as mass conservation, are maintained in the coupled-DMD extrapolation. The mathematical and numerical analysis suggests that DMD, properly applied, may be a powerful tool for this common class of problems In particular, we show interesting numerical applications to a continuous delayed-SIRD model for Covid-19, and to a problem from additive manufacturing considering a nonlinear temperature field and the resulting change of material phase from powder, liquid, and solid states. [ABSTRACT FROM AUTHOR]

Physics-informed neural networks (PINNs) are increasingly employed to replace/augment traditional numerical methods in solving partial differential equations (PDEs). While state-of-the-art PINNs have many attractive features, they approximate a specific realization of a PDE system and hence are problem-specific. That is, the model needs to be re-trained each time the boundary conditions (BCs) and domain shape/size change. This limitation prohibits the application of PINNs to realistic or large-scale engineering problems especially since the costs and efforts associated with their training are considerable. We introduce a transferable framework for solving boundary value problems (BVPs) via deep neural networks which can be trained once and used forever for various unseen domains and BCs. We first introduce genomic flow network (GFNet), a neural network that can infer the solution of a BVP across arbitrary BCs on a small square domain called genome. Then, we propose mosaic flow (MF) predictor, a novel iterative algorithm that assembles the GFNet's inferences for BVPs on large domains with unseen sizes/shapes and BCs while preserving the spatial regularity of the solution. We demonstrate that our framework can estimate the solution of Laplace and Navier–Stokes equations in domains of unseen shapes and BCs that are, respectively, 1200 and 12 times larger than the training domains. Since our framework eliminates the need to re-train models for unseen domains and BCs, it demonstrates up to 3 orders-of-magnitude speedups compared to the state-of-the-art. • Mosaic flow (MF) solves boundary value problems via deep neural networks. • MF can be trained once and used forever for unseen domains and boundary conditions. • We present an in-depth error analysis of MF. • MF can enforce the exact boundary condition in learning PDEs solutions. • MF is faster and more accurate compared to the state-of-the-art models in solving PDEs. [ABSTRACT FROM AUTHOR]

• We analyze the training dynamics of PINNs using neural tangent kernel theory. • We derive the NTK of PINNs and study its limiting behavior. • Our analysis reveals a remarkable discrepancy in the convergence rate of PINNs loss functions. • We propose a novel NTK-guided gradient descent algorithm to effectively resolve this pathology. • We demonstrate significant and consistent accuracy improvements across a range of benchmarks. Physics-informed neural networks (PINNs) have lately received great attention thanks to their flexibility in tackling a wide range of forward and inverse problems involving partial differential equations. However, despite their noticeable empirical success, little is known about how such constrained neural networks behave during their training via gradient descent. More importantly, even less is known about why such models sometimes fail to train at all. In this work, we aim to investigate these questions through the lens of the Neural Tangent Kernel (NTK); a kernel that captures the behavior of fully-connected neural networks in the infinite width limit during training via gradient descent. Specifically, we derive the NTK of PINNs and prove that, under appropriate conditions, it converges to a deterministic kernel that stays constant during training in the infinite-width limit. This allows us to analyze the training dynamics of PINNs through the lens of their limiting NTK and find a remarkable discrepancy in the convergence rate of the different loss components contributing to the total training error. To address this fundamental pathology, we propose a novel gradient descent algorithm that utilizes the eigenvalues of the NTK to adaptively calibrate the convergence rate of the total training error. Finally, we perform a series of numerical experiments to verify the correctness of our theory and the practical effectiveness of the proposed algorithms. The data and code accompanying this manuscript are publicly available at https://github.com/PredictiveIntelligenceLab/PINNsNTK. [ABSTRACT FROM AUTHOR]

We develop a framework for Gaussian processes regression constrained by boundary value problems. The framework may be applied to infer the solution of a well-posed boundary value problem with a known second-order differential operator and boundary conditions, but for which only scattered observations of the source term are available. Scattered observations of the solution may also be used in the regression. The framework combines co-kriging with the linear transformation of a Gaussian process together with the use of kernels given by spectral expansions in eigenfunctions of the boundary value problem. Thus, it benefits from a reduced-rank property of covariance matrices. We demonstrate that the resulting framework yields more accurate and stable solution inference as compared to physics-informed Gaussian process regression without boundary condition constraints. • Combine boundary condition constraints with PDE co-kriging constraints in GPR. • First demonstration of GPR constrained by Neumann boundary conditions. • Can infer the solution from scattered observations of the source term alone. [ABSTRACT FROM AUTHOR]

• Construction and implementation of new domain-decomposition based parallel algorithm is proposed for cPINNs and XPINNs methods. • The proposed algorithm adds another dimension of parallelism in SciML primarily driven by data and model parallelism. • The proposed algorithm is shown its scaling for CPU and CPU+GPU architectures. We develop a distributed framework for the physics-informed neural networks (PINNs) based on two recent extensions, namely conservative PINNs (cPINNs) and extended PINNs (XPINNs), which employ domain decomposition in space and in time-space, respectively. This domain decomposition endows cPINNs and XPINNs with several advantages over the vanilla PINNs, such as parallelization capacity, large representation capacity, efficient hyperparameter tuning, and is particularly effective for multi-scale and multi-physics problems. Here, we present a parallel algorithm for cPINNs and XPINNs constructed with a hybrid programming model described by MPI + X, where X ∈ { CPUs , GPUs }. The main advantage of cPINN and XPINN over the more classical data and model parallel approaches is the flexibility of optimizing all hyperparameters of each neural network separately in each subdomain. We compare the performance of distributed cPINNs and XPINNs for various forward problems, using both weak and strong scalings. Our results indicate that for space domain decomposition, cPINNs are more efficient in terms of communication cost but XPINNs provide greater flexibility as they can also handle time-domain decomposition for any differential equations, and can deal with any arbitrarily shaped complex subdomains. To this end, we also present an application of the parallel XPINN method for solving an inverse diffusion problem with variable conductivity on the United States map, using ten regions as subdomains. [ABSTRACT FROM AUTHOR]

There is a compelling need for the regression capability of mapping the initial field and applied conditions to the evolved field, e.g., given current flow field and fluid properties predicting next-step flow field. Such a capability can provide a maximum to full substitute of a physics-based model, enabling fast simulation of various field evolvements. We propose a conceptually simple, lightweight, but powerful multi-input convolutional network (ConvNet), yNet, that merges multi-input signals by manipulating high-level encodings of field/image input. yNet can significantly reduce the model size compared with its ConvNet counterpart (e.g., to only one-tenth for main architecture of 38-layer depth) and is as much as six orders of magnitude faster than a physics-based model. yNet is applied for data-driven modeling of fluid dynamics, porosity evolution in sintering, stress field development, and grain growth. It consistently shows great extrapolative prediction beyond training datasets in terms of temporal ranges, spatial domains, and geometrical shapes., Competing Interests: The authors declare no competing interests., (© 2022 The Authors.)

Lithium-ion batteries are commonly used to power unmanned aircraft vehicles (UAVs). The ability to model and forecast remaining useful life of these batteries enables UAV reliability assurance. Building principled accurate models is challenging due to the complex electrochemistry that governs battery operation. Alternatively, reduced order models have the advantage of capturing the overall behavior of battery discharge, although they suffer from simplifications and residual discrepancy. This paper presents a hybrid modeling approach that directly implements physics within deep neural networks. While most of the input–output relationship is captured by reduced-order models, data-driven kernels reduce the gap between predictions and observations. A reduced-order model based on Nernst and Butler–Volmer equations represents the overall battery discharge, and a multilayer perceptron models the battery non-ideal voltage. Battery aging is characterized by time-dependent internal resistance and the amount of available Li-ions, which are modeled through an ensemble of variational Bayesian multilayer perceptrons. The approach is validated using data publicly available through the NASA Prognostics Center of Excellence website. Results showed that our hybrid battery prognosis model can be successfully calibrated, even with a limited number of observations. Moreover, the model can help optimizing battery operation by offering long-term forecast of battery capacity. • Physics-informed neural network implements physics into a deep learning framework. • Model capable of forecasting end-of-discharge while accounting for battery aging. • Aging prognostic uses information from a fleet of batteries via an ensemble model. [ABSTRACT FROM AUTHOR]

• Method (modBIP) presented to pre-train random coefficients of extreme learning machines (ELM). • modBIP can work with any activation function and is more accurate than the BIP method. • ELM/modBIP is highly accurate and accuracy is insensitive to initial random coefficients. In extreme learning machines (ELM) the hidden-layer coefficients are randomly set and fixed, while the output-layer coefficients of the neural network are computed by a least squares method. The randomly-assigned coefficients in ELM are known to influence its performance and accuracy significantly. In this paper we present a modified batch intrinsic plasticity (modBIP) method for pre-training the random coefficients in the ELM neural networks. The current method is devised based on the same principle as the batch intrinsic plasticity (BIP) method, namely, by enhancing the information transmission in every node of the neural network. It differs from BIP in two prominent aspects. First, modBIP does not involve the activation function in its algorithm, and it can be applied with any activation function in the neural network. In contrast, BIP employs the inverse of the activation function in its construction, and requires the activation function to be invertible (or monotonic). The modBIP method can work with the often-used non-monotonic activation functions (e.g. Gaussian, swish, Gaussian error linear unit, and radial-basis type functions), with which BIP breaks down. Second, modBIP generates target samples on random intervals with a minimum size, which leads to highly accurate computation results when combined with ELM. The combined ELM/modBIP method is markedly more accurate than ELM/BIP in numerical simulations. Ample numerical experiments are presented with shallow and deep neural networks for function approximation and boundary/initial value problems with partial differential equations. They demonstrate that the combined ELM/modBIP method produces highly accurate simulation results, and that its accuracy is insensitive to the random-coefficient initializations in the neural network. This is in sharp contrast with the ELM results without pre-training of the random coefficients. [ABSTRACT FROM AUTHOR]

Physics-informed neural networks (PINNs) are demonstrating remarkable promise in integrating physical models with gappy and noisy observational data, but they still struggle in cases where the target functions to be approximated exhibit high-frequency or multi-scale features. In this work we investigate this limitation through the lens of Neural Tangent Kernel (NTK) theory and elucidate how PINNs are biased towards learning functions along the dominant eigen-directions of their limiting NTK. Using this observation, we construct novel architectures that employ spatio-temporal and multi-scale random Fourier features, and justify how such coordinate embedding layers can lead to robust and accurate PINN models. Numerical examples are presented for several challenging cases where conventional PINN models fail, including wave propagation and reaction–diffusion dynamics, illustrating how the proposed methods can be used to effectively tackle both forward and inverse problems involving partial differential equations with multi-scale behavior. All code an data accompanying this manuscript will be made publicly available at https://github.com/PredictiveIntelligenceLab/MultiscalePINNs. • We argue that spectral bias in deep neural networks in fact corresponds to "NTK eigenvector bias". • We show that Fourier feature mappings can modulate the frequency of the NTK eigenvectors. • By analyzing the NTK eigenspace, we engineer new effective architectures for multi-scale problems. • We put forth a collection of challenging benchmarks for PINNs. [ABSTRACT FROM AUTHOR]

This paper exploits a new direction of structural identification by means of Neural Ordinary Differential Equations (Neural ODEs), particularly constrained by domain knowledge, such as structural dynamics, thus forming Physics-informed Neural ODEs, aiming at governing equations discovery/approximation. Structural identification problems often entail complex setups featuring high-dimensionality, or stiff ODEs, which pose difficulties in the training and learning of conventional data-driven algorithms who seek to unveil the governing dynamics of a system of interest. In this work, Neural ODEs are re-casted as a two-level representation involving a physics-informed term, that stems from possible prior knowledge of a dynamical system, and a discrepancy term, captured by means of a feed-forward neural network. The re-casted format is highly adaptive and flexible to structural monitoring problems, such as linear/nonlinear structural identification, model updating, structural damage detection, driving force identification, etc. As an added step, for inferring an explainable model, we propose the adoption of sparse identification of nonlinear dynamical systems as an additional tool to distill closed-form expressions for the trained nets, that embed a more straightforward engineering interpretation. We demonstrate the framework on a series of numerical and experimental examples, with the latter pertaining to a structural system featuring highly nonlinear behavior, which is successfully learned by the proposed framework. The proposed structural identification with Physics-informed Neural ODEs comes with the benefits of direct approximation of the governing dynamics, and a versatile and flexible framework for discrepancy modeling in structural identification problems. [ABSTRACT FROM AUTHOR]

• We present a novel hybrid framework that enables discovery of unknown fields in inverse partial differential problems. • We implement trainable finite discretization solver layers that are composable with pre-existing neural layers. • The network can be pre-trained in a self-supervised fashion and used on unseen data without further training. • This framework enables consideration of domain specific knowledge about the unknown fields. • In contrast to constrained optimization methods, the loss function is simply difference between data and prediction. We propose a novel composite framework to find unknown fields in the context of inverse problems for partial differential equations (PDEs). We blend the high expressibility of deep neural networks as universal function estimators with the accuracy and reliability of existing numerical algorithms for partial differential equations as custom layers in semantic autoencoders. Our design brings together techniques of computational mathematics, machine learning and pattern recognition under one umbrella to incorporate domain-specific knowledge and physical constraints to discover the underlying hidden fields. The network is explicitly aware of the governing physics through a hard-coded PDE solver layer in contrast to most existing methods that incorporate the governing equations in the loss function or rely on trainable convolutional layers to discover proper discretizations from data. This subsequently focuses the computational load to only the discovery of the hidden fields and therefore is more data efficient. We call this architecture Blended inverse-PDE networks (hereby dubbed BiPDE networks) and demonstrate its applicability for recovering the variable diffusion coefficient in Poisson problems in one and two spatial dimensions, as well as the diffusion coefficient in the time-dependent and nonlinear Burgers' equation in one dimension. We also show that the learned hidden parameters are robust to added noise on input data. [ABSTRACT FROM AUTHOR]

This paper presents a deep learning framework for epidemiology system identification from noisy and sparse observations with quantified uncertainty. The proposed approach employs an ensemble of deep neural networks to infer the time-dependent reproduction number of an infectious disease by formulating a tensor-based multi-step loss function that allows us to efficiently calibrate the model on multiple observed trajectories. The method is applied to a mobility and social behavior-based SEIR model of COVID-19 spread. The model is trained on Google and Unacast mobility data spanning a period of 66 days, and is able to yield accurate future forecasts of COVID-19 spread in 203 US counties within a time-window of 15 days. Interestingly, a sensitivity analysis that assesses the importance of different mobility and social behavior parameters reveals that attendance of close places, including workplaces, residential, and retail and recreational locations, has the largest impact on the effective reproduction number. The model enables us to rapidly probe and quantify the effects of government interventions, such as lock-down and re-opening strategies. Taken together, the proposed framework provides a robust workflow for data-driven epidemiology model discovery under uncertainty and produces probabilistic forecasts for the evolution of a pandemic that can judiciously provide information for policy and decision making. All codes and data accompanying this manuscript are available at https://github.com/PredictiveIntelligenceLab/DeepCOVID19. • We present a novel deep learning framework for epidemiology system identification from incomplete observations. • We study the sensitivity of COVID-19 dynamics on human mobility and social behavior. • We assess the effectiveness of government interventions, such as lockdown and re-opening strategies. • We produce probabilistic forecasts for the evolution of a pandemic that can provide information for policy making. [ABSTRACT FROM AUTHOR]

We present the results of a study where we use machine learning to enhance hohlraum design for opacity measurement experiments. Opacity experiments on laser facilities use hohlraums, which, when their interior walls are illuminated by the National Ignition Facility (NIF) lasers, produce a high radiation flux that heats a central sample to a temperature that is constant over a measurement time window. Given a baseline hohlraum design and a computational model, we train a deep neural network to predict the time evolution of the radiation temperature as measured by the Dante diagnostic. This enables us to rapidly explore design space and determine the effect of adjusting design parameters. We also construct an "inverse" machine learning model that predicts the design parameters given a desired time history of radiation temperature. Calculations using the machine learning model demonstrate that improved performance over the baseline hohlraum could reduce sensitivities and uncertainties in experimental opacity measurements. • We demonstrate that machine learning models can be used to augment simulation to design opacity experiments. • Forward machine learning models allow design space to be explored. • We develop a machine learning model to learn the inverse mapping from diagnostics to experiment design. • Simulations indicate that measurement uncertainties can be improved using our models. [ABSTRACT FROM AUTHOR]

• Novel methodology for combining physics-based models with machine learning models. • Physics capture trends while data-driven models quantify model inadequacy. • Implementation of ordinary differential equations based on recurrent neural networks. A number of physical systems can be described by ordinary differential equations. When physics is well understood, the time dependent responses are easily obtained numerically. The particular numerical method used for integration depends on the application. Unfortunately, when physics is not fully understood, the discrepancies between predictions and observed responses can be large and unacceptable. In this paper, we propose an approach that uses observed data to estimate the missing physics in the original model (i.e., model-form uncertainty). In our approach, we first design recurrent neural networks to perform numerical integration of the ordinary differential equations. Then, we implement the recurrent neural network as a directed graph. This way, the nodes in the graph represent the physics-informed kernels found in the ordinary differential equations. We quantify the missing physics by carefully introducing data-driven in the directed graph. This allows us to estimate the missing physics (discrepancy term) even for hidden nodes of the graph. We studied the performance of our proposed approach with the aid of three case studies (fatigue crack growth, corrosion-fatigue crack growth, and bearing fatigue) and state-of-the-art machine learning software packages. Our results demonstrate the ability to perform estimation of discrepancy, reducing gap between predictions and observations, at reasonable computational cost. [ABSTRACT FROM AUTHOR]

• We propose a unified computational framework to tackle different types of free boundary problems. • We demonstrate the effectiveness of this framework in solving direct and inverse Stefan problems. • We propose and solve a new type of data-driven inverse Stefan problems. Free boundary problems appear naturally in numerous areas of mathematics, science and engineering. These problems present a great computational challenge because they necessitate numerical methods that can yield an accurate approximation of free boundaries and complex dynamic interfaces. In this work, we propose a multi-network model based on physics-informed neural networks to tackle a general class of forward and inverse free boundary problems called Stefan problems. Specifically, we approximate the unknown solution as well as any moving boundaries by two deep neural networks. Besides, we formulate a new type of inverse Stefan problems that aim to reconstruct the solution and free boundaries directly from sparse and noisy measurements. We demonstrate the effectiveness of our approach in a series of benchmarks spanning different types of Stefan problems, and illustrate how the proposed framework can accurately recover solutions of partial differential equations with moving boundaries and dynamic interfaces. All code and data accompanying this manuscript are publicly available at https://github.com/PredictiveIntelligenceLab/DeepStefan. [ABSTRACT FROM AUTHOR]

• A method for leveraging known physics, expressed in a PDE, to learn closures for missing physics. • Optimization of combined PDE-ML system with stochastic gradient descent, utilizing adjoint PDEs. • Complete demonstration for sub-filter-scale closure for large-eddy simulation (LES) of turbulence. • Outperforms common LES models. • Outperforms direct a priori learning of simple mismatch. A framework is introduced that leverages known physics to reduce overfitting in machine learning for scientific applications. The partial differential equation (PDE) that expresses the physics is augmented with a neural network that uses available data to learn a description of the corresponding unknown or unrepresented physics. Training within this combined system corrects for missing, unknown, or erroneously represented physics, including discretization errors associated with the PDE's numerical solution. For optimization of the network within the PDE, an adjoint PDE is solved to provide high-dimensional gradients, and a stochastic adjoint method (SAM) further accelerates training. The approach is demonstrated for large-eddy simulation (LES) of turbulence. High-fidelity direct numerical simulations (DNS) of decaying isotropic turbulence provide the training data used to learn sub-filter-scale closures for the filtered Navier–Stokes equations. Out-of-sample comparisons show that the deep learning PDE method outperforms widely-used models, even for filter sizes so large that they become qualitatively incorrect. It also significantly outperforms the same neural network when a priori trained based on simple data mismatch, not accounting for the full PDE. Measures of discretization errors, which are well-known to be consequential in LES, point to the importance of the unified training formulation's design, which without modification corrects for them. For comparable accuracy, simulation runtime is significantly reduced. A relaxation of the typical discrete enforcement of the divergence-free constraint in the solver is also successful, instead allowing the DPM to approximately enforce incompressibility physics. Since the training loss function is not restricted to correspond directly to the closure to be learned, training can incorporate diverse data, including experimental data. [ABSTRACT FROM AUTHOR]

This work presents a non-intrusive model reduction method to learn low-dimensional models of dynamical systems with non-polynomial nonlinear terms that are spatially local and that are given in analytic form. In contrast to state-of-the-art model reduction methods that are intrusive and thus require full knowledge of the governing equations and the operators of a full model of the discretized dynamical system, the proposed approach requires only the non-polynomial terms in analytic form and learns the rest of the dynamics from snapshots computed with a potentially black-box full-model solver. The proposed method learns operators for the linear and polynomially nonlinear dynamics via a least-squares problem, where the given non-polynomial terms are incorporated on the right-hand side. The least-squares problem is linear and thus can be solved efficiently in practice. The proposed method is demonstrated on three problems governed by partial differential equations, namely the diffusion–reaction Chafee–Infante model, a tubular reactor model for reactive flows, and a batch-chromatography model that describes a chemical separation process. The numerical results provide evidence that the proposed approach learns reduced models that achieve comparable accuracy as models constructed with state-of-the-art intrusive model reduction methods that require full knowledge of the governing equations. • Method learns low-dimensional models of dynamical systems with non-polynomial nonlinear terms. • Requires non-polynomial terms analytically; learns the other dynamics from snapshots. • Can achieve comparable accuracy as state-of-the-art intrusive model reduction methods on numerical examples. [ABSTRACT FROM AUTHOR]

We present a tutorial on how to directly implement integration of ordinary differential equations through recurrent neural networks using Python. In order to simplify the implementation, we leveraged modern machine learning frameworks such as TensorFlow and Keras. Besides, offering implementation of basic models (such as multilayer perceptrons and recurrent neural networks) and optimization methods, these frameworks offer powerful automatic differentiation. With all that, the main advantage of our approach is that one can implement hybrid models combining physics-informed and data-driven kernels, where data-driven kernels are used to reduce the gap between predictions and observations. Alternatively, we can also perform model parameter identification. In order to illustrate our approach, we used two case studies. The first one consisted of performing fatigue crack growth integration through Euler's forward method using a hybrid model combining a data-driven stress intensity range model with a physics-based crack length increment model. The second case study consisted of performing model parameter identification of a dynamic two-degree-of-freedom system through Runge–Kutta integration. The examples presented here as well as source codes are all open-source under the GitHub repository https://github.com/PML-UCF/pinn_code_tutorial. • Numerical integration of ordinary differential equations combining physics-based models with machine learning. • Implementation in Python and leverages state-of-the art machine learning frameworks. • Source and data readily available at https://github.com/PML-UCF/pinn_code_tutorial. [ABSTRACT FROM AUTHOR]

We present Lift & Learn , a physics-informed method for learning low-dimensional models for large-scale dynamical systems. The method exploits knowledge of a system's governing equations to identify a coordinate transformation in which the system dynamics have quadratic structure. This transformation is called a lifting map because it often adds auxiliary variables to the system state. The lifting map is applied to data obtained by evaluating a model for the original nonlinear system. This lifted data is projected onto its leading principal components, and low-dimensional linear and quadratic matrix operators are fit to the lifted reduced data using a least-squares operator inference procedure. Analysis of our method shows that the Lift & Learn models are able to capture the system physics in the lifted coordinates at least as accurately as traditional intrusive model reduction approaches. This preservation of system physics makes the Lift & Learn models robust to changes in inputs. Numerical experiments on the FitzHugh–Nagumo neuron activation model and the compressible Euler equations demonstrate the generalizability of our model. • Lift & Learn: A new physics-informed machine learning method for nonlinear PDEs. • Lifting transformations expose quadratic structure that is exploited for learning. • Lift & Learn models generalize as well as intrusive projection-based reduced models. [ABSTRACT FROM AUTHOR]

• Scalable sensitivity-based model reduction method for high-dimensional reaction networks. • Applicable to stiff and non-linear models. • The information loss of the reduced model is controlled. • No biochemical knowledge is required. • Intuitive and automatizable. In this work we present new scalable, information theory-based variational methods for the efficient model reduction of high-dimensional deterministic and stochastic reaction networks. The proposed methodology combines, (a) information theoretic tools for sensitivity analysis that allow us to identify the proper coarse variables of the reaction network, with (b) variational approximate inference methods for training a best-fit reduced model. This approach takes advantage of both physicochemical modeling and data-based approaches and allows to construct optimal parameterized reduced dynamics in the number of variables, reactions and parameters, while controlling the information loss due to the reduction. We demonstrate the effectiveness of our model reduction method on several complex, high-dimensional chemical reaction networks arising in biochemistry. [ABSTRACT FROM AUTHOR]