Search

In this paper, we are interested in the acceleration of numerical simulations. We focus on a hypersonic planetary reentry problem whose simulation involves coupling fluid dynamics and chemical reactions. Simulating chemical reactions takes most of the computational time but, on the other hand, cannot be avoided to obtain accurate predictions. We face a trade-off between cost-efficiency and accuracy: the numerical scheme has to be sufficiently efficient to be used in an operational context but accurate enough to predict the phenomenon faithfully. To tackle this trade-off, we design a hybrid numerical scheme coupling a traditional fluid dynamic solver with a neural network approximating the chemical reactions. We rely on their power in terms of accuracy and dimension reduction when applied in a big data context and on their efficiency stemming from their matrix-vector structure to achieve important acceleration factors (×10 to ×18.6). This paper aims to explain how we design such cost-effective hybrid numerical schemes in practice. Above all, we describe methodologies to ensure accuracy guarantees, allowing us to go beyond traditional surrogate modeling and to use these schemes as references. • Deep Learning-based hybridization speeds up numerical schemes of atmospheric reentry while maintaining high accuracy. • Initializing a scheme with a hybrid code's prediction reduces the convergence time and keeps the exact same guarantees. • Uncertainty analysis provides statistical guarantees concerning approximation errors when using hybridization code. • Neural network approximation error is statistically lower than many other sources of error inherent to numerical simulations. [ABSTRACT FROM AUTHOR]

• A novel physics-informed convolutional neural network f-PICNN for PDEs without any labelled data. • Nonlinear convolutional units (NCUs). • Memory mechanism (numerical results show it can considerably speed up the convergence). • Finite discretization schemes (e.g., Euler, Crank Nicolson and fourth-order Runge-Kutta). • Auto-regressive framework. The physics and interdisciplinary problems in science and engineering are mainly described as partial differential equations (PDEs). Recently, a novel method using physics-informed neural networks (PINNs) to solve PDEs by employing deep neural networks with physical constraints as data-driven models has been pioneered for surrogate modelling and inverse problems. However, the original PINNs based on fully connected neural networks pose intrinsic limitations and poor performance for the PDEs with nonlinearity, drastic gradients, multiscale characteristics or high dimensionality in which the complex features are hard to capture. This leads to difficulties in convergence to correct solutions and high computational costs. To address the above problems, in this paper, a novel physics-informed convolutional neural network framework based on finite discretization schemes with a stack of a series of nonlinear convolutional units (NCUs) for solving PDEs in the space-time domain without any labelled data (f-PICNN) is proposed, in which the memory mechanism can considerably speed up the convergence. Specifically, the initial conditions (ICs) are hard-encoded into the network as the first time-step solution and used to extrapolate the next time-step solution. The Dirichlet boundary conditions (BCs) are constrained by soft BC enforcement while the Neumann BCs are hard enforced. Furthermore, the loss function is designed as a set of discretized PDE residuals and optimized to conform to physics laws. Finally, the proposed auto-regressive model has been proven to be effective in a wide range of 1D and 2D nonlinear PDEs in both space and time under different finite discretization schemes (e.g., Euler, Crank Nicolson and fourth-order Runge-Kutta). The numerical results demonstrate that the proposed framework not only shows the ability to learn the PDEs efficiently but also provides an opportunity for greater conceptual simplicity, and potential for extrapolation from learning the PDEs using a limited dataset. [ABSTRACT FROM AUTHOR]

We propose, in this paper, a Variable Spiking Wavelet Neural Operator (VS-WNO), which aims to bridge the gap between theoretical and practical implementation of Artificial Intelligence (AI) algorithms for mechanics applications. With recent developments like the introduction of neural operators, AI's potential for being used in mechanics applications has increased significantly. However, AI's immense energy and resource requirements are a hurdle in its practical field use case. The proposed VS-WNO is based on the principles of spiking neural networks, which have shown promise in reducing the energy requirements of the neural networks. This makes possible the use of such algorithms in edge computing. The proposed VS-WNO utilizes variable spiking neurons, which promote sparse communication, thus conserving energy, and its use is further supported by its ability to tackle regression tasks, often faced in the field of mechanics. Various examples dealing with partial differential equations, like Burger's equation, Allen Cahn's equation, and Darcy's equation, have been shown. Comparisons have been shown against wavelet neural operator utilizing leaky integrate and fire neurons (direct and encoded inputs) and vanilla wavelet neural operator utilizing artificial neurons. The results produced illustrate the ability of the proposed VS-WNO to converge to ground truth while promoting sparse communication. • Neuroscience inspired operator learning is proposed for scientific computing. • Proposed VS-WNO promotes sparse communications and is energy efficient. • We introduce a tailored spiking loss function to limit spiking activity. • Numerical examples solved illustrate accuracy and energy efficiency of VS-WNO. [ABSTRACT FROM AUTHOR]

• Deep learning accelerates vapor liquid equilibrium prediction with verified accuracy. • Comprehensive comparison between deep learning and various flash calculation methods. • Optimized deep learning model for estimating VLE properties. • Understandable introduction of deep learning and flash calculation for general audience. • Remarks on further applications of deep learning in classical engineering problems. In the past two decades, researchers have made remarkable progress in accelerating flash calculation, which is very useful in a variety of engineering processes. In this paper, general phase splitting problem statements and flash calculation procedures using the Successive Substitution Method are reviewed, while the main shortages are pointed out. Two acceleration methods, Newton's method and the Sparse Grids Method are presented afterwards as a comparison with the deep learning model proposed in this paper. A detailed introduction from artificial neural networks to deep learning methods is provided here with the authors' own remarks. Factors in the deep learning model are investigated to show their effect on the final result. A selected model based on that has been used in a flash calculation predictor with comparison with other methods mentioned above. It is shown that results from the optimized deep learning model meet the experimental data well with the shortest CPU time. More comparison with experimental data has been conducted to show the robustness of our model. [ABSTRACT FROM AUTHOR]

We present a numerical framework for learning unknown stochastic dynamical systems using measurement data. Termed stochastic flow map learning (sFML), the new framework is an extension of flow map learning (FML) that was developed for learning deterministic dynamical systems. For learning stochastic systems, we define a stochastic flow map that is a superposition of two sub-flow maps: a deterministic sub-map and a stochastic sub-map. The stochastic training data are used to construct the deterministic sub-map first, followed by the stochastic sub-map. The deterministic sub-map takes the form of residual network (ResNet), similar to the work of FML for deterministic systems. For the stochastic sub-map, we employ a generative model, particularly generative adversarial networks (GANs) in this paper. The final constructed stochastic flow map then defines a stochastic evolution model that is a weak approximation, in term of distribution, of the unknown stochastic system. A comprehensive set of numerical examples are presented to demonstrate the flexibility and effectiveness of the proposed sFML method for various types of stochastic systems. • Proposed a new numerical method for modeling SDEs with observation data. • Established a mathematical framework that utilizes stochastic flow map for effective modeling. • Developed novel stochastic flow map modeling and its DNN structure. [ABSTRACT FROM AUTHOR]

Fourier analysis based optimization techniques have been developed for numerical schemes on structured grids and result in significant performance improvements. However, there lacks an effective optimization technique for numerical schemes on general unstructured grids, due to the mesh geometry complexity that makes Fourier transformation infeasible. To address this issue, an optimization framework based on machine learning is developed in this paper for numerical schemes on unstructured grids. The optimization of a compact high-order variational reconstruction on triangular grids is performed to illustrate the framework. An artificial neural network (ANN) is used to predict the optimal values of the derivative weights on cell interfaces that are the free parameters of the variational reconstruction, given the local geometry as input. The ANN is trained by minimizing the solution errors of a linear advection equation on a set of generated compact stencils, with a set of sampled sine waves as initial conditions. The developed optimization framework is applicable to general numerical schemes on unstructured grids as the training does not require explicit computation of dispersion or dissipation. Numerical results for inviscid flow problems show that the optimized variational finite volume scheme is remarkably more accurate and efficient than its non-optimized counterpart, demonstrating the effectiveness of the machine learning optimization. • A machine learning optimization framework for numerical schemes on unstructured grids. • A feedforward neural network to predict optimal parameter values of numerical schemes. • Training of the network by minimizing solution errors of a linear advection equation. • A machine learning optimized compact high-order variational reconstruction. • Generalization of the optimized variational reconstruction to nonlinear problems. [ABSTRACT FROM AUTHOR]

The Allen-Cahn equation is a well-known stiff semilinear parabolic equation used to describe the process of phase separation and transition in phase field modeling of multi-component physical systems, while the conservative Allen-Cahn equation is a modified version of the classic Allen-Cahn equation that can additionally conserve the mass. As neural networks and deep learning techniques have achieved significant successes in recent years in scientific and engineering applications, there has been growing interest in developing deep learning algorithms for numerical solutions of partial differential equations. In this paper, we propose a novel deep learning method for predicting the dynamics of the classic and conservative Allen-Cahn equations. Specifically, we design two special convolutional neural network models, one for each of the two equations, to learn the fully-discrete operators between two adjacent time steps. The loss functions of the two models are defined using the residual of the fully-discrete systems, which result from applying the central finite difference discretization in space and the Crank–Nicolson approximation in time. This approach enables us to train the models without requiring any ground-truth data. Moreover, we introduce an effective training strategy that automatically generates useful samples along the time evolution to facilitate training of the models. Finally, we conduct extensive experiments in two and three dimensions to demonstrate outstanding performance of our proposed method, including its dynamics prediction and generalization ability under different scenarios. • We propose two novel convolutional neural networks to learn the fully-discrete operators of the classic and conservative Allen-Chan equations. • The loss functions are defined using the residual of the fully-discrete systems, which enables us to train the models without ground-truth data. • We introduce a novel training strategy that automatically generates samples along the time to facilitate effective training of the two models. • Extensive experiments demonstrate outstanding performance of the proposed models for dynamics prediction under different scenarios. [ABSTRACT FROM AUTHOR]

Abstract This paper proposes a novel neural network architecture inspired by the nonstandard form proposed by Beylkin et al. (1991) [7]. The nonstandard form is a highly effective wavelet-based compression scheme for linear integral operators. In this work, we first represent the matrix-vector product algorithm of the nonstandard form as a linear neural network where every scale of the multiresolution computation is carried out by a locally connected linear sub-network. In order to address nonlinear problems, we propose an extension, called BCR-Net, by replacing each linear sub-network with a deeper and more powerful nonlinear one. Numerical results demonstrate the efficiency of the new architecture by approximating nonlinear maps that arise in homogenization theory and stochastic computation. Highlights • Represented the BCR matrix-vector product algorithm as a linear neural network. • Extended the linear neural network to nonlinear case, called BCR-Net. • BCR-Net can learn certain nonlinear extensions of pseudo-differential operators. • Demonstrated numerically the efficiency of BCR-Net. [ABSTRACT FROM AUTHOR]

The multi-scale nature of gaseous flows poses tremendous difficulties for theoretical and numerical analysis. The Boltzmann equation, while possessing a wider applicability than hydrodynamic equations, requires significantly more computational resources due to the increased degrees of freedom in the model. The success of a hybrid fluid-kinetic flow solver for the study of multi-scale flows relies on accurate prediction of flow regimes. In this paper, we draw on binary classification in machine learning and propose the first neural network classifier to detect near-equilibrium and non-equilibrium flow regimes based on local flow conditions. Compared with classical semi-empirical criteria of continuum breakdown, the current method provides a data-driven alternative where the parameterized implicit function is trained by solutions of the Boltzmann equation. The ground-truth labels are derived rigorously from the deviation of particle distribution functions and the approximations based on the Chapman-Enskog ansatz. Therefore, no tunable parameter is needed in the criterion. Following the entropy closure of the Boltzmann moment system, a data generation strategy is developed to produce training and test sets. Numerical analysis shows its superiority over simulation-based samplings. A hybrid Boltzmann-Navier-Stokes flow solver is built correspondingly with an adaptive partition of local flow regimes. Numerical experiments including the one-dimensional Riemann problem, shear flow layer, and hypersonic flow around a circular cylinder are presented to validate the current scheme for simulating cross-scale and non-equilibrium flow physics. The quantitative comparison with a semi-empirical criterion and benchmark results demonstrates the capability of the current neural classifier to accurately predict continuum breakdown. The code for the data generator, hybrid solver, and neural network implementation is available in the open source repositories [1,2]. • The first neural network to predict equilibrium and non-equilibrium flow regimes is proposed. • A data generation strategy is developed based on the entropy closure theory of the Boltzmann equation. • The label of flow regime is rigorously derived with no tunable parameters. • A hybrid Boltzmann-Navier-Stokes flow solver is built upon the neural classifier. [ABSTRACT FROM AUTHOR]

Recent work has focused on data-driven learning of the evolution of unknown systems via deep neural networks (DNNs), with the goal of conducting long time prediction of the evolution of the unknown system. Training a DNN with low generalization error is a particularly important task in this case as error is accumulated over time. Because of the inherent randomness in DNN training, chiefly in stochastic optimization, there is uncertainty in the resulting prediction, and therefore in the generalization error. Hence, the generalization error can be viewed as a random variable with some probability distribution. Well-trained DNNs, particularly those with many hyperparameters, typically result in probability distributions for generalization error with low bias but high variance. High variance causes variability and unpredictably in the results of a trained DNN. This paper presents a computational technique which decreases the variance of the generalization error, thereby improving the reliability of the DNN model to generalize consistently. In the proposed ensemble averaging method, multiple models are independently trained and model predictions are averaged at each time step. A mathematical foundation for the method is presented, including results regarding the distribution of the local truncation error. In addition, three time-dependent differential equation problems are considered as numerical examples, demonstrating the effectiveness of the method to decrease variance of DNN predictions generally. • Proposed ensemble averaged learning for deep neural network (DNN) modeling unknown systems. • Established mathematical foundation for variance reduction by the ensemble averaged learning. • Provided extensive numerical studies to verify the theoretical results and demonstrate the practical advantage of the method. [ABSTRACT FROM AUTHOR]

Physics-Informed Neural Networks (PINNs) have emerged recently as a promising application of deep neural networks to the numerical solution of nonlinear partial differential equations (PDEs). However, it has been recognized that adaptive procedures are needed to force the neural network to fit accurately the stubborn spots in the solution of "stiff" PDEs. In this paper, we propose a fundamentally new way to train PINNs adaptively, where the adaptation weights are fully trainable and applied to each training point individually, so the neural network learns autonomously which regions of the solution are difficult and is forced to focus on them. The self-adaptation weights specify a soft multiplicative soft attention mask, which is reminiscent of similar mechanisms used in computer vision. The basic idea behind these SA-PINNs is to make the weights increase as the corresponding losses increase, which is accomplished by training the network to simultaneously minimize the losses and maximize the weights. In addition, we show how to build a continuous map of self-adaptive weights using Gaussian Process regression, which allows the use of stochastic gradient descent in problems where conventional gradient descent is not enough to produce accurate solutions. Finally, we derive the Neural Tangent Kernel matrix for SA-PINNs and use it to obtain a heuristic understanding of the effect of the self-adaptive weights on the dynamics of training in the limiting case of infinitely-wide PINNs, which suggests that SA-PINNs work by producing a smooth equalization of the eigenvalues of the NTK matrix corresponding to the different loss terms. In numerical experiments with several linear and nonlinear benchmark problems, the SA-PINN outperformed other state-of-the-art PINN algorithm in L2 error, while using a smaller number of training epochs. • Self-Adaptive PINNs is a new paradigm for training PINNs using trainable weights. • The SA training regime focuses on where error is the highest in the domain. • We demonstrate the use of SGD to mitigate full-batch computational limitations. • We show the SA-PINN improves gradient dynamics during training via NTK analysis. [ABSTRACT FROM AUTHOR]

In this paper, we investigate neural networks applied to multiscale simulations of porous media flows and discuss a design of a novel deep neural network model reduction approach for multiscale problems. Due to the multiscale nature of the medium, the fine-grid resolution gives rise to a huge number of degrees of freedom. In practice, low-order models are derived to reduce the computational cost. In our paper, we use a non-local multicontinuum (NLMC) approach [18] , which represents the solution on a coarse grid for porous media flows. In specific, we construct a reduced dimensional space which takes into account multi-continuum information. The numerical solutions are then sought in such space. Using multi-layer learning techniques, we formulate and learn input-output maps constructed with NLMC on a coarse grid. We study the features of the coarse-grid solutions that neural networks capture via relating the input-output optimization to ℓ 1 minimization of flow solutions. In proposed multi-layer networks, we can learn the forward operators in a reduced way without computing them as in POD like approaches. We present soft thresholding operators as activation function, which our studies show to have some advantages. With these activation functions, the neural network identifies and selects important multiscale features which are crucial in modeling the underlying flow. Using trained neural network approximation of the input-output map, we construct a reduced-order model for the solution approximation. We use multi-layer networks for the time stepping and reduced-order modeling, where at each time step the appropriate important modes are selected. For a class of nonlinear flow problems, we suggest an efficient strategy. Numerical examples are presented to examine the performance of our method. • Capturing important multiscale modes of porous media flows by neural networks. • Relating minimization to model reduction and deriving robust networks. • An efficient strategy for some nonlinear problems from porous media applications. • Multi-layer networks for combined time stepping and reduced-order modeling. [ABSTRACT FROM AUTHOR]

In a recent paper (Ray and Hesthaven, 2018) [38] , we proposed a new type of troubled-cell indicator to detect discontinuities in the numerical solutions of one-dimensional conservation laws. This was achieved by suitably training an artificial neural network on canonical local solution structures for conservation laws. The proposed indicator was independent of problem-dependent parameters, giving it an advantage over existing limiter-based indicators. In the present paper, we extend this approach to train a similar network capable of detecting troubled-cells on two-dimensional unstructured grids. The proposed network has a smaller architecture compared to its one-dimensional predecessor, making it computationally efficient. Several numerical results are presented to demonstrate the performance of the new indicator. • Designed an MLP network to serve as a troubled-cell indicator on two-dimensional unstructured grids. • The proposed indicator is free of problem-dependent parameters. • Tested with scalar conservation laws and the compressible Euler equations. • The MLP indicator correctly classifies smooth regions, and reduces computational cost associated with limiting. • Comparison with the minmod-type TVB indicator. [ABSTRACT FROM AUTHOR]

This paper develops a physics-informed neural network (PINN) combined with a Schwarz waveform relaxation (SWR) method for solving local and nonlocal advection-diffusion-reaction equations. Specifically, we derive the algorithm by constructing subdomain-dependent local solutions by minimizing local loss functions, allowing the decomposition of the training process in different domains in an embarrassingly parallel procedure. Provided the convergence of PINN, the overall proposed algorithm is convergent. By constructing local solutions, one can, in particular, adapt the depth of the deep neural networks, depending on the solution's spectral space and time complexity in each subdomain. One of the main advantages of using NN compared to standard solvers, is that the PINN algorithm introduces some learning in the SWR algorithm allowing for an acceleration of the overall algorithm, especially close to SWR convergence. We present some numerical experiments based on classical and Robin-SWR algorithms to illustrate the performance and comment on the convergence of the proposed method. • Schwarz Waveform Relaxation algorithms combined with Physics-Informed Neural Networks. • Mathematical justification of the proposed algorithms. • Analysis of complexity and gain compared to standard approach. • Numerical experiments. [ABSTRACT FROM AUTHOR]

We introduce in this paper new and very effective numerical methods based on neural networks for the approximation of the mean curvature flow of either oriented or non-orientable surfaces. To learn the correct interface evolution law, our neural networks are trained on phase field representations of exact evolving interfaces. The structures of the networks draw inspiration from splitting schemes used for the discretization of the Allen-Cahn equation. But when the latter approximate the mean curvature motion of oriented interfaces only, the approach we propose extends very naturally to the non-orientable case. Through a variety of examples, we show that our networks, trained only on flows of smooth and simplistic interfaces, generalize very well to more complex interfaces, either oriented or non-orientable, and possibly with singularities. Furthermore, they can be coupled easily with additional constraints which opens the way to various applications illustrating the flexibility and effectiveness of our approach: mean curvature flows with volume constraint, multiphase mean curvature flows, numerical approximation of Steiner trees or minimal surfaces. • Approximation with neural networks of the motion by mean curvature of general interfaces. • Introduction of neural network structures well adapted to phase field representations. • Effective numerical approach for computing approximations of Steiner trees and minimal surfaces. [ABSTRACT FROM AUTHOR]

In this paper, we propose forward and backward stochastic differential equations (FBSDEs) based deep neural network (DNN) learning algorithms for the solution of high dimensional quasi-linear parabolic partial differential equations (PDEs), which is related to the FBSDEs from the Pardoux-Peng theory. The algorithms rely on a learning process by minimizing the path-wise difference between two discrete stochastic processes, which are defined by the time discretization of the FBSDEs and the DNN representation of the PDE solution, respectively. The proposed algorithms are shown to generate DNN solution for a 100-dimensional Black–Scholes–Barenblatt equation, which is accurate in a finite region in the solution space, and has a convergence rate close to that of the Euler–Maruyama scheme used for discretizing the FBSDEs. • FBSDE deep neural network (DNN) with loss comparing two stochastic processes for the PDEs from Pardoux-Peng theory. • Strong 1/2 Convergence as the Euler-Maruyama method in solving a 100-dimensional Black-Scholes-Barenblatt equation. • Use of Richardson extrapolation for accuracy enhancement. • Multiscale DNN captures temporal high frequency content of the PDEs more accurately. [ABSTRACT FROM AUTHOR]

• We develop the FBSTD, an NN based temporal-difference learning for PDEs. • The cost of the FBSTD is significantly lower than some popular NN methods for FBSDEs. • The FBSTD can be applied to solve general nonlinear, high order and high-dimensional PDEs. Solving high-dimensional partial differential equations (PDEs) is a long-standing challenge, for which classical numerical methods suffer from the well-known curse of dimensionality. In this paper, we propose deep neural networks (NN) based temporal-difference (TD) learning methods for numerically solving high-dimensional parabolic PDEs. To this end, we approximate the solution of the original PDE with an NN function. To calculate this neural network function, we first transform the deterministic parabolic PDE to a forward-backward stochastic differential equations (FBSDE) system with the nonlinear Feynman-Kac formula, and then transform the forward updating process of FBSDE as a Markov reward process (MRP). On this basis, we approximate the solution of the PDE by training an NN function with a reinforcement learning technique described as below: we first discretize the temporal interval to a finite number of time steps and then at each time step, we generate many trajectories and design the loss function as the mean square of temporal difference error on all trajectories. With this loss function, we update the parameters of the NN function with the stochastic gradient descent (SGD) method. Numerical experiments show that comparing to some other existed deep learning methods, our method not only accelerates the computational speed but also improves the computational accuracy. In particular, the relative errors of our algorithm achieve the order of O (10 − 4) even the dimension of the problem is as high as 100. [ABSTRACT FROM AUTHOR]

Generating quasirandom points with high uniformity is a fundamental task in many fields. Existing number-theoretic approaches produce evenly distributed points in [ 0 , 1 ] d in asymptotic sense but may not yield a good distribution for a given set size. It is also difficult to extend those techniques to other geometries like a disk or a manifold. In this paper, we present a novel physics-informed framework to transform a given set of points into a distribution with better uniformity. We model each point as a particle and assign the system with a potential energy. Upon minimizing the energy, the uniformity of distribution can be improved correspondingly. Two kinds of schemes are introduced: one based on molecular dynamics and another based on deep neural networks. The new physics-informed framework serves as a black-box transformer that is able to improve given distributions and can be easily extended to other geometries such as disks, spheres, complex manifolds, etc. Various experiments with different geometries are provided to demonstrate that the new framework is able to transform poorly distributed input into one with superior uniformity. • A novel physics-informed framework for improving the uniformity of a given distribution. • Molecular dynamics and deep neural networks are used for transforming distributions. • The new framework works for different geometries including general manifolds. • A new matrix-based metric is introduced to measure uniformity on general geometry. • Experiments show that the new approach is effective for various geometries. [ABSTRACT FROM AUTHOR]

In many scientific fields, empirical models are employed to facilitate computational simulations of engineering systems. For example, in fluid mechanics, empirical Reynolds stress closures enable computationally-efficient Reynolds Averaged Navier Stokes simulations. Likewise, in solid mechanics, constitutive relations between the stress and strain in a material are required in deformation analysis. Traditional methods for developing and tuning empirical models usually combine physical intuition with simple regression techniques on limited data sets. The rise of high performance computing has led to a growing availability of high fidelity simulation data. These data open up the possibility of using machine learning algorithms, such as random forests or neural networks, to develop more accurate and general empirical models. A key question when using data-driven algorithms to develop these empirical models is how domain knowledge should be incorporated into the machine learning process. This paper will specifically address physical systems that possess symmetry or invariance properties. Two different methods for teaching a machine learning model an invariance property are compared. In the first method, a basis of invariant inputs is constructed, and the machine learning model is trained upon this basis, thereby embedding the invariance into the model. In the second method, the algorithm is trained on multiple transformations of the raw input data until the model learns invariance to that transformation. Results are discussed for two case studies: one in turbulence modeling and one in crystal elasticity. It is shown that in both cases embedding the invariance property into the input features yields higher performance at significantly reduced computational training costs. [ABSTRACT FROM AUTHOR]

In this paper, we consider the eigenvalue PDE problem of the infinitesimal generators of metastable diffusion processes. We propose a numerical algorithm based on training artificial neural networks for solving the leading eigenvalues and eigenfunctions of such high-dimensional eigenvalue problem. The algorithm is able to find multiple leading eigenpairs by solving a single training task. It is useful in understanding the dynamical behaviors of metastable processes on large timescales. We demonstrate the capability of our algorithm on a high-dimensional model problem, and on the simple molecular system alanine dipeptide. • Training algorithm for certain eigenvalue PDE problems using neural networks. • Training loss designed based on variational principle. • Compute multiple eigenvalues and eigenfunctions by solving a single training task. [ABSTRACT FROM AUTHOR]

Simulation of multiphase flow in porous media is crucial for the effective management of subsurface energy and environment-related activities. The numerical simulators used for modeling such processes rely on spatial and temporal discretization of the governing mass and energy balance partial-differential equations (PDEs) into algebraic systems via finite-difference/volume/element methods. These simulators usually require dedicated software development and maintenance, and suffer low efficiency from a runtime and memory standpoint for problems with multi-scale heterogeneity, coupled-physics processes or fluids with complex phase behavior. Therefore, developing cost-effective, data-driven models can become a practical choice, and in this work, we choose deep learning approaches as they can handle high dimensional data and accurately predict state variables with strong nonlinearity. In this paper, we describe a gradient-based deep neural network (GDNN) constrained by the physics related to multiphase flow in porous media. We tackle the nonlinearity of flow in porous media induced by rock heterogeneity, fluid properties, and fluid-rock interactions by decomposing the nonlinear PDEs into a dictionary of elementary differential operators. We use a combination of operators to handle rock spatial heterogeneity and fluid flow by advection. Since the augmented differential operators are inherently related to the physics of fluid flow, we treat them as first principles prior knowledge to regularize the GDNN training. We use the example of pressure management at geologic C O 2 storage sites, where C O 2 is injected in saline aquifers and brine is produced, and apply GDNN to construct a predictive model that is trained with physics-based simulation data and emulates the physics process. We demonstrate that GDNN can effectively predict the nonlinear patterns of subsurface responses, including the temporal and spatial evolution of the pressure and C O 2 saturation plumes. We also successfully extend the GDNN to convolutional neural network (CNN), namely gradient-based CNN (GCNN), and validate its capability to improve the prediction accuracy. GDNN has great potential to tackle challenging problems that are governed by highly nonlinear physics and enable the development of data-driven models with higher fidelity. • A workflow based on Gradient-based deep neural network (GDNN) emulates multiphase flow in porous media with high fidelity. • Differential operators derived from the governing PDEs are used as first principle prior knowledge to regularize the training of GDNN. • GDNN is further successfully extended to image-based approaches such as convolutional neural networks (CNN). [ABSTRACT FROM AUTHOR]

• We present a computational model of cardiac electromechanics of the left ventricle. • 3D electromechanics is coupled with a 0D circulation model by a novel numerical scheme. • We show that the coupled model is compliant with the principle of energy conservation. • We employ a Neural Network based surrogate of an high fidelity active force model. • We present a robust algorithm to reconstruct the stress-free configuration. We propose a novel mathematical and numerical model for cardiac electromechanics, wherein biophysically detailed core models describe the different physical processes concurring to the cardiac function. The core models, once suitably approximated, are coupled by a computationally efficient strategy. Our model is based on: (1) the combination of implicit-explicit (IMEX) schemes to solve the different core cardiac models, (2) an Artificial Neural Network based model, that surrogates a biophysically detailed but computationally demanding microscale model of active force generation and (3) appropriate partitioned schemes to couple the different models in this multiphysics setting. We employ a flexible and scalable intergrid transfer operator, which allows to interpolate Finite Element functions between nested meshes and, possibly, among arbitrary Finite Element spaces for the different core models. Our core 3D electromechanical model of the left ventricle is coupled with a closed-loop 0D model of the vascular network (and the other cardiac chambers) by an approach that is energy preserving. More precisely, we derive a balance law for the mechanical energy of the whole circulatory network. This provides a quantitative insight into the energy utilization, dissipation and transfer among the different compartments of the cardiovascular network and during different stages of the heartbeat. On this ground, a new tool is proposed to validate some energy indicators adopted in the daily clinical practice. A further contribution of this paper is the proposition of a robust algorithm for the reconstruction of the stress-free reference configuration. This feature is fundamental to correctly initialize our electromechanical simulations. As a matter of fact, the geometry acquired from medical imaging typically refers to a configuration affected by residual internal stresses, whereas the elastodynamics equations that govern the mechanics core model are related to a stress-free configuration. To prove the biophysical accuracy of our computational model, we address different scenarios of clinical interest, namely by varying preload, afterload and contractility. [ABSTRACT FROM AUTHOR]

• A fiction stir welding model with coupled fluid flow and heat transfer is used for parametric study. • A combination of model reduction and machine learning techniques is shown to be effective. • Proper orthogonal decomposition (POD) can reduce the computational time by 70%. • An artificial neural network (ANN) provides an efficient alternative to search over the parameter space. The friction stir welding process can be modeled using a system of heat transfer and Navier-Stokes equations with a shear dependent viscosity. Finding numerical solutions of this system of nonlinear partial differential equations over a set of parameter space, however, is extremely time-consuming. Therefore, it is desirable to find a computationally efficient method that can be used to obtain an approximation of the solution with acceptable accuracy. In this paper, we present a reduced basis method for solving the parametrized coupled system of heat and Navier-Stokes equations using a proper orthogonal decomposition (POD). In addition, we apply a machine learning algorithm based on an artificial neural network (ANN) to learn (approximately) the relationship between relevant parameters and the POD coefficients. Our computational experiments demonstrate that substantial speed-up can be achieved while maintaining sufficient accuracy. [ABSTRACT FROM AUTHOR]

• Low-rank multifidelity construction of surrogate models for Physics-Informed Neural Networks. • Width, depth, and optimization hyperparameters of Physics-Informed Neural Networks as a form of fidelity. • Theoretical and implementation considerations to guide tuning and set up. • Cost and accuracy improvement on four canonical forward partial differential equation problems. Multifidelity simulation methodologies are often used in an attempt to judiciously combine low-fidelity and high-fidelity simulation results in an accuracy-increasing, cost-saving way. Candidates for this approach are simulation methodologies for which there are fidelity differences connected with significant computational cost differences. Physics-informed Neural Networks (PINNs) are candidates for these types of approaches due to the significant difference in training times required when different fidelities (expressed in terms of architecture width and depth as well as optimization criteria) are employed. In this paper, we propose a particular multifidelity approach applied to PINNs that exploits low-rank structure. We demonstrate that width, depth, and optimization criteria can be used as parameters related to model fidelity and show numerical justification of cost differences in training due to fidelity parameter choices. We test our multifidelity scheme on various canonical forward PDE models that have been presented in the emerging PINNs literature. [ABSTRACT FROM AUTHOR]

• Study a highly challenging inverse scattering with severe lack of information, which is new to the literature. • Propose a novel imaging scheme by incorporating physical insights into a machine learning method. • Extensive numerical results evidence the salient and highly promising features of the proposed method. • The new perspective should be able to develop novel schemes for tackling other inverse problems that cannot be tackled by traditional methods. In this paper, we consider artificial neural networks for inverse scattering problems. As a working model, we consider the inverse problem of recovering a scattering object from the (possibly) limited-aperture radar cross section (RCS) data collected corresponding to a single incident field. This nonlinear and ill-posed inverse problem is practically important and highly challenging due to the severe lack of information. From a geometrical and physical point of view, the low-frequency data should be able to resolve the unique identifiability issue, but meanwhile lose the resolution. On the other hand, the machine learning can be used to break through the resolution limit. By combining the two perspectives, we develop a fully connected neural network (FCNN) for the inverse problem. Extensive numerical results show that the proposed method can produce stunning reconstructions. The proposed strategy can be extended to tackling other inverse scattering problems with limited measurement information. [ABSTRACT FROM AUTHOR]

Abstract: Uncertainty quantification is an increasingly important aspect of many areas of computational science, where the challenge is to make reliable predictions about the performance of complex physical systems in the absence of complete or reliable data. Predicting flows of oil and water through oil reservoirs is an example of a complex system where accuracy in prediction is needed primarily for financial reasons. Simulation of fluid flow in oil reservoirs is usually carried out using large commercially written finite difference simulators solving conservation equations describing the multi-phase flow through the porous reservoir rocks. This paper examines a Bayesian Framework for uncertainty quantification in porous media flows that uses a stochastic sampling algorithm to generate models that match observed data. Machine learning algorithms are used to speed up the identification of regions in parameter space where good matches to observed data can be found. [Copyright &y& Elsevier]

Full-scale system tests are often prohibitively expensive, therefore experiments on simpler configurations are used in engineering to gain insights that can be applied to the behavior prediction of the full-scale system. However, the transfer of information from the experimental configuration to the prediction configuration is not trivial. In this paper, we present a methodology for predicting the response of a dynamic system using data from tests conducted on a different experimental configuration. The experimental and prediction configurations are connected by a set of common model form errors, which are unrelated to geometry, boundary conditions, or loading. These errors cause discrepancies between the model prediction of system response and the corresponding measured values. The proposed methodology is based on estimating the model form errors for the experimental configuration, using the measured data, and propagating the estimated errors to the prediction configuration. The governing equations of the two systems – the experimental and prediction configurations – are discretized using the finite element method, using elements of the same type and dimensions. Bayesian state estimation is adopted to estimate the discrepancies in unmeasured responses of the tested configuration, and subsequently, these estimates are used to evaluate the model form errors. These errors are propagated to untested configurations using an artificial neural network (ANN). The proposed methodology is illustrated on predicting the deformation of a simply supported beam and a curved plate, and for predicting the temperature time histories along a rigid bar under one-dimensional heat conduction. • Model error estimation for dynamic systems using Bayesian state estimation. • Propagating model errors to untested configurations. • Discrepancy prediction for untested system configurations. [ABSTRACT FROM AUTHOR]

• Novel machine learning architecture for data-driven constitutive modeling. • Reducing required training data by incorporation of continuum mechanical knowledge. • Predictive constitutive modeling as required for designing new materials. In this paper we introduce constitutive artificial neural networks (CANNs), a novel machine learning architecture for data-driven modeling of the mechanical constitutive behavior of materials. CANNs are able to incorporate by their very design information from three different sources, namely stress-strain data, theoretical knowledge from materials theory, and diverse additional information (e.g., about microstructure or materials processing). CANNs can easily and efficiently be implemented in standard computational software. They require only a low-to-moderate amount of training data and training time to learn without human guidance the constitutive behavior also of complex nonlinear and anisotropic materials. Moreover, in a simple academic example we demonstrate how the input of microstructural data can endow CANNs with the ability to describe not only the behavior of known materials but to predict also the properties of new materials where no stress-strain data are available yet. This ability may be particularly useful for the future in-silico design of new materials. The developed source code of the CANN architecture and accompanying example data sets are available at https://github.com/ConstitutiveANN/CANN. [ABSTRACT FROM AUTHOR]

Deep Reinforcement Learning (DRL) has recently spread into a range of domains within physics and engineering, with multiple remarkable achievements. Still, much remains to be explored before the capabilities of these methods are well understood. In this paper, we present the first application of DRL to direct shape optimization. We show that, given adequate reward, an artificial neural network trained through DRL is able to generate optimal shapes on its own, without any prior knowledge and in a constrained time. While we choose here to apply this methodology to aerodynamics, the optimization process itself is agnostic to details of the use case, and thus our work paves the way to new generic shape optimization strategies both in fluid mechanics, and more generally in any domain where a relevant reward function can be defined. • Deep Reinforcement Learning coupled with Computational fluid Dynamics. • Optimal shapes generation through Deep Reinforcement Learning. • Shape generation using Bézier curves and implemented in a new DRL environment. • Adequate reward function and shaping based on the lift-to-drag ratio. • Introduction of a degenerate DRL approach as general purpose optimizers. [ABSTRACT FROM AUTHOR]

• A machine-learning approach is developed for determining optimal mesh anisotropy. • The artificial network uses normalized primal, adjoint, and error features. • The network input and outputs generalize across flowfields and quantities of interest. • Regularization implicit in the training improves network performance over MOESS. This paper presents a machine-learning approach for determining the optimal anisotropy in a computational mesh, in the context of an output-based adaptive solution procedure. Artificial neural networks are used to predict the desired element aspect ratio from readily accessible features of the primal and adjoint solutions. Whereas the sizing of the element is still based on an adjoint-weighted residual error estimate, the network augments this information with element stretching magnitude and direction, at lower computational and implementation costs compared to a more rigorous approach: mesh optimization through error sampling and synthesis (MOESS). The network is trained to provide anisotropy information in the form of a normalized metric field, computed from primal, adjoint, and error indicator features. MOESS-optimized meshes for a variety of steady aerodynamic flows governed by the Reynolds-averaged Navier-Stokes equations in two dimensions provide data for training several multi-layer perceptron networks, which differ in size and inputs. The networks are then deployed and tested by driving complete mesh adaptation sequences, and the results show improvements in mesh efficiency compared to pure primal Hessian-based anisotropy detection and in many cases to MOESS itself. [ABSTRACT FROM AUTHOR]

• Transport sweeps are replaced by the recursive evaluation of an Artificial Neural Network (ANN). • ANNs can accelerate radiation transport sweeps by a factor of ∼3.6, while introducing errors of 0.5-2. • The structure observed in transport problems allows us to optimize the ANNs. • New training/testing set generation and DropConnect techniques have been developed. Discontinuous Galerkin Finite Element Methods (DGFEM) have been widely used for solving S N radiation transport problems in participative and non-participative media. Global matrices are not assembled when sweeping through the computational domain, but only small matrix-vector systems are assembled and solved for each cell, angle, energy group, and time step (e.g., systems with 8 independent equations for tri-linear DGFEM in 3D hexahedral cells). These systems are generally solved directly using Gaussian elimination. The computational cost of assembling and solving these local systems, repeated for each cell in the phase-space, can amount to a large fraction of the total computation time. Therefore, a Machine Learning algorithm is designed in this paper, based on Artificial Neural Networks (ANNs), to replace the assembling and solution of the local systems, enabling a sizable speed up in the solution process. The key idea is to train an ANN with a large set of solutions to random one-cell transport problems and, then, replace the assembling and solution of the local systems by the feedforward evaluation of the trained ANN in large-scale transport solvers. These ANNs are optimized to reproduce the solutions obtained in radiation transport solves, while minimizing the number of operations involved in its feedforward evaluation. It is observed that the optimized ANNs are able to reduce the compute times by a factor of ∼3.6 per source iteration, while introducing mean absolute errors between 0.5 − 2 % in transport solutions. [ABSTRACT FROM AUTHOR]

Artificial neural networks together with associated computational libraries provide a powerful framework for constructing both classification and regression algorithms. In this paper we use neural networks to design linear and non-linear discrete differential operators. We show that neural network based operators can be used to construct stable discretizations of initial boundary-value problems by ensuring that the operators satisfy a discrete analogue of integration-by-parts known as summation-by-parts. Our neural network approach with linear activation functions is compared and contrasted with a more traditional linear algebra approach. An application to overlapping grids is explored. The strategy developed in this work opens the door for constructing stable differential operators on general meshes. [ABSTRACT FROM AUTHOR]

• Represent the linearized forward and inverse maps of electrical impedance tomography (EIT) with deep neural network. • Propose compact neural network architectures for the nonlinear forward and inverse maps. • Demonstrate numerically the efficiency of the proposed neural networks. This paper introduces a new approach for solving electrical impedance tomography (EIT) problems using deep neural networks. The mathematical problem of EIT is to invert the electrical conductivity from the Dirichlet-to-Neumann (DtN) map. Both the forward map from the electrical conductivity to the DtN map and the inverse map are high-dimensional and nonlinear. Motivated by the linear perturbative analysis of the forward map and based on a numerically low-rank property, we propose compact neural network architectures for the forward and inverse maps for both 2D and 3D problems. Numerical results demonstrate the efficiency of the proposed neural networks. [ABSTRACT FROM AUTHOR]

Partial differential equations (PDEs) are commonly derived based on empirical observations. However, recent advances of technology enable us to collect and store massive amount of data, which offers new opportunities for data-driven discovery of PDEs. In this paper, we propose a new deep neural network, called PDE-Net 2.0, to discover (time-dependent) PDEs from observed dynamic data with minor prior knowledge on the underlying mechanism that drives the dynamics. The design of PDE-Net 2.0 is based on our earlier work [1] where the original version of PDE-Net was proposed. PDE-Net 2.0 is a combination of numerical approximation of differential operators by convolutions and a symbolic multi-layer neural network for model recovery. Comparing with existing approaches, PDE-Net 2.0 has the most flexibility and expressive power by learning both differential operators and the nonlinear response function of the underlying PDE model. Numerical experiments show that the PDE-Net 2.0 has the potential to uncover the hidden PDE of the observed dynamics, and predict the dynamical behavior for a relatively long time, even in a noisy environment. • The proposal of a numeric-symbolic hybrid deep network to recover PDEs from observed dynamic data. • The symbolic network is able to recover concise analytic form of the hidden PDE model. • Our approach only requires minor prior knowledge on the mechanism of the observed dynamic data. • The network can perform accurate long-term prediction without re-training for new initial conditions. [ABSTRACT FROM AUTHOR]