Search

Phytoplankton blooms exhibit varying patterns in timing and number of peaks within ecosystems. These differences in blooming patterns are partly explained by phytoplankton:nutrient interactions and external factors such as temperature, salinity and light availability. Understanding these interactions and drivers is essential for effective bloom management and modelling as driving factors potentially differ or are shared across ecosystems on regional scales. Here, we used a 22-year data set (19 years training and 3 years validation data) containing chlorophyll, nutrients (dissolved and total), and external drivers (temperature, salinity, light) of the southern Baltic Sea coast, a European brackish shelf sea, which constituted six different phytoplankton blooming patterns. We employed generalized additive mixed models to characterize similar blooming patterns and trained an artificial neural network within the Universal Differential Equation framework to learn a differential equation representation of these pattern. Applying Sparse Identification of Nonlinear Dynamics uncovered algebraic relationships in phytoplankton:nutrient:external driver interactions. Nutrients availability was driving factor for blooms in enclosed coastal waters; nutrients and temperature in more open regions. We found evidence of hydrodynamical export of phytoplankton, natural mortality or external grazing not explicitly measured in the data. This data-driven workflow allows new insight into driver-differences in region specific blooming dynamics. [ABSTRACT FROM AUTHOR]

A posteriori reduced‐order models (ROM), for example, based on proper orthogonal decomposition (POD), are essential to affordably tackle realistic parametric problems. They rely on a trustful training set, that is a family of full‐order solutions (snapshots) representative of all possible outcomes of the parametric problem. Having such a rich collection of snapshots is not, in many cases, computationally viable. A strategy for data augmentation, designed for parametric laminar incompressible flows, is proposed to enrich poorly populated training sets. The goal is to include in the new, artificial snapshots emerging features, not present in the original basis, that do enhance the quality of the reduced basis (RB) constructed using POD dimensionality reduction. The methodologies devised are based on exploiting basic physical principles, such as mass and momentum conservation, to construct physically relevant, artificial snapshots at a fraction of the cost of additional full‐order solutions. Interestingly, the numerical results show that the ideas exploiting only mass conservation (i.e., incompressibility) are not producing significant added value with respect to the standard linear combinations of snapshots. Conversely, accounting for the linearized momentum balance via the Oseen equation does improve the quality of the resulting approximation and therefore is an effective data augmentation strategy in the framework of viscous incompressible laminar flows. Numerical experiments of parametric flow problems, in two and three dimensions, at low and moderate values of the Reynolds number are presented to showcase the superior performance of the data‐enriched POD‐RB with respect to the standard ROM in terms of both accuracy and efficiency. [ABSTRACT FROM AUTHOR]

In the context of hybrid twins, a data-driven enrichment is added to the physics-based solution to represent with higher accuracy the reference solution assumed to be known at different points in the physical domain. Such an approach enables better predictions. However, the data-driven enrichment is usually represented by a regression, whose main drawbacks are (i) the difficulty of understanding the subjacent physics and (ii) the risks induced by the data-driven model extrapolation. This paper proposes a procedure enabling the extraction of a differential operator associated with the enrichment provided by the data-driven regression. For that purpose, a sparse Singular Value Decomposition, SVD, is introduced. It is then employed, first, in a full operator representation regularized optimization problem, where sparsity is promoted, leading to a linear programming problem, and then in a tensor decomposition of the operator's identification procedure. The results show the ability of the method to identify the exact missing operators from the model. The regularized optimization problem was also able to identify the weights of the missing terms with a relative error of about 10% on average, depending on the selected use case. [ABSTRACT FROM AUTHOR]

We develop a novel methodology for data-driven simulation of coupled multi-physics systems. The result of the method is a learned numerical integrator of the coupled system dynamics. In order to preserve the fundamental physics of the coupled systems, and thus preserve the geometrical properties of the governing equations—even if they may be completely unknown—we impose a port-metriplectic structure on the system evolution, i.e., a combination of a symplectic evolution for the system energy with a gradient flow for the entropy of each system, which can be exchanged through predefined ports. The resulting method guarantees by construction the satisfaction of the laws of thermodynamics for open systems, leading to accurate predictions of the future states of their dynamics. Examples are given for systems of varying complexity, based on synthetic as well as experimental data. [ABSTRACT FROM AUTHOR]

Generalization is a key property of machine learning models to perform accurately on unseen data. Conversely, in the field of scientific machine learning (SciML), generalization entails not only predictive accuracy but also the capacity of the model to encapsulate underlying physical principles. In this paper, we delve into the concept of generalization for Physics-informed neural networks (PINNs) by investigating the consistency of the predictions of a PINN outside of its training domain. Through the lenses of a novel metric and statistical analysis, we study the scenarios in which a PINN can provide consistent predictions outside the region considered for training and hereinafter assess whether the algorithmic setup of the model can influence its potential for generalizing. Our results highlight why overparametrization is not a crucial component in SciML while encouraging overfitting on the training data. Despite being counterintuitive, the outcome of our analysis serves as a guideline for training PINNs for engineering applications. [ABSTRACT FROM AUTHOR]

Radiative heat transfer is a fundamental process in high energy density physics and inertial fusion. Accurately predicting the behavior of Marshak waves across a wide range of material properties and drive conditions is crucial for design and analysis of these systems. Conventional numerical solvers and analytical approximations often face challenges in terms of accuracy and computational efficiency. In this work, we propose a novel approach to model Marshak waves using Fourier Neural Operators (FNO). We develop two FNO-based models: (1) a base model that learns the mapping between the drive condition and material properties to a solution approximation based on the widely used analytic model by Hammer & Rosen (2003), and (2) a model that corrects the inaccuracies of the analytic approximation by learning the mapping to a more accurate numerical solution. Our results demonstrate the strong generalization capabilities of the FNOs and show significant improvements in prediction accuracy compared to the base analytic model. [ABSTRACT FROM AUTHOR]

Gradient-based meta-learning methods have primarily been applied to classical machine learning tasks such as image classification. Recently, PDE-solving deep learning methods, such as neural operators, are starting to make an important impact on learning and predicting the response of a complex physical system directly from observational data. Taking the material modeling problems for example, the neural operator approach learns a surrogate mapping from the loading field to the corresponding material response field, which can be seen as learning the solution operator of a hidden PDE. The microstructure and mechanical parameters of each material specimen correspond to the (possibly heterogeneous) parameter field in this hidden PDE. Due to the limitation on experimental measurement techniques, the data acquisition for each material specimen is commonly challenging and costly. This fact calls for the utilization and transfer of existing knowledge to new and unseen material specimens, which corresponds to sampling efficient learning of the solution operator of a hidden PDE with a different parameter field. Herein, we propose a novel meta-learning approach for neural operators, which can be seen as transferring the knowledge of solution operators between governing (unknown) PDEs with varying parameter fields. Our approach is a provably universal solution operator for multiple PDE solving tasks, with a key theoretical observation that underlying parameter fields can be captured in the first layer of neural operator models, in contrast to typical final-layer transfer in existing meta-learning methods. As applications, we demonstrate the efficacy of our proposed approach on PDE-based datasets and a real-world material modeling problem, illustrating that our method can handle complex and nonlinear physical response learning tasks while greatly improving the sampling efficiency in unseen tasks.

Closure problems are omnipresent when simulating multiscale systems, where some quantities and processes cannot be fully prescribed despite their effects on the simulation's accuracy. Recently, scientific machine learning approaches have been proposed as a way to tackle the closure problem, combining traditional (physics-based) modeling with data-driven (machine-learned) techniques, typically through enriching differential equations with neural networks. This paper reviews the different reduced model forms, distinguished by the degree to which they include known physics, and the different objectives of a priori and a posteriori learning. The importance of adhering to physical laws (such as symmetries and conservation laws) in choosing the reduced model form and choosing the learning method is discussed. The effect of spatial and temporal discretization and recent trends toward discretization-invariant models are reviewed. In addition, we make the connections between closure problems and several other research disciplines: inverse problems, Mori-Zwanzig theory, and multi-fidelity methods. In conclusion, much progress has been made with scientific machine learning approaches for solving closure problems, but many challenges remain. In particular, the generalizability and interpretability of learned models is a major issue that needs to be addressed further. [ABSTRACT FROM AUTHOR]

Human tissues are highly organized structures with collagen fiber arrangements varying from point to point. Anisotropy of the tissue arises from the natural orientation of the fibers, resulting in location-dependent anisotropy. Heterogeneity also plays an important role in tissue function. It is therefore critical to discover and understand the distribution of fiber orientations from experimental mechanical measurements such as digital image correlation (DIC) data. To this end, we introduce the Heterogeneous Peridynamic Neural Operator (HeteroPNO) approach for data-driven constitutive modeling of heterogeneous anisotropic materials. Our goal is to learn a nonlocal constitutive law together with the material microstructure, in the form of a heterogeneous fiber orientation field, from load-displacement field measurements. We propose a two-phase learning approach. First, we learn a homogeneous constitutive law in the form of a neural network-based kernel function and a nonlocal bond force. This captures the complex homogeneous material responses from data. Second, we reinitialize the learnt bond force and train it together with the kernel and the fiber orientation field for each material point. Due to the inherent properties of the state-based peridynamic framework that we use, our HeteroPNO-learned material models are guaranteed to be objective and to satisfy the balances of linear and angular momenta. The effects of heterogeneity and the nonlinearity of the constitutive relationship are captured by the nonlocal kernel function, enabling physical interpretability. These features enable our HeteroPNO architecture to learn a constitutive model for biological tissue with anisotropic, heterogeneous response undergoing large deformation. To demonstrate the applicability of our approach, we apply the HeteroPNO in learning a material model and fiber orientation field from DIC measurements in biaxial testing. We evaluate the learnt fiber architecture, which is found to be consistent with observations from polarized spatial frequency domain imaging. The framework is capable of providing displacement and stress field predictions for new and unseen loading instances. [ABSTRACT FROM AUTHOR]

Machine learning (ML) methods, which fit data to the parameters of a given parameterized model class, have garnered significant interest as potential methods for learning surrogate models for complex engineering systems where traditional simulation is expensive. However, in many scientific and engineering settings, generating high-fidelity data to train ML models is expensive, and the available budget for generating training data is limited, making high-fidelity training data scarce. ML models trained on scarce data have high variance, resulting in poor expected generalization performance. We propose a new multifidelity training approach for scientific machine learning via linear regression that exploits the scientific context where data of varying fidelities and costs are available; for example, high-fidelity data may be generated by an expensive fully resolved physics simulation whereas lower-fidelity data may arise from a cheaper model based on simplifying assumptions. We use the multifidelity data within an approximate control variate framework to define new multifidelity Monte Carlo estimators for linear regression models. We provide bias and variance analysis of our new estimators that guarantee the approach's accuracy and improved robustness to scarce high-fidelity data. Numerical results demonstrate that our multifidelity training approach achieves similar accuracy to the standard high-fidelity-only approach, significantly reducing high-fidelity data requirements. [ABSTRACT FROM AUTHOR]

Abstract This research introduces an accelerated training approach for Vanilla Physics-Informed Neural Networks (PINNs) that addresses three factors affecting the loss function: the initial weight state of the neural network, the ratio of domain to boundary points, and the loss weighting factor. The proposed method involves two phases. In the initial phase, a unique loss function is created using a subset of boundary conditions and partial differential equation terms. Furthermore, we introduce preprocessing procedures that aim to decrease the variance during initialization and choose domain points according to the initial weight state of various neural networks. The second phase resembles Vanilla-PINN training, but a portion of the random weights are substituted with weights from the first phase. This implies that the neural network’s structure is designed to prioritize the boundary conditions, subsequently affecting the overall convergence. The study evaluates the method using three benchmarks: two-dimensional flow over a cylinder, an inverse problem of inlet velocity determination, and the Burger equation. Incorporating weights generated in the first training phase neutralizes imbalance effects. Notably, the proposed approach outperforms Vanilla-PINN in terms of speed, convergence likelihood and eliminates the need for hyperparameter tuning to balance the loss function.

Abstract Learning operators with deep neural networks is an emerging paradigm for scientific computing. Deep Operator Network (DeepONet) is a modular operator learning framework that allows for flexibility in choosing the kind of neural network to be used in the trunk and/or branch of the DeepONet. This is beneficial as it has been shown many times that different types of problems require different kinds of network architectures for effective learning. In this work, we design an efficient neural operator based on the DeepONet architecture. We introduce U-Net enhanced DeepONet (U-DeepONet) for learning the solution operator of highly complex CO2-water two-phase flow in heterogeneous porous media. The U-DeepONet is more accurate in predicting gas saturation and pressure buildup than the state-of-the-art U-Net based Fourier Neural Operator (U-FNO) and the Fourier-enhanced Multiple-Input Operator (Fourier-MIONet) trained on the same dataset. Moreover, our U-DeepONet is significantly more efficient in training times than both the U-FNO (more than 18 times faster) and the Fourier-MIONet (more than 5 times faster), while consuming less computational resources. We also show that the U-DeepONet is more data efficient and better at generalization than both the U-FNO and the Fourier-MIONet.

This research introduces an accelerated training approach for Vanilla Physics-Informed Neural Networks (PINNs) that addresses three factors affecting the loss function: the initial weight state of the neural network, the ratio of domain to boundary points, and the loss weighting factor. The proposed method involves two phases. In the initial phase, a unique loss function is created using a subset of boundary conditions and partial differential equation terms. Furthermore, we introduce preprocessing procedures that aim to decrease the variance during initialization and choose domain points according to the initial weight state of various neural networks. The second phase resembles Vanilla-PINN training, but a portion of the random weights are substituted with weights from the first phase. This implies that the neural network's structure is designed to prioritize the boundary conditions, subsequently affecting the overall convergence. The study evaluates the method using three benchmarks: two-dimensional flow over a cylinder, an inverse problem of inlet velocity determination, and the Burger equation. Incorporating weights generated in the first training phase neutralizes imbalance effects. Notably, the proposed approach outperforms Vanilla-PINN in terms of speed, convergence likelihood and eliminates the need for hyperparameter tuning to balance the loss function. [ABSTRACT FROM AUTHOR]

Partial Differential Equation (PDE) is among the most fundamental tools employed to model dynamic systems. Existing PDE modeling methods are typically derived from established knowledge and known phenomena, which are time-consuming and labor-intensive. Recently, discovering governing PDEs from collected actual data via Physics Informed Neural Networks (PINNs) provides a more efficient way to analyze fresh dynamic systems and establish PED models. This study proposes Sequentially Threshold Least Squares-Lasso (STLasso), a module constructed by incorporating Lasso regression into the Sequentially Threshold Least Squares (STLS) algorithm, which can complete sparse regression of PDE coefficients with the constraints of l norm. It further introduces PINN-STLasso, a physics informed neural network combined with Lasso sparse regression, able to find underlying PDEs from data with reduced data requirements and better interpretability. In addition, this research conducts experiments on canonical inverse PDE problems and compares the results to several recent methods. The results demonstrated that the proposed PINN-STLasso outperforms other methods, achieving lower error rates even with less data. [ABSTRACT FROM AUTHOR]

Learning operators with deep neural networks is an emerging paradigm for scientific computing. Deep Operator Network (DeepONet) is a modular operator learning framework that allows for flexibility in choosing the kind of neural network to be used in the trunk and/or branch of the DeepONet. This is beneficial as it has been shown many times that different types of problems require different kinds of network architectures for effective learning. In this work, we design an efficient neural operator based on the DeepONet architecture. We introduce U-Net enhanced DeepONet (U-DeepONet) for learning the solution operator of highly complex CO2-water two-phase flow in heterogeneous porous media. The U-DeepONet is more accurate in predicting gas saturation and pressure buildup than the state-of-the-art U-Net based Fourier Neural Operator (U-FNO) and the Fourier-enhanced Multiple-Input Operator (Fourier-MIONet) trained on the same dataset. Moreover, our U-DeepONet is significantly more efficient in training times than both the U-FNO (more than 18 times faster) and the Fourier-MIONet (more than 5 times faster), while consuming less computational resources. We also show that the U-DeepONet is more data efficient and better at generalization than both the U-FNO and the Fourier-MIONet. [ABSTRACT FROM AUTHOR]

We propose to enhance the training of physics-informed neural networks. To this aim, we introduce nonlinear additive and multiplicative preconditioning strategies for the widely used L-BFGS optimizer. The nonlinear preconditioners are constructed by utilizing the Schwarz domain decomposition framework, where the parameters of the network are decomposed in a layer-wise manner. Through a series of numerical experiments, we demonstrate that both additive and multiplicative preconditioners significantly improve the convergence of the standard L-BFGS optimizer while providing more accurate solutions of the underlying PDEs. Moreover, the additive preconditioner is inherently parallel, thus giving rise to a novel approach to model parallelism. [ABSTRACT FROM AUTHOR]

Deep neural networks (DNNs) and, in particular, convolutional neural networks (CNNs) have brought significant advances in a wide range of modern computer application problems. However, the increasing availability of large numbers of datasets and the increasing available computational power of modern computers have led to steady growth in the complexity and size of DNN and CNN models, respectively, and thus, to longer training times. Hence, various methods and attempts have been developed to accelerate and parallelize the training of complex network architectures. In this work, a novel CNN-DNN architecture is proposed that naturally supports a model parallel training strategy and that is loosely inspired by two-level domain decomposition methods (DDMs). First, local CNN models, that is, subnetworks, are defined that operate on overlapping or nonoverlapping parts of the input data, for example, subimages. The subnetworks can be trained completely in parallel and independently of each other. Each subnetwork then outputs a local decision for the given machine learning problem which is exclusively based on the respective local input data. Subsequently, in a second step, an additional DNN model is trained which evaluates the local decisions of the local subnetworks and generates a final, global decision. With respect to the analogy to DDMs, the DNN models can be loosely interpreted as a coarse problem and hence, the new approach can be interpreted as a two-level domain decomposition. In this paper, we apply the proposed architecture to image classification problems using CNNs. Experimental results for different two-dimensional image classification problems are provided, as well as a face recognition problem and a classification problem for three-dimensional computed tomography (CT) scans. Therefore, classical Residual Network (ResNet) and VGG architectures are considered. More modern architectures, such as, e.g., MobileNet2, are left for future work. The results show that the proposed approach can significantly accelerate the required training time compared to the global model and, additionally, can also help to improve the accuracy of the underlying classification problem. [ABSTRACT FROM AUTHOR]

By learning the mappings between infinite function spaces using carefully designed neural networks, the operator learning methodology has exhibited significantly more efficiency than traditional methods in solving differential equations, but faces concerns about their accuracy and reliability. To overcome these limitations through robustly enforcing boundary conditions (BCs), a general neural architecture named spectral operator learning is introduced by combining with the structures of the spectral numerical method. One variant called the orthogonal polynomial neural operator (OPNO) is proposed later, aiming at PDEs with Dirichlet, Neumann, and Robin BCs. The strict BC satisfaction properties and the universal approximation capacity of the OPNO are theoretically proven. A variety of numerical experiments with physical backgrounds demonstrate that the OPNO outperforms other existing deep learning methodologies, showcasing potential of comparable accuracy with the traditional second-order finite difference method that employs a considerably fine mesh (with relative errors on the order of 10–6 ), and is up to almost 5 magnitudes faster over the traditional method. [ABSTRACT FROM AUTHOR]

Supervised operator learning centers on the use of training data, in the form of inputoutput pairs, to estimate maps between infinite-dimensional spaces. It is emerging as a powerful tool to complement traditional scientific computing, which may often be framed in terms of operators mapping between spaces of functions. Building on the classical random features methodology for scalar regression, this paper introduces the function-valued random features method. This leads to a supervised operator learning architecture that is practical for nonlinear problems yet is structured enough to facilitate efficient training through the optimization of a convex, quadratic cost. Due to the quadratic structure, the trained model is equipped with convergence guarantees and error and complexity bounds, properties that are not readily available for most other operator learning architectures. At its core, the proposed approach builds a linear combination of random operators. This turns out to be a low-rank approximation of an operator-valued kernel ridge regression algorithm, and hence the method also has strong connections to Gaussian process regression. The paper designs function-valued random features that are tailored to the structure of two nonlinear operator learning benchmark problems arising from parametric partial differential equations. Numerical results demonstrate the scalability, discretization invariance, and transferability of the function-valued random features method. [ABSTRACT FROM AUTHOR]

Learning controllers from data for stabilizing dynamical systems typically follows a two-step process of first identifying a model and then constructing a controller based on the identified model. However, learning models means identifying generic descriptions of the dynamics of systems, which can require large amounts of data and extracting information that are unnecessary for the specific task of stabilization. The contribution of this work is to show that if a linear dynamical system has dimension (McMillan degree) n , then there always exist n states from which a stabilizing feedback controller can be constructed, independent of the dimension of the representation of the observed states and the number of inputs. By building on previous work, this finding implies that any linear dynamical system can be stabilized from fewer observed states than the minimal number of states required for learning a model of the dynamics. The theoretical findings are demonstrated with numerical experiments that show the stabilization of the flow behind a cylinder from less data than necessary for learning a model. [ABSTRACT FROM AUTHOR]

This work introduces Pontryagin Neural Networks (PoNNs), a specialised subset of Physics-Informed Neural Networks (PINNs) that aim to learn optimal control actions for optimal control problems (OCPs) characterised by integral quadratic cost functions. PoNNs employ the Pontryagin Minimum Principle (PMP) to establish necessary conditions for optimality, resulting in a two-point boundary value problem (TPBVP) that involves both state and costate variables within a system of ordinary differential equations (ODEs). By modelling the unknown solutions of the TPBVP with neural networks, PoNNs effectively learn the optimal control strategies. We also derive upper bounds on the generalisation error of PoNNs in solving these OCPs, taking into account the selection and quantity of training points along with the training error. To validate our theoretical analysis, we perform numerical experiments on benchmark linear and nonlinear OCPs. The results indicate that PoNNs can successfully learn open-loop control actions for the considered class of OCPs, outperforming the commercial software GPOPS-II in terms of both accuracy and computational efficiency. The reduced computational time suggests that PoNNs hold promise for real-time applications.

We introduce a machine-learning framework named statistics-informed neural network (SINN) for learning stochastic dynamics from data. This new architecture was theoretically inspired by a universal approximation theorem for stochastic systems, which we introduce in this paper, and the projection-operator formalism for stochastic modeling. We devise mechanisms for training the neural network model to reproduce the correct statistical behavior of a target stochastic process. Numerical simulation results demonstrate that a well-trained SINN can reliably approximate both Markovian and non-Markovian stochastic dynamics. We demonstrate the applicability of SINN to coarse-graining problems and the modeling of transition dynamics. Furthermore, we show that the obtained reduced-order model can be trained on temporally coarse-grained data and hence is well suited for rare-event simulations.

Julia is a general purpose programming language that was designed for simplifying and accelerating numerical analysis and computational science. In particular the Scientific Machine Learning (SciML) ecosystem of Julia packages includes frameworks for high-performance symbolic-numeric computations. It allows users to automatically enhance high-level descriptions of their models with symbolic preprocessing and automatic sparsification and parallelization of computations. This enables performant solution of differential equations, efficient parameter estimation and methodologies for automated model discovery with neural differential equations and sparse identification of nonlinear dynamics. To give the systems biology community easy access to SciML, we developed SBMLToolkit.jl. SBMLToolkit.jl imports dynamic SBML models into the SciML ecosystem to accelerate model simulation and fitting of kinetic parameters. By providing computational systems biologists with easy access to the open-source Julia ecosystevnm, we hope to catalyze the development of further Julia tools in this domain and the growth of the Julia bioscience community. SBMLToolkit.jl is freely available under the MIT license. The source code is available at https://github.com/SciML/SBMLToolkit.jl.