Search

In this paper, we provide a theoretical analysis of a type of operator learning method without data reliance based on the classical finite element approximation, which is called the finite element operator network (FEONet). We first establish the convergence of this method for general second-order linear elliptic PDEs with respect to the parameters for neural network approximation. In this regard, we address the role of the condition number of the finite element matrix in the convergence of the method. Secondly, we derive an explicit error estimate for the self-adjoint case. For this, we investigate some regularity properties of the solution in certain function classes for a neural network approximation, verifying the sufficient condition for the solution to have the desired regularity. Finally, we will also conduct some numerical experiments that support the theoretical findings, confirming the role of the condition number of the finite element matrix in the overall convergence.

Recent advancements in artificial intelligence (AI)-based design strategies for metamaterials have revolutionized the creation of customizable architectures spanning nano- to macro-scale dimensions, achieving unprecedented mechanical behaviors that surpass the inherent properties of the constituent materials. However, the growing complexity of these methods poses challenges in generating diverse metamaterials without substantial human and computational resources, hindering widespread adoption. Addressing this, our study introduces an innovative design strategy capable of generating various three-dimensional graph metamaterials using simple arithmetic operations within the latent space. By seamlessly integrating hidden representations of disentangled latent space and latent diffusion processes, our approach provides a comprehensive understanding of complex design spaces, generating diverse graph metamaterials through arithmetic operations. This methodology stands as a versatile tool for creating structures ranging from repetitive lattice structures to functionally graded mechanical metamaterials. It also serves as an inverse design strategy for diverse lattice structures, including crystalline structures and those made of trabecular bone. We believe that this methodology represents a foundational step in advancing our comprehension of the intricate latent design space, offering the potential to establish a unified model for various traditional generative models in the realm of mechanical metamaterials.

In this study, we introduce a method based on Separable Physics-Informed Neural Networks (SPINNs) for effectively solving the BGK model of the Boltzmann equation. While the mesh-free nature of PINNs offers significant advantages in handling high-dimensional partial differential equations (PDEs), challenges arise when applying quadrature rules for accurate integral evaluation in the BGK operator, which can compromise the mesh-free benefit and increase computational costs. To address this, we leverage the canonical polyadic decomposition structure of SPINNs and the linear nature of moment calculation, achieving a substantial reduction in computational expense for quadrature rule application. The multi-scale nature of the particle density function poses difficulties in precisely approximating macroscopic moments using neural networks. To improve SPINN training, we introduce the integration of Gaussian functions into SPINNs, coupled with a relative loss approach. This modification enables SPINNs to decay as rapidly as Maxwellian distributions, thereby enhancing the accuracy of macroscopic moment approximations. The relative loss design further ensures that both large and small-scale features are effectively captured by the SPINNs. The efficacy of our approach is demonstrated through a series of five numerical experiments, including the solution to a challenging 3D Riemann problem. These results highlight the potential of our novel method in efficiently and accurately addressing complex challenges in computational physics.

Singularly perturbed boundary value problems pose a significant challenge for their numerical approximations because of the presence of sharp boundary layers. These sharp boundary layers are responsible for the stiffness of solutions, which leads to large computational errors, if not properly handled. It is well-known that the classical numerical methods as well as the Physics-Informed Neural Networks (PINNs) require some special treatments near the boundary, e.g., using extensive mesh refinements or finer collocation points, in order to obtain an accurate approximate solution especially inside of the stiff boundary layer. In this article, we modify the PINNs and construct our new semi-analytic SL-PINNs suitable for singularly perturbed boundary value problems. Performing the boundary layer analysis, we first find the corrector functions describing the singular behavior of the stiff solutions inside boundary layers. Then we obtain the SL-PINN approximations of the singularly perturbed problems by embedding the explicit correctors in the structure of PINNs or by training the correctors together with the PINN approximations. Our numerical experiments confirm that our new SL-PINN methods produce stable and accurate approximations for stiff solutions., Comment: 22 pages, 6 figures

This research explores neural network-based numerical approximation of two-dimensional convection-dominated singularly perturbed problems on square, circular, and elliptic domains. Singularly perturbed boundary value problems pose significant challenges due to sharp boundary layers in their solutions. Additionally, the characteristic points of these domains give rise to degenerate boundary layer problems. The stiffness of these problems, caused by sharp singular layers, can lead to substantial computational errors if not properly addressed. Conventional neural network-based approaches often fail to capture these sharp transitions accurately, highlighting a critical flaw in machine learning methods. To address these issues, we conduct a thorough boundary layer analysis to enhance our understanding of sharp transitions within the boundary layers, guiding the application of numerical methods. Specifically, we employ physics-informed neural networks (PINNs) to better handle these boundary layer problems. However, PINNs may struggle with rapidly varying singularly perturbed solutions in small domain regions, leading to inaccurate or unstable results. To overcome this limitation, we introduce a semi-analytic method that augments PINNs with singular layers or corrector functions. Our numerical experiments demonstrate significant improvements in both accuracy and stability, showcasing the effectiveness of our proposed approach.

In this paper, we introduce the Spectral Coefficient Learning via Operator Network (SCLON), a novel operator learning-based approach for solving parametric partial differential equations (PDEs) without the need for data harnessing. The cornerstone of our method is the spectral methodology that employs expansions using orthogonal functions, such as Fourier series and Legendre polynomials, enabling accurate PDE solutions with fewer grid points. By merging the merits of spectral methods - encompassing high accuracy, efficiency, generalization, and the exact fulfillment of boundary conditions - with the prowess of deep neural networks, SCLON offers a transformative strategy. Our approach not only eliminates the need for paired input-output training data, which typically requires extensive numerical computations, but also effectively learns and predicts solutions of complex parametric PDEs, ranging from singularly perturbed convection-diffusion equations to the Navier-Stokes equations. The proposed framework demonstrates superior performance compared to existing scientific machine learning techniques, offering solutions for multiple instances of parametric PDEs without harnessing data. The mathematical framework is robust and reliable, with a well-developed loss function derived from the weak formulation, ensuring accurate approximation of solutions while exactly satisfying boundary conditions. The method's efficacy is further illustrated through its ability to accurately predict intricate natural behaviors like the Kolmogorov flow and boundary layers. In essence, our work pioneers a compelling avenue for parametric PDE solutions, serving as a bridge between traditional numerical methodologies and cutting-edge machine learning techniques in the realm of scientific computation., Comment: 28 pages, 8 figures

The recent surge in large-scale foundation models has spurred the development of efficient methods for adapting these models to various downstream tasks. Low-rank adaptation methods, such as LoRA, have gained significant attention due to their outstanding parameter efficiency and no additional inference latency. This paper investigates a more general form of adapter module based on the analysis that parallel and sequential adaptation branches learn novel and general features during fine-tuning, respectively. The proposed method, named Hydra, due to its multi-head computational branches, combines parallel and sequential branch to integrate capabilities, which is more expressive than existing single branch methods and enables the exploration of a broader range of optimal points in the fine-tuning process. In addition, the proposed adaptation method explicitly leverages the pre-trained weights by performing a linear combination of the pre-trained features. It allows the learned features to have better generalization performance across diverse downstream tasks. Furthermore, we perform a comprehensive analysis of the characteristics of each adaptation branch with empirical evidence. Through an extensive range of experiments, encompassing comparisons and ablation studies, we substantiate the efficiency and demonstrate the superior performance of Hydra. This comprehensive evaluation underscores the potential impact and effectiveness of Hydra in a variety of applications. Our code is available on \url{https://github.com/extremebird/Hydra}

Partial differential equations (PDEs) underlie our understanding and prediction of natural phenomena across numerous fields, including physics, engineering, and finance. However, solving parametric PDEs is a complex task that necessitates efficient numerical methods. In this paper, we propose a novel approach for solving parametric PDEs using a Finite Element Operator Network (FEONet). Our proposed method leverages the power of deep learning in conjunction with traditional numerical methods, specifically the finite element method, to solve parametric PDEs in the absence of any paired input-output training data. We performed various experiments on several benchmark problems and confirmed that our approach has demonstrated excellent performance across various settings and environments, proving its versatility in terms of accuracy, generalization, and computational flexibility. Our FEONet framework shows potential for application in various fields where PDEs play a crucial role in modeling complex domains with diverse boundary conditions and singular behavior. Furthermore, we provide theoretical convergence analysis to support our approach, utilizing finite element approximation in numerical analysis., Comment: 23 pages, 11 figures

The purpose of this research is to devise a tactic that can closely track the daily cumulative volume-weighted average price (VWAP) using reinforcement learning. Previous studies often choose a relatively short trading horizon to implement their models, making it difficult to accurately track the daily cumulative VWAP since the variations of financial data are often insignificant within the short trading horizon. In this paper, we aim to develop a strategy that can accurately track the daily cumulative VWAP while minimizing the deviation from the VWAP. We propose a method that leverages the U-shaped pattern of intraday stock trade volumes and use Proximal Policy Optimization (PPO) as the learning algorithm. Our method follows a dual-level approach: a Transformer model that captures the overall(global) distribution of daily volumes in a U-shape, and a LSTM model that handles the distribution of orders within smaller(local) time intervals. The results from our experiments suggest that this dual-level architecture improves the accuracy of approximating the cumulative VWAP, when compared to previous reinforcement learning-based models., Comment: Submitted to Expert Systems with Applications (Under 2nd review)

Physics-informed neural networks (PINNs) have recently emerged as promising data-driven PDE solvers showing encouraging results on various PDEs. However, there is a fundamental limitation of training PINNs to solve multi-dimensional PDEs and approximate highly complex solution functions. The number of training points (collocation points) required on these challenging PDEs grows substantially, but it is severely limited due to the expensive computational costs and heavy memory overhead. To overcome this issue, we propose a network architecture and training algorithm for PINNs. The proposed method, separable PINN (SPINN), operates on a per-axis basis to significantly reduce the number of network propagations in multi-dimensional PDEs unlike point-wise processing in conventional PINNs. We also propose using forward-mode automatic differentiation to reduce the computational cost of computing PDE residuals, enabling a large number of collocation points (>10^7) on a single commodity GPU. The experimental results show drastically reduced computational costs (62x in wall-clock time, 1,394x in FLOPs given the same number of collocation points) in multi-dimensional PDEs while achieving better accuracy. Furthermore, we present that SPINN can solve a chaotic (2+1)-d Navier-Stokes equation significantly faster than the best-performing prior method (9 minutes vs 10 hours in a single GPU), maintaining accuracy. Finally, we showcase that SPINN can accurately obtain the solution of a highly nonlinear and multi-dimensional PDE, a (3+1)-d Navier-Stokes equation. For visualized results and code, please see https://jwcho5576.github.io/spinn.github.io/., Comment: To appear in NeurIPS 2023 (28 pages, 13 figures). workshop paper: arXiv:2211.08761

In this paper, we perform the convergence analysis of unsupervised Legendre--Galerkin neural networks (ULGNet), a deep-learning-based numerical method for solving partial differential equations (PDEs). Unlike existing deep learning-based numerical methods for PDEs, the ULGNet expresses the solution as a spectral expansion with respect to the Legendre basis and predicts the coefficients with deep neural networks by solving a variational residual minimization problem. Since the corresponding loss function is equivalent to the residual induced by the linear algebraic system depending on the choice of basis functions, we prove that the minimizer of the discrete loss function converges to the weak solution of the PDEs. Numerical evidence will also be provided to support the theoretical result. Key technical tools include the variant of the universal approximation theorem for bounded neural networks, the analysis of the stiffness and mass matrices, and the uniform law of large numbers in terms of the Rademacher complexity.

Physics-informed neural networks (PINNs) have emerged as new data-driven PDE solvers for both forward and inverse problems. While promising, the expensive computational costs to obtain solutions often restrict their broader applicability. We demonstrate that the computations in automatic differentiation (AD) can be significantly reduced by leveraging forward-mode AD when training PINN. However, a naive application of forward-mode AD to conventional PINNs results in higher computation, losing its practical benefit. Therefore, we propose a network architecture, called separable PINN (SPINN), which can facilitate forward-mode AD for more efficient computation. SPINN operates on a per-axis basis instead of point-wise processing in conventional PINNs, decreasing the number of network forward passes. Besides, while the computation and memory costs of standard PINNs grow exponentially along with the grid resolution, that of our model is remarkably less susceptible, mitigating the curse of dimensionality. We demonstrate the effectiveness of our model in various PDE systems by significantly reducing the training run-time while achieving comparable accuracy. Project page: https://jwcho5576.github.io/spinn/, Comment: To appear in NeurIPS 2022 Workshop on The Symbiosis of Deep Learning and Differential Equations (DLDE) - II, 12 pages, 5 figures, full paper: arXiv:2306.15969

Normalizing flows model probability distributions by learning invertible transformations that transfer a simple distribution into complex distributions. Since the architecture of ResNet-based normalizing flows is more flexible than that of coupling-based models, ResNet-based normalizing flows have been widely studied in recent years. Despite their architectural flexibility, it is well-known that the current ResNet-based models suffer from constrained Lipschitz constants. In this paper, we propose the monotone formulation to overcome the issue of the Lipschitz constants using monotone operators and provide an in-depth theoretical analysis. Furthermore, we construct an activation function called Concatenated Pila (CPila) to improve gradient flow. The resulting model, Monotone Flows, exhibits an excellent performance on multiple density estimation benchmarks (MNIST, CIFAR-10, ImageNet32, ImageNet64). Code is available at https://github.com/mlvlab/MonotoneFlows., Comment: NeurIPS 2022

Predicting volatility is important for asset predicting, option pricing and hedging strategies because it cannot be directly observed in the financial market. The Black-Scholes option pricing model is one of the most widely used models by market participants. Notwithstanding, the Black-Scholes model is based on heavily criticized theoretical premises, one of which is the constant volatility assumption. The dynamics of the volatility surface is difficult to estimate. In this paper, we establish a novel architecture based on physics-informed neural networks and convolutional transformers. The performance of the new architecture is directly compared to other well-known deep-learning architectures, such as standard physics-informed neural networks, convolutional long-short term memory (ConvLSTM), and self-attention ConvLSTM. Numerical evidence indicates that the proposed physics-informed convolutional transformer network achieves a superior performance than other methods., Comment: Submitted to Quantitative Finance (under final review); Revised Version

We propose a new semi-analytic physics informed neural network (PINN) to solve singularly perturbed boundary value problems. The PINN is a scientific machine learning framework that offers a promising perspective for finding numerical solutions to partial differential equations. The PINNs have shown impressive performance in solving various differential equations including time-dependent and multi-dimensional equations involved in a complex geometry of the domain. However, when considering stiff differential equations, neural networks in general fail to capture the sharp transition of solutions, due to the spectral bias. To resolve this issue, here we develop the semi-analytic PINN methods, enriched by using the so-called corrector functions obtained from the boundary layer analysis. Our new enriched PINNs accurately predict numerical solutions to the singular perturbation problems. Numerical experiments include various types of singularly perturbed linear and nonlinear differential equations., Comment: 19 pages

With the increases in computational power and advances in machine learning, data-driven learning-based methods have gained significant attention in solving PDEs. Physics-informed neural networks (PINNs) have recently emerged and succeeded in various forward and inverse PDE problems thanks to their excellent properties, such as flexibility, mesh-free solutions, and unsupervised training. However, their slower convergence speed and relatively inaccurate solutions often limit their broader applicability in many science and engineering domains. This paper proposes a new kind of data-driven PDEs solver, physics-informed cell representations (PIXEL), elegantly combining classical numerical methods and learning-based approaches. We adopt a grid structure from the numerical methods to improve accuracy and convergence speed and overcome the spectral bias presented in PINNs. Moreover, the proposed method enjoys the same benefits in PINNs, e.g., using the same optimization frameworks to solve both forward and inverse PDE problems and readily enforcing PDE constraints with modern automatic differentiation techniques. We provide experimental results on various challenging PDEs that the original PINNs have struggled with and show that PIXEL achieves fast convergence speed and high accuracy. Project page: https://namgyukang.github.io/PIXEL/, Comment: Accepted to the 37th AAAI Conference on Artificial Intelligence (AAAI 2023) Main Track, DLDE-II NeurIPS 2022 Workshop (Spotlight), https://namgyukang.github.io/PIXEL/ (Project Page), https://github.com/NamGyuKang/CosineSampler (Customized CUDA) (17 pages, 17 figures)

Machine learning methods have been lately used to solve partial differential equations (PDEs) and dynamical systems. These approaches have been developed into a novel research field known as scientific machine learning in which techniques such as deep neural networks and statistical learning are applied to classical problems of applied mathematics. In this paper, we develop a novel numerical algorithm that incorporates machine learning and artificial intelligence to solve PDEs. Based on the Legendre-Galerkin framework, we propose the {\it unsupervised machine learning} algorithm to learn {\it multiple instances} of the solutions for different types of PDEs. Our approach overcomes the limitations of data-driven and physics-based methods. The proposed neural network is applied to general 1D and 2D PDEs with various boundary conditions as well as convection-dominated {\it singularly perturbed PDEs} that exhibit strong boundary layer behavior., Comment: 16 pages, 8 figures

Recent advances in objective-based uncertainty quantification (objective-UQ) have shown that such a goal-driven approach for quantifying model uncertainty is extremely useful in real-world problems that aim at achieving specific objectives based on complex uncertain systems. Central to this objective-UQ is the concept of mean objective cost of uncertainty (MOCU), which provides effective means of quantifying the impact of uncertainty on the operational goals at hand. MOCU is especially useful for optimal experimental design (OED) as the potential efficacy of an experimental (or data acquisition) campaign can be quantified by estimating the MOCU that is expected to remain after the campaign. However, MOCU-based OED tends to be computationally expensive, which limits its practical applicability. In this paper, we propose a novel machine learning (ML) scheme that can significantly accelerate MOCU computation and expedite MOCU-based experimental design. The main idea is to use an ML model to efficiently search for the optimal robust operator under model uncertainty, a necessary step for computing MOCU. We apply the proposed ML-based OED acceleration scheme to design experiments aimed at optimally enhancing the control performance of uncertain Kuramoto oscillator models. Our results show that the proposed scheme results in up to 154-fold speed improvement without any degradation of the OED performance.

Nonlinear differential equations are challenging to solve numerically and are important to understanding the dynamics of many physical systems. Deep neural networks have been applied to help alleviate the computational cost that is associated with solving these systems. We explore the performance and accuracy of various neural architectures on both linear and nonlinear differential equations by creating accurate training sets with the spectral element method. Next, we implement a novel Legendre-Galerkin Deep Neural Network (LGNet) algorithm to predict solutions to differential equations. By constructing a set of a linear combination of the Legendre basis, we predict the corresponding coefficients, $\alpha_i$ which successfully approximate the solution as a sum of smooth basis functions $u \simeq \sum_{i=0}^{N} \alpha_i \varphi_i$. As a computational example, linear and nonlinear models with Dirichlet or Neumann boundary conditions are considered., Comment: 19 pages, 7 figures, 1 table

Deep neural networks have achieved state-of-the-art performance in a variety of fields. Recent works observe that a class of widely used neural networks can be viewed as the Euler method of numerical discretization. From the numerical discretization perspective, Strong Stability Preserving (SSP) methods are more advanced techniques than the explicit Euler method that produce both accurate and stable solutions. Motivated by the SSP property and a generalized Runge-Kutta method, we propose Strong Stability Preserving networks (SSP networks) which improve robustness against adversarial attacks. We empirically demonstrate that the proposed networks improve the robustness against adversarial examples without any defensive methods. Further, the SSP networks are complementary with a state-of-the-art adversarial training scheme. Lastly, our experiments show that SSP networks suppress the blow-up of adversarial perturbations. Our results open up a way to study robust architectures of neural networks leveraging rich knowledge from numerical discretization literature., Comment: ECCV 2020

We consider the optimal experimental design problem for an uncertain Kuramoto model, which consists of N interacting oscillators described by coupled ordinary differential equations. The objective is to design experiments that can effectively reduce the uncertainty present in the coupling strengths between the oscillators, thereby minimizing the cost of robust control of the uncertain Kuramoto model. We demonstrate the importance of quantifying the operational impact of the potential experiments in designing optimal experiments.

It is well known that the solution of the 3d Navier--Stokes equations remains bounded if the initial data and the forcing are sufficiently small relative to the viscosity, and for a finite time given any bounded initial data. In this article, we consider two temporal discretisations (semi-implicit and fully implicit) of the 3d Navier--Stokes equations in a periodic domain and prove that their solutions remain bounded in $H^1$ subject to essentially the same smallness conditions (on initial data, forcing or time) as the continuous system and to suitable timestep restrictions., Comment: 12 pages

We aim to study a finite volume scheme to solve the two dimensional inviscid primitive equations of the atmosphere with humidity and saturation, in presence of topography and subject to physically plausible boundary conditions to the system of equations. In that respect, a version of a projection method is introduced to enforce the compatibility condition on the horizontal velocity field, which comes from the boundary conditions. The resulting scheme allows for a significant reduction of the errors near the topography when compared to more standard finite volume schemes. In the numerical simulations, we first present the associated good convergence results that are satisfied by the solutions simulated by our scheme when compared to particular analytic solutions. We then report on numerical experiments using realistic parameters. Finally, the effects of a random small-scale forcing on the velocity equation is numerically investigated. The numerical results show that such a forcing is responsible for recurrent large-scale patterns to emerge in the temperature and velocity fields., Comment: 34 pages, 18 figures

In this article we study the two dimensional singularly perturbed heat equation in a circular domain. The aim is to develop a numerical method with a uniform mesh, avoiding mesh refinement at the boundary thanks to the use of a relatively simple representation of the boundary layer. We provide the asymptotic expansion of the solution at first order and derive the boundary layer element resulting from the boundary layer analysis. We then perform the convergence analysis introducing the boundary layer element in the finite element space thus obtaining what is called an "enriched Galerkin space". Finally we present and comment on numerical simulations using a quasi-uniform grid and the modified finite element method., Comment: 26 pages, 12 figures