Search

Solving partial differential equations (PDEs) is a fundamental problem in engineering and science. While neural PDE solvers can be more efficient than established numerical solvers, they often require large amounts of training data that is costly to obtain. Active Learning (AL) could help surrogate models reach the same accuracy with smaller training sets by querying classical solvers with more informative initial conditions and PDE parameters. While AL is more common in other domains, it has yet to be studied extensively for neural PDE solvers. To bridge this gap, we introduce AL4PDE, a modular and extensible active learning benchmark. It provides multiple parametric PDEs and state-of-the-art surrogate models for the solver-in-the-loop setting, enabling the evaluation of existing and the development of new AL methods for PDE solving. We use the benchmark to evaluate batch active learning algorithms such as uncertainty- and feature-based methods. We show that AL reduces the average error by up to 71% compared to random sampling and significantly reduces worst-case errors. Moreover, AL generates similar datasets across repeated runs, with consistent distributions over the PDE parameters and initial conditions. The acquired datasets are reusable, providing benefits for surrogate models not involved in the data generation., Comment: Code will be made available at https://github.com/dmusekamp/al4pde

Machine learning plays an increasingly important role in computational chemistry and materials science, complementing computationally intensive ab initio and first-principles methods. Despite their utility, machine-learning models often lack generalization capability and robustness during atomistic simulations, yielding unphysical energy and force predictions that hinder their real-world applications. We address this challenge by introducing a physics-informed, weakly supervised approach for training machine-learned interatomic potentials (MLIPs). We introduce two novel loss functions, extrapolating the potential energy via a Taylor expansion and using the concept of conservative forces. Our approach improves the accuracy of MLIPs applied to training tasks with sparse training data sets and reduces the need for pre-training computationally demanding models with large data sets. Particularly, we perform extensive experiments demonstrating reduced energy and force errors -- often lower by a factor of two -- for various baseline models and benchmark data sets. Finally, we show that our approach facilitates MLIPs' training in a setting where the computation of forces is infeasible at the reference level, such as those employing complete-basis-set extrapolation., Comment: 24 pages, 2 figures, 18 Tables

The ability to perform fast and accurate atomistic simulations is crucial for advancing the chemical sciences. By learning from high-quality data, machine-learned interatomic potentials achieve accuracy on par with ab initio and first-principles methods at a fraction of their computational cost. The success of machine-learned interatomic potentials arises from integrating inductive biases such as equivariance to group actions on an atomic system, e.g., equivariance to rotations and reflections. In particular, the field has notably advanced with the emergence of equivariant message passing. Most of these models represent an atomic system using spherical tensors, tensor products of which require complicated numerical coefficients and can be computationally demanding. Cartesian tensors offer a promising alternative, though state-of-the-art methods lack flexibility in message-passing mechanisms, restricting their architectures and expressive power. This work explores higher-rank irreducible Cartesian tensors to address these limitations. We integrate irreducible Cartesian tensor products into message-passing neural networks and prove the equivariance and traceless property of the resulting layers. Through empirical evaluations on various benchmark data sets, we consistently observe on-par or better performance than that of state-of-the-art spherical and Cartesian models., Comment: Accepted at NeurIPS 2024 (camera-ready version)

Efficiently creating a concise but comprehensive data set for training machine-learned interatomic potentials (MLIPs) is an under-explored problem. Active learning, which uses biased or unbiased molecular dynamics (MD) to generate candidate pools, aims to address this objective. Existing biased and unbiased MD-simulation methods, however, are prone to miss either rare events or extrapolative regions -- areas of the configurational space where unreliable predictions are made. This work demonstrates that MD, when biased by the MLIP's energy uncertainty, simultaneously captures extrapolative regions and rare events, which is crucial for developing uniformly accurate MLIPs. Furthermore, exploiting automatic differentiation, we enhance bias-forces-driven MD with the concept of bias stress. We employ calibrated gradient-based uncertainties to yield MLIPs with similar or, sometimes, better accuracy than ensemble-based methods at a lower computational cost. Finally, we apply uncertainty-biased MD to alanine dipeptide and MIL-53(Al), generating MLIPs that represent both configurational spaces more accurately than models trained with conventional MD.

Scientific Machine Learning (SciML) is concerned with the development of learned emulators of physical systems governed by partial differential equations (PDE). In application domains such as weather forecasting, molecular dynamics, and inverse design, ML-based surrogate models are increasingly used to augment or replace inefficient and often non-differentiable numerical simulation algorithms. While a number of ML-based methods for approximating the solutions of PDEs have been proposed in recent years, they typically do not adapt to the parameters of the PDEs, making it difficult to generalize to PDE parameters not seen during training. We propose a Channel Attention mechanism guided by PDE Parameter Embeddings (CAPE) component for neural surrogate models and a simple yet effective curriculum learning strategy. The CAPE module can be combined with neural PDE solvers allowing them to adapt to unseen PDE parameters. The curriculum learning strategy provides a seamless transition between teacher-forcing and fully auto-regressive training. We compare CAPE in conjunction with the curriculum learning strategy using a popular PDE benchmark and obtain consistent and significant improvements over the baseline models. The experiments also show several advantages of CAPE, such as its increased ability to generalize to unseen PDE parameters without large increases inference time and parameter count., Comment: accepted for publication in ICML2023

Machine learning-based modeling of physical systems has experienced increased interest in recent years. Despite some impressive progress, there is still a lack of benchmarks for Scientific ML that are easy to use but still challenging and representative of a wide range of problems. We introduce PDEBench, a benchmark suite of time-dependent simulation tasks based on Partial Differential Equations (PDEs). PDEBench comprises both code and data to benchmark the performance of novel machine learning models against both classical numerical simulations and machine learning baselines. Our proposed set of benchmark problems contribute the following unique features: (1) A much wider range of PDEs compared to existing benchmarks, ranging from relatively common examples to more realistic and difficult problems; (2) much larger ready-to-use datasets compared to prior work, comprising multiple simulation runs across a larger number of initial and boundary conditions and PDE parameters; (3) more extensible source codes with user-friendly APIs for data generation and baseline results with popular machine learning models (FNO, U-Net, PINN, Gradient-Based Inverse Method). PDEBench allows researchers to extend the benchmark freely for their own purposes using a standardized API and to compare the performance of new models to existing baseline methods. We also propose new evaluation metrics with the aim to provide a more holistic understanding of learning methods in the context of Scientific ML. With those metrics we identify tasks which are challenging for recent ML methods and propose these tasks as future challenges for the community. The code is available at https://github.com/pdebench/PDEBench., Comment: 16 pages (main body) + 34 pages (supplemental material), accepted for publication in NeurIPS 2022 Track Datasets and Benchmarks

Open Information Extraction (OpenIE) is the task of extracting (subject, predicate, object) triples from natural language sentences. Current OpenIE systems extract all triple slots independently. In contrast, we explore the hypothesis that it may be beneficial to extract triple slots iteratively: first extract easy slots, followed by the difficult ones by conditioning on the easy slots, and therefore achieve a better overall extraction. Based on this hypothesis, we propose a neural OpenIE system, milIE, that operates in an iterative fashion. Due to the iterative nature, the system is also modular -- it is possible to seamlessly integrate rule based extraction systems with a neural end-to-end system, thereby allowing rule based systems to supply extraction slots which milIE can leverage for extracting the remaining slots. We confirm our hypothesis empirically: milIE outperforms SOTA systems on multiple languages ranging from Chinese to Arabic. Additionally, we are the first to provide an OpenIE test dataset for Arabic and Galician.

It is well-known that training of generative adversarial networks (GANs) requires huge iterations before the generator's providing good-quality samples. Although there are several studies to tackle this problem, there is still no universal solution. In this paper, we investigated the effect of sample mixing methods, that is, Mixup, CutMix, and newly proposed Smoothed Regional Mix (SRMix), to alleviate this problem. The sample-mixing methods are known to enhance the accuracy and robustness in the wide range of classification problems, and can naturally be applicable to GANs because the role of the discriminator can be interpreted as the classification between real and fake samples. We also proposed a new formalism applying the sample-mixing methods to GANs with the saturated losses which do not have a clear "label" of real and fake. We performed a vast amount of numerical experiments using LSUN and CelebA datasets. The results showed that Mixup and SRMix improved the quality of the generated images in terms of FID in most cases, in particular, SRMix showed the best improvement in most cases. Our analysis indicates that the mixed-samples can provide different properties from the vanilla fake samples, and the mixing pattern strongly affects the decision of the discriminators. The generated images of Mixup have good high-level feature but low-level feature is not so impressible. On the other hand, CutMix showed the opposite tendency. Our SRMix showed the middle tendency, that is, showed good high and low level features. We believe that our finding provides a new perspective to accelerate the GANs convergence and improve the quality of generated samples., Comment: draft version, accepted by IEEE 4th International Conference on Multimedia Information Processing and Retrieval (IEEE MIPR 2021)

Compressing deep neural network (DNN) models becomes a very important and necessary technique for real-world applications, such as deploying those models on mobile devices. Knowledge distillation is one of the most popular methods for model compression, and many studies have been made on developing this technique. However, those studies mainly focused on classification problems, and very few attempts have been made on regression problems, although there are many application of DNNs on regression problems. In this paper, we propose a new formalism of knowledge distillation for regression problems. First, we propose a new loss function, teacher outlier rejection loss, which rejects outliers in training samples using teacher model predictions. Second, we consider a multi-task network with two outputs: one estimates training labels which is in general contaminated by noisy labels; And the other estimates teacher model's output which is expected to modify the noise labels following the memorization effects. By considering the multi-task network, training of the feature extraction of student models becomes more effective, and it allows us to obtain a better student model than one trained from scratch. We performed comprehensive evaluation with one simple toy model: sinusoidal function, and two open datasets: MPIIGaze, and Multi-PIE. Our results show consistent improvement in accuracy regardless of the annotation error level in the datasets., Comment: 7 pages, 2 figures, draft version of a paper accepted for IEEE 3rd International Conference on Multimedia Information Processing and Retrieval (MIPR2020)

In this paper, we report our recent findings on the relativistic Weibel instability and its nonlinear saturation by performing numerical simulations of collisionless plasmas. Analysis of the obtained numerical results revealed that the nonlinear phase of the Weibel instability can be described by characteristic phases based on the Weibel filaments' current density in terms of particle and Alfv\'en limit currents. We also analyzed the relativistic kink instability based on the energy principle in the magnetohydrodynamic (MHD) regime, and found that the Weibel filaments do not suffer from the kink-type instability in the MHD regime up to 1000 $\omega_{p,i}^{-1}$. This finding allowed a magnetic field to be sustained by relativistic Weibel instability that was stable enough to be a seed for MHD dynamos., Comment: 15 pages, 8 figures, accepted for publication in ApJ

The time evolution and saturation of the Weibel instability at the ion Alfv\'en current are presented by ab initio particle-in-cell simulations. We found that the ion Weibel current in 3D could evolve into the Alfv\'en current where the magnetic field energy is sustained at 1.5\% of the initial beam kinetic energy. The current filaments are no longer isolated at saturation, but rather connected to each other to form a network structure. Electrons are continuously heated during the coalescence of the filaments, which is crucial for obtaining sustained magnetic fields with much stronger levels than with 2D simulations. The results highlight again the importance of the Weibel instability in generating magnetic fields in laboratory, astrophysical, and cosmological situations., Comment: 4 pages, 4 figures, accepted for publication in ApJL

In this paper, the temporal evolution of 3-dimensional relativistic current sheets in Poynting-dominated plasma is studied for the first time. Over the past few decades, a lot of efforts have been conducted on studying the evolution of current sheets in 2-dimensional space, and concluded that sufficiently long current sheets always evolves into the so-called "plasmoid-chain", which provides fast reconnection rate independent of its resistivity. However, it is suspected that plasmoid-chain can exist only in the case of 2-dimensional approximation, and would show transition to turbulence in 3-dimensional space. We performed 3-dimensional numerical simulation of relativistic current sheet using resistive relativistic magnetohydrodynamic approximation. The results showed that the 3-dimensional current sheet evolve not into plasmoid-chain but turbulence. The resulting reconnection rate is $0.004$ which is much smaller than that of plasmoid-chain. The energy conversion from magnetic field to kinetic energy of turbulence is just 0.01\% which is much smaller than typical non-relativistic cases. Using the energy principle, we also showed that the plasmoid is always unstable for a displacement in opposite direction to its acceleration, probably interchange-type instability, and this always results in seeds of turbulence behind the plasmoids. Finally, the temperature distribution along the sheet is discussed, and it is found that the sheet is less active than plasmoid-chain. Our finding can be applied for many high energy astrophysical phenomena, and can provide a basic model of the general current sheet in Poynting-dominated plasma., Comment: 9 pages, 10 figures, accepted for publication in MNRAS

In this paper, we report our detailed analysis of the new strong-coupling regime between Alfv\'en and fast modes in Poynting-dominated plasma turbulence, reported in our previous work Takamoto & Lazarian (2016), which is an important effect for many relativistic plasma phenomena, and calls for new theories of Poynting-dominated MHD turbulence. We performed numerical simulations of relativistic MHD turbulence in isothermal plasmas, and analyzed the ratio of fast to Alfv\'en mode energy. We found that the increase of the fast mode with the background $\sigma$-parameter can be observed even in isothermal plasma, showing that such a phenomena is universal in trans-Alfv\'enic turbulence in Poynting-dominated plasmas. To study the detailed energy conversion process, we also performed a series of simulations of decaying turbulence injecting pure Alfv\'en, fast, and slow modes, respectively, and investigated the development of the mode conversion from the each mode. We also found that the mode conversion between Alfv\'en and fast modes is nearly insensitive to the background temperature. Finally, we report a result of a simulation with initially fast mode dominated turbulence. It developed into a temporally strong-coupling regime, which is a strong evidence for the existence of our suggesting strong-coupling regime of fast and Alfv\'en modes. Our result suggests that the strong turbulence in Poynting-dominated plasma is very different from that in the non-relativistic plasma. It will also give an important guidance to studies of particle acceleration and non-thermal photon emission from Poynting-dominated plasma., Comment: 10 pages, 9 figures, accepted for publication in MNRAS

In this Letter, we report compressible mode effects on relativistic magnetohydrodynamic (RMHD) turbulence in Poynting-dominated plasmas using 3-dimensional numerical simulations. We decomposed fluctuations in the turbulence into 3 MHD modes (fast, slow, and Alfv\'en) following the procedure mode decomposition in (Cho & Lazarian 2002), and analyzed their energy spectra and structure functions separately. We also analyzed the ratio of compressible mode to Alfv\'en mode energy with respect to its Mach number. We found the ratio of compressible mode increases not only with the Alfv\'en Mach number but with the background magnetization, which indicates a strong coupling between the fast and Alfv\'en modes and appearance of a new regime of RMHD turbulence in Poynting-dominated plasmas where the fast and Alfv\'en modes strongly couples and cannot be separated, different from the non-relativistic MHD case. This finding will affect particle acceleration efficiency obtained by assuming Alfv\'enic critical balance turbulence, and will change the resulting photon spectra by non-thermal electrons., Comment: 5 pages, 3 figures, accepted for publication in ApJL

We report turbulence effects on magnetic reconnection in relativistic plasmas using 3-dimensional relativistic resistive magnetohydrodynamics simulations. We found reconnection rate became independent of the plasma resistivity due to turbulence effects similarly to non-relativistic cases. We also found compressible turbulence effects modified the turbulent reconnection rate predicted in non-relativistic incompressible plasmas; The reconnection rate saturates and even decays as the injected velocity approaches to the Alfv\'en velocity. Our results indicate the compressibility cannot be neglected when compressible component becomes about half of incompressible mode occurring when the Alfv\'en Mach number reaches about $0.3$. The obtained maximum reconnection rate is around $0.05$ to $0.1$, which will be able to reach around $0.1$ to $0.2$ if injection scales are comparable to the sheet length., Comment: 8 pages, 6 figures, accepted for publication in ApJ

We study the magneto-hydrodynamic tearing instability occurring in a double current sheet configuration when a guide field is present. This is investigated by means of resistive relativistic magneto-hydrodynamic (RRMHD) simulations. Following the dynamics of the double tearing mode (DTM), we are able to compute synthetic synchrotron spectra in the explosive reconnection phase. The pulsar striped wind model represents a site where such current sheets are formed, including a guide field. The variability of the Crab nebula/pulsar system, seen as flares, can be therefore naturally explained by the DTM explosive phase in the striped wind. Our results indicate that the Crab GeV flare can be explained by the double tearing mode in the striped wind region if the magnetization parameter $\sigma$ is around $10^5$., Comment: 10 pages, 12 figures, accepted for publication in MNRAS

Perpendicular shocks are shown to be rapid particle accelerators that perform optimally when the ratio $u_{\rm s}$ of the shock speed to the particle speed roughly equals the ratio $1/\eta$ of the scattering rate to the gyro frequency. We use analytical methods and Monte-Carlo simulations to solve the kinetic equation that governs the anisotropy generated at these shocks, and find, for $\eta u_{\rm s}\approx1$, that the spectral index softens by unity and the acceleration time increases by a factor of two compared to the standard result of diffusive shock acceleration theory. These results provide a theoretical basis for the thirty-year-old conjecture that a supernova exploding into the wind of a Wolf-Rayet star may accelerate protons to an energy exceeding $10^{15}\,$eV., Comment: 12 pages, 2 figures, accepted for publication in ApJ

In this paper we propose a new mechanism describing the initial spike of giant flares in the framework of the starquake model. We investigate the evolution of a plasma on a closed magnetic flux tube in the magnetosphere of a magnetar in the case of a sudden energy release and discuss the relationship with observations of giant flares. We perform one-dimensional numerical simulations of the relativistic magnetohydrodynamics in Schwarzschild geometry. We assume energy is injected at the footpoints of the loop by a hot star surface containing random perturbations of the transverse velocity. Alfv\'en waves are generated and propagate upward, accompanying very hot plasma that is also continuously heated by nonlinearly generated compressive waves. We find that the front edges of the fireball regions collide at the top of the tube with their symmetrically launched counterparts. This collision results in an energy release which can describe the light curve of initial spikes of giant flares., Comment: 13 pages, 11 figures, accepted for publication in ApJ

In this paper, we investigate the evolution of the plasmoid-chain in a Poynting-dominated plasma. We model the relativistic current sheet with cold background plasma using the relativistic resistive magnetohydrodynamic approximation, and solve its temporal evolution numerically. We perform various calculations using different magnetization parameters of the background plasma and different Lundquist numbers. Numerical results show that the initially induced plasmoid triggers a secondary tearing instability, which gradually fills the current sheet with plasmoids, as has also been observed in the non-relativistic case. We find the plasmoid-chain greatly enhances the reconnection rate, which becomes independent of the Lundquist number, when this exceeds a critical value. In addition, we show the distribution of plasmoid size becomes a power law. Since magnetic reconnection is expected to play an important role in various high energy astrophysical phenomena, our results can be used for explaining the physical mechanism of them., Comment: 10 pages, 9 figures, accepted for publication in ApJ

In this article, we present a state-of-the-art algorithm for solving the relativistic viscous hydrodynamics equation with the QCD equation of state. The numerical method is based on the second-order Godunov method and has less numerical dissipation, which is crucial in describing of quark-gluon plasma in high-energy heavy-ion collisions. We apply the algorithm to several numerical test problems such as sound wave propagation, shock tube and blast wave problems. In sound wave propagation, the intrinsic numerical viscosity is measured and its explicit expression is shown, which is the second-order of spatial resolution both in the presence and absence of physical viscosity. The expression of the numerical viscosity can be used to determine the maximum cell size in order to accurately measure the effect of physical viscosity in the numerical simulation., Comment: 38pages, 31 figures; published version

Stationary pulsar magnetospheres in the force-free system are governed by the pulsar equation. In 1999, Contopoulos, Kazanas, and Fendt (hereafter CKF) numerically solved the pulsar equation and obtained a pulsar magnetosphere model called the CKF solution that has both closed and open magnetic field lines. The CKF solution is a successful solution, but it contains a poloidal current sheet that flows along the last open field line. This current sheet is artificially added to make the current system closed. In this paper, we suggest an alternative method to solve the pulsar equation and construct pulsar magnetosphere models without a current sheet. In our method, the pulsar equation is decomposed into Ampere's law and the force-free condition. We numerically solve these equations simultaneously with a fixed poloidal current. As a result, we obtain a pulsar magnetosphere model without a current sheet, which is similar to the CKF solution near the neutron star and has a jet-like structure at a distance along the pole. In addition, we discuss physical properties of the model and find that the force-free condition breaks down in a vicinity of the light cylinder due to dissipation that is included implicitly in the numerical method., Comment: 15 pages, 10 figures, 1 table, change title, improve contents, accepted for publication in PASJ

In this letter we report on turbulent acceleration of the dissipation of magnetic field in the postshock re- gion of a Poynting flux-dominated flow, such as the Crab pulsar wind nebula. We have performed two- dimensional resistive relativistic magnetohydrodynamics simulations of subsonic turbulence driven by the Richtmyer-Meshkov instability at the shock fronts of the Poynting flux-dominated flows in pulsar winds. We find that turbulence stretches current sheets which substantially enhances the dissipation of magnetic field, and that most of the initial magnetic field energy is dissipated within a few eddy-turnover times. We also develop a simple analytical model for turbulent dissipation of magnetic field that agrees well with our simulations. The analytical model indicates that the dissipation rate does not depend on resistivity even in the small resistivity limit. Our findings can possibly alleviate the {\sigma}-problem in the Crab pulsar wind nebulae., Comment: 6 pages, 11 figures, accepted to ApJ

In this paper, we develop a stable and fast numerical scheme for relativistic dissipative hydrodynamics based on Israel-Stewart theory. Israel-Stewart theory is a stable and causal description of dissipation in relativistic hydrodynamics although it includes relaxation process with the timescale for collision of constituent particles, which introduces stiff equations and makes practical numerical calculation difficult. In our new scheme, we use Strang's splitting method, and use the piecewise exact solutions for solving the extremely short timescale problem. In addition, since we split the calculations into inviscid step and dissipative step, Riemann solver can be used for obtaining numerical flux for the inviscid step. The use of Riemann solver enables us to capture shocks very accurately. Simple numerical examples are shown. The present scheme can be applied to various high energy phenomena of astrophysics and nuclear physics., Comment: 23 pages, 12 figures, accepted in Journal of Computational Physics

We present a new numerical method of special relativistic resistive magnetohydrodynamics with scalar resistivity that can treat a range of phenomena, from nonrelativistic to relativistic (shock, contact discontinuity, and Alfv\'en wave). The present scheme calculates the numerical flux of fluid by using an approximate Riemann solver, and electromagnetic field by using the method of characteristics. Since this scheme uses appropriate characteristic velocities, it is capable of accurately solving problems that cannot be approximated as ideal magnetohydrodynamics and whose characteristic velocity is much lower than light velocity. The numerical results show that our scheme can solve the above problems as well as nearly ideal MHD problems. Our new scheme is particularly well suited to systems with initially weak magnetic field, and mixed phenomena of relativistic and non-relativistic velocity; for example, MRI in accretion disk, and super Alfv\'enic turbulence., Comment: 15 pages, 20 figures, accepted in ApJ

In this paper, we study the Cauchy problem of the linearized kinetic equations for the models of Marle and Anderson-Witting, and compare these dispersion relations with the 14-moment theory. First, we propose a modification of the Marle model to improve the resultant transport coefficients in accord with those obtained by the full Boltzmann equation. Using the modified Marle model and Anderson-Witting model, we calculate dispersion relations that are kinetically correct within the validity of the BGK approximation. The 14-moment theory that includes the time derivative of dissipation currents has causal structure, in contrast to the acausal first-order Chapman-Enskog approximation. However, the dispersion relation of the 14-moment theory does not accurately describe the result of the kinetic equation. Thus, our calculation indicates that keeping these second-order terms does not simply correspond to improving the physical description of the relativistic hydrodynamics., Comment: 20 pages, 22 figures, accepted for publication in Physica A

We study the linearized kinetic equation of relaxation model which was proposed by Bhatnagar, Gross and Krook (also called BGK model) and solve the dispersion relation. Using the solution of the dispersion relation, we analyze the relaxation of the macroscopic mode and kinetic mode. Since BGK model is not based on the expansion in the mean free path in contrast to the Chapman-Enskog expansion, the solution can describe accurate relaxation of initial disturbance with any wavelength. This non-relativistic analysis gives suggestions for our next work of relativistic analysis of relaxation., Comment: 18 pages, 14 figures, accepted for publication in Prog. Theor. Phys.