Search

ipie is a Python-based auxiliary-field quantum Monte Carlo (AFQMC) package that has undergone substantial improvements since its initial release [Malone et al., J. Chem. Theory Comput. 19(1), 109–121 (2023)]. This paper outlines the improved modularity and new capabilities implemented in ipie. We highlight the ease of incorporating different trial and walker types and the seamless integration of ipie with external libraries. We enable distributed Hamiltonian simulations of large systems that otherwise would not fit on a single central processing unit node or graphics processing unit (GPU) card. This development enabled us to compute the interaction energy of a benzene dimer with 84 electrons and 1512 orbitals with multi-GPUs. Using CUDA and cupy for NVIDIA GPUs, ipie supports GPU-accelerated multi-slater determinant trial wavefunctions [Huang et al. arXiv:2406.08314 (2024)] to enable efficient and highly accurate simulations of large-scale systems. This allows for near-exact ground state energies of multi-reference clusters, [Cu2O2]2+ and [Fe2S2(SCH3)4]2−. We also describe implementations of free projection AFQMC, finite temperature AFQMC, AFQMC for electron–phonon systems, and automatic differentiation in AFQMC for calculating physical properties. These advancements position ipie as a leading platform for AFQMC research in quantum chemistry, facilitating more complex and ambitious computational method development and their applications. [ABSTRACT FROM AUTHOR]

Automatic differentiation (AD) emerged as an integral part of machine learning, accelerating model development by enabling gradient-based optimization without explicit analytical derivatives. Recently, the benefits of AD and computing arbitrary-order derivatives with respect to any variable were also recognized in the field of quantum chemistry. In this work, we present dxtb—an open-source, fully differentiable framework for semiempirical extended tight-binding (xTB) methods. Developed entirely in Python and leveraging PyTorch for array operations, dxtb facilitates extensibility and rapid prototyping while maintaining computational efficiency. Through comprehensive code vectorization and optimization, we essentially reach the speed of compiled xTB programs for high-throughput calculations of small molecules. The excellent performance also scales to large systems, and batch operability yields additional benefits for execution on parallel hardware. In particular, energy evaluations are on par with existing programs, whereas the speed of automatically differentiated nuclear derivatives is only 2 to 5 times slower compared to their analytical counterparts. We showcase the utility of AD in dxtb by calculating various molecular and spectroscopic properties, highlighting its capacity to enhance and simplify such evaluations. Furthermore, the framework streamlines optimization tasks and offers seamless integration of semiempirical quantum chemistry in machine learning, paving the way for physics-inspired end-to-end differentiable models. Ultimately, dxtb aims to further advance the capabilities of semiempirical methods, providing an extensible foundation for future developments and hybrid machine learning applications. The framework is accessible at https://github.com/grimme-lab/dxtb. [ABSTRACT FROM AUTHOR]

In this work, we introduce a differentiable implementation of the local natural orbital coupled cluster (LNO-CC) method within the automatic differentiation framework of the PySCFAD package. The implementation is comprehensively tuned for enhanced performance, which enables the calculation of first-order static response properties on medium-sized molecular systems using coupled cluster theory with single, double, and perturbative triple excitations [CCSD(T)]. We evaluate the accuracy of our method by benchmarking it against the canonical CCSD(T) reference for nuclear gradients, dipole moments, and geometry optimizations. In addition, we demonstrate the possibility of property calculations for chemically interesting systems through the computation of bond orders and Mössbauer spectroscopy parameters for a [NiFe]-hydrogenase active site model, along with the simulation of infrared spectra via ab initio LNO-CC molecular dynamics for a protonated water hexamer. [ABSTRACT FROM AUTHOR]

We present a method for calculating first-order response properties in phaseless auxiliary field quantum Monte Carlo by applying automatic differentiation (AD). Biases and statistical efficiency of the resulting estimators are discussed. Our approach demonstrates that AD enables the calculation of reduced density matrices with the same computational cost scaling per sample as energy calculations, accompanied by a cost prefactor of less than four in our numerical calculations. We investigate the role of self-consistency and trial orbital choice in property calculations. We find that orbitals obtained using density functional theory perform well for the dipole moments of selected molecules compared to those optimized self-consistently. [ABSTRACT FROM AUTHOR]

Machine-learning potentials provide computationally efficient and accurate approximations of the Born–Oppenheimer potential energy surface. This potential determines many materials properties and simulation techniques usually require its gradients, in particular forces and stress for molecular dynamics, and heat flux for thermal transport properties. Recently developed potentials feature high body order and can include equivariant semi-local interactions through message-passing mechanisms. Due to their complex functional forms, they rely on automatic differentiation (AD), overcoming the need for manual implementations or finite-difference schemes to evaluate gradients. This study discusses how to use AD to efficiently obtain forces, stress, and heat flux for such potentials, and provides a model-independent implementation. The method is tested on the Lennard-Jones potential, and then applied to predict cohesive properties and thermal conductivity of tin selenide using an equivariant message-passing neural network potential. [ABSTRACT FROM AUTHOR]

AbstractWe present a comprehensive methodology for calculating angular kinematic quantities up to the fourth order using automatic differentiation with dual numbers. Two approaches are employed: the vector method, which uses position vectors and derivatives of three non-collinear points, and the body-fixed frame (BFF) method, which differentiates an attached basis to the rigid body. The BFF method avoids indeterminacy and divergence issues. Numerical examples, drawn from robotics and classical mechanics, validate the methodology and demonstrate its robustness. Our approach effectively addresses challenging inverse problems of acceleration, jerk, and jounce/snap. [ABSTRACT FROM AUTHOR]

As an anode material for lithium-ion batteries, amorphous silicon offers a significantly higher energy density than the graphite anodes currently used. Alloying reactions of lithium and silicon, however, induce large deformation and lead to volume changes up to 300%. We formulate a thermodynamically consistent continuum model for the chemo-elasto-plastic diffusion-deformation behavior of amorphous silicon and it's alloy with lithium based on finite deformations. In this paper, two plasticity theories, i.e. a rate-independent theory with linear isotropic hardening and a rate-dependent one, are formulated to allow the evolution of plastic deformations and reduce occurring stresses. Using modern numerical techniques, such as higher order finite element methods as well as efficient space and time adaptive solution algorithms, the diffusion-deformation behavior resulting from both theories is compared. In order to further increase the computational efficiency, an automatic differentiation scheme is used, allowing for a significant speed up in assembling time as compared to an algorithmic linearization for the global finite element Newton scheme. Both plastic approaches lead to a more heterogeneous concentration distribution and to a change to tensile tangential Cauchy stresses at the particle surface at the end of one charging cycle. Different parameter studies show how an amplification of the plastic deformation is affected. Interestingly, an elliptical particle shows only plastic deformation at the smaller half axis. With the demonstrated efficiency of the applied methods, results after five charging cycles are also discussed and can provide indications for the performance of lithium-ion batteries in long term use. [ABSTRACT FROM AUTHOR]

The challenging limitations of prohibitive computation and Frequency Principle remain significantly difficult for deep learning methods to effectively resolve multi-scale problems. In this work, a higher-order multi-scale deep Ritz method (HOMS-DRM) is developed to address this issue and effectively compute thermal transfer equation of composite materials with highly oscillatory, discontinuous and high-contrast coefficients. In the computational framework of HOMS-DRM, higher-order multi-scale modeling is first employed to overcome limitations of prohibitive computation and Frequency Principle when direct deep learning simulation. Then, improved deep Ritz method is designed to high-accuracy and mesh-free simulation for lower-order and higher-order microscopic unit cell functions, and macroscopic homogenized equations of multi-scale composites, which are then assembled into higher-order multi-scale solutions for multi-scale thermal transfer problems by using automatic differentiation technology. Besides, corresponding numerical algorithm of HOMS-DRM is developed for implementing high-accuracy multi-scale simulation in periodic composite medium. Moreover, the theoretical convergence of the proposed HOMS-DRM is rigorously demonstrated under appropriate assumptions. Finally, 2D and 3D numerical experiments including high-contrast composite materials are presented to validate the computational performance of HOMS-DRM. [ABSTRACT FROM AUTHOR]

In this article, we combine a Fast Fourier Transform based computational approach and a supervised machine learning strategy to discover models for the anisotropic effective viscosity of shear-thinning fiber suspensions. Using the Fast Fourier Transform based computational approach, we study the effects of the fiber orientation state and the imposed macroscopic shear rate tensor on the effective viscosity for a broad range of shear rates of engineering process interest. We visualize the effective viscosity in three dimensions and find that the anisotropy of the effective viscosity and its shear rate dependence vary strongly with the fiber orientation state. Combining the results of this work with insights from literature, we formulate four requirements a model of the effective viscosity should satisfy for shear-thinning fiber suspensions with a Cross-type matrix fluid. Furthermore, we introduce four model candidates with differing numbers of parameters and different theoretical motivations, and use supervised machine learning techniques for non-convex optimization to identify parameter sets for the model candidates. By doing so, we leverage the flexibility of automatic differentiation and the robustness of gradient based, supervised machine learning. Finally, we identify the most suitable model by comparing the prediction accuracy of the model candidates on the fiber orientation triangle, and find that multiple models predict the anisotropic shear-thinning behavior to engineering accuracy over a broad range of shear rates. [ABSTRACT FROM AUTHOR]

The inherent feature of Bayesian empirical analysis is the dependence of posterior inference on prior parameters, which researchers typically specify. However, quantifying the magnitude of this dependence remains difficult. This article extends Infinitesimal Perturbation Analysis, widely used in classical simulation, to compute asymptotically unbiased and consistent sensitivities of posterior statistics with respect to prior parameters from Markov chain Monte Carlo inference via Gibbs sampling. The method demonstrates the possibility of efficiently computing the complete set of prior sensitivities for a wide range of posterior statistics, alongside the estimation algorithm using Automatic Differentiation. The method's application is exemplified in Bayesian Vector Autoregression analysis of fiscal policy in U.S. macroeconomic time series data. The analysis assesses the sensitivities of posterior estimates, including the Impulse response functions and Forecast error variance decompositions, to prior parameters under common Minnesota shrinkage priors. The findings illuminate the significant and intricate influence of prior specification on the posterior distribution. This effect is particularly notable in crucial posterior statistics, such as the substantial absolute eigenvalue of the companion matrix, ultimately shaping the structural analysis. [ABSTRACT FROM AUTHOR]

Turbulence significantly influences fluid flow topology optimization, and this has already been verified under the incompressible flow regime. However, the same cannot be said about the compressible flow regime, in which the density field now affects and couples all of the fluid flow and turbulence equations and makes obtaining the adjoint model, which is necessary for topology optimization, extremely difficult. Up to now, the turbulence phenomenon has still not been considered in compressible flow topology optimization, which is what is being proposed and analyzed here. Rather than being based in the Reynolds‐Averaged Navier–Stokes (RANS) equations which are defined only for incompressible flow, the equations are now based on the Favre‐Averaged Navier–Stokes (FANS) equations, which are the counterpart of the RANS equations for compressible flow and feature different dependencies and terms. The compressible turbulence model being considered is the compressible version of the Spalart–Allmaras model, which differs from the usual Spalart–Allmaras model, since now there are some new spatially varying density and specific heat terms that depend on the primal variables and that act over some of the turbulence terms of the overall model. The adjoint equations are obtained by using an automatic differentiation scheme through a coupled software platform. The optimization algorithm is IPOPT, and some examples are presented to show the effect of turbulence in the compressible flow topology optimization. [ABSTRACT FROM AUTHOR]

Schizophrenia is a serious mental disorder with a complex neurobiological background and a well-defined psychopathological picture. Despite many efforts, a definitive disease biomarker has still not been identified. One of the promising candidates for a disease-related biomarker could involve retinal morphology , given that the retina is a part of the central nervous system that is known to be affected in schizophrenia and related to multiple illness features. In this study Optical Coherence Tomography (OCT) data is applied to assess the different layers of the retina. OCT data were applied in the process of automatic differentiation of schizophrenic patients from healthy controls. Numerical experiments involved applying several individual 1D Convolutional Neural Network-based models as well as further using the aggregation of classification results to improve the initial classification results. The main goal of the study was to check how methods based on the aggregation of classification results work in classifying neuroanatomical features of schizophrenia. Among over 300, 000 different variants of tested aggregation operators, a few versions provided satisfactory results. [ABSTRACT FROM AUTHOR]

Defining end-to-end (or online) training schemes for the calibration of neural sub-models in hybrid systems requires working with an optimization problem that involves the solver of the physical equations. Online learning methodologies thus require the numerical model to be differentiable, which is not the case for most modeling systems. To overcome this, we present an efficient and practical online learning approach for hybrid systems. The method, called EGA for Euler Gradient Approximation, assumes an additive neural correction to the physical model, and an explicit Euler approximation of the gradients. We demonstrate that the EGA converges to the exact gradients in the limit of infinitely small time steps. Numerical experiments show significant improvements over offline learning, highlighting the potential of end-to-end learning for hybrid modeling. End-to-end learning in hybrid numerical models involves solving an optimization problem that integrates the model's solver. In many fields, these solvers are written in low-abstraction programming languages that lack automatic differentiation. This work presents a practical approach to solving the optimization problem by efficiently approximating the gradient of the end-to-end objective function. [ABSTRACT FROM AUTHOR]

Driven by human ingenuity and creativity, the discovery of super-resolution techniques, which circumvent the classical diffraction limit of light, represent a leap in optical microscopy. However, the vast space encompassing all possible experimental configurations suggests that some powerful concepts and techniques might have not been discovered yet, and might never be with a human-driven direct design approach. Thus, AI-based exploration techniques could provide enormous benefit, by exploring this space in a fast, unbiased way. We introduce XLuminA, an open-source computational framework developed using JAX, a high-performance computing library in Python. XLuminA offers enhanced computational speed enabled by JAX's accelerated linear algebra compiler (XLA), just-in-time compilation, and its seamlessly integrated automatic vectorization, automatic differentiation capabilities and GPU compatibility. XLuminA demonstrates a speed-up of 4 orders of magnitude compared to well-established numerical optimization methods. We showcase XLuminA's potential by re-discovering three foundational experiments in advanced microscopy, and identifying an unseen experimental blueprint featuring sub-diffraction imaging capabilities. This work constitutes an important step in AI-driven scientific discovery of new concepts in optics and advanced microscopy. Researchers have developed XLuminA, an AI framework for the automated discovery of super-resolution microscopy techniques. With 10,000x faster optimization than traditional methods, it discovers unexplored designs breaking the diffraction limit. [ABSTRACT FROM AUTHOR]

Terrestrial biosphere models offer a comprehensive view of the global carbon cycle by integrating ecological processes across scales, yet they introduce significant uncertainties in climate and biogeochemical projections due to diverse process representations and parameter variations. For instance, different soil water limitation functions lead to wide productivity ranges across models. To address this, we propose the Differentiable Land Model (DifferLand), a novel hybrid machine learning approach replacing unknown water limitation functions in models with neural networks (NNs) to learn from data. Using automatic differentiation, we calibrated the embedded NN and the physical model parameters against daily observations of evapotranspiration, gross primary productivity, ecosystem respiration, and leaf area index across 16 FLUXNET sites. We evaluated six model configurations where NNs simulate increasingly complex soil water and photosynthesis interactions against test data sets to find the optimal structure‐performance tradeoff. Our findings show that a simple hybrid model with a univariate NN effectively captures site‐level water and carbon fluxes on a monthly timescale. Across a global aridity gradient, the magnitude of water stress limitation varies, but its functional form consistently converges to a piecewise linear relationship with saturation at high water levels. While models incorporating more interactions between soil water and meteorological drivers better fit observations at finer time scales, they risk overfitting and equifinality issues. Our study demonstrates that hybrid models have great potential in learning unknown parameterizations and testing ecological hypotheses. Nevertheless, careful structure‐performance tradeoffs are warranted in light of observational constraints to translate the retrieved relationships into robust process understanding. Plain Language Summary: Terrestrial carbon cycles simulations commonly focus on either describing the ecological processes with physical yet empirical equations or capturing the statistical relationships between variables using data‐driven techniques. Both approaches have their advantages and disadvantages. Process‐based simulations are grounded in scientific principles but may be inaccurate due to imperfect knowledge of the equations. Machine‐learning techniques can potentially capture the complex relationships between environmental variables but can be hard to extrapolate. In this study, we combine the two approaches into a hybrid model by embedding a set of neural networks within a process‐based model. We tested the model at different locations to study whether it can learn how plants respond to water limitations. The results showed the hybrid modeling approach can successfully retrieve the functional relationships between ecological variables. In addition, the overall performance of the hybrid model improved compared to the baseline model due to increased structural flexibility. We envision such a hybrid approach to help in the presence of imperfect knowledge of the governing equations in terrestrial carbon simulations. Instead of prescribing uncertain governing equations for the unknown ecological relationships, we can let the hybrid model learn these functional relationships from data, while preserving the temporal consistency of the model. Key Points: An automatically differentiable hybrid model is developed to learn parameters and functional relationships in land carbon and water cyclesNeural network emulators simulate ecological dynamics well but risk equifinality with limited data due to increased degrees of freedomMonthly soil water impacts on GPP and ET are well‐captured by piecewise linear functions, but finer time effects may need more complexity [ABSTRACT FROM AUTHOR]

Full waveform inversion (FWI), a powerful geophysical technique for subsurface imaging through seismic velocity-model construction, relies on numerical optimization, thus requiring the computation of derivatives for an objective function. This paper proposes a discrete development for accurate computation of the gradient and Hessian-vector product, providing second-order optimization benefits like higher convergence rates and improved resolution. The approach is a promising alternative for computing the gradient and Hessian action in time-domain FWI, applicable to various geophysical problems. Computational costs and memory requirements are comparable to the Adjoint-State Method and more avorable than Automatic Differentiation. While efficient automatic differentiation algorithms have transformed gradient computation in applications like FWI, challenges may arise in 3D due to unforeseen memory allocations. Our approach addresses this by exploring the reverse mode differentiation algorithm, mapping temporary memory allocations and computational complexity. By means of introducing auxiliary fields all involved wavefield evolutions can be carried out with the very same evolution scheme, in this way simplifying the implementation and focusing the performance improvement effort in a single routine thus reducing the maintenance cost of these algorithms, especially when using GPU implementations. [ABSTRACT FROM AUTHOR]

The global minimum point of an optimization problem is of interest in engineering fields and it is difficult to solve, especially for a nonconvex large-scale optimization problem. In this article, we consider a new memetic algorithm for this problem. That is to say, we use the continuation Newton method with the deflation technique to find multiple stationary points of the objective function and use those found stationary points as the initial seeds of the evolutionary algorithm, other than the random initial seeds of the known evolutionary algorithms. Meanwhile, in order to retain the usability of the derivative-free method and the fast convergence of the gradient-based method, we use the automatic differentiation technique to compute the gradient and replace the Hessian matrix with its finite difference approximation. According to our numerical experiments, this new algorithm works well for unconstrained optimization problems and finds their global minima efficiently, in comparison to the other representative global optimization methods such as the multi-start methods (the built-in subroutine GlobalSearch.m of MATLAB R2021b, GLODS, and VRBBO), the branch-and-bound method (Couenne, a state-of-the-art open-source solver for mixed integer nonlinear programming problems), and the derivative-free algorithms (CMA-ES and MCS). [ABSTRACT FROM AUTHOR]

Numerical simulations of seismic wave propagation in heterogeneous 3-D media are central to investigating subsurface structures and understanding earthquake processes, yet are computationally expensive for large problems. This is particularly problematic for full-waveform inversion (FWI), which typically involves numerous runs of the forward process. In machine learning there has been considerable recent work in the area of operator learning, with a new class of models called neural operators allowing for data-driven solutions to partial differential equations. Recent work in seismology has shown that when neural operators are adequately trained, they can significantly shorten the compute time for wave propagation. However, the memory required for the 3-D time domain equations may be prohibitive. In this study, we show that these limitations can be overcome by solving the wave equations in the frequency domain, also known as the Helmholtz equations, since the solutions for a set of frequencies can be determined in parallel. The 3-D Helmholtz neural operator is 40 times more memory-efficient than an equivalent time-domain version. We use a Helmholtz neural operator for 2-D and 3-D elastic wave modelling, achieving two orders of magnitude acceleration compared to a baseline spectral element method. The neural operator accurately generalizes to variable velocity structures and can be evaluated on denser input meshes than used in the training simulations. We also show that when solving for wavefields strictly at the free surface, the accuracy can be significantly improved via a graph neural operator layer. In leveraging automatic differentiation, the proposed method can serve as an alternative to the adjoint-state approach for 3-D FWI, reducing the computation time by a factor of 350. [ABSTRACT FROM AUTHOR]

We present ReactionMechanismSimulator.jl (RMS), a modern differentiable software for the simulation and analysis of chemical kinetic mechanisms, including multiphase systems. RMS has already been applied to problems in combustion, pyrolysis, polymers, pharmaceuticals, catalysis, and electrocatalysis. RMS is written in Julia, making it easy to develop and allowing it to take advantage of Julia's extensive numerical computing ecosystem. In addition to its extensive library of optimized analytic Jacobians, RMS can generate and use Jacobians computed using automatic differentiation and symbolically generated analytic Jacobians. RMS is demonstrated to be faster than Cantera and Chemkin in several benchmarks. RMS also implements an extensive set of features for analyzing chemical mechanisms, including a library of easy‐to‐call plotting functions, molecular structure resolved flux diagram generation, crash analysis, traditional sensitivity analysis, transitory sensitivity analysis, and an automatic mechanism analysis toolkit. RMS implements efficient adjoint and parallel forward sensitivity analyses. We also demonstrate the ease of adding new features to RMS. [ABSTRACT FROM AUTHOR]

This paper introduces PATO - a producibility-aware topology optimization (TO) framework to help efficiently explore the design space of components fabricated using metal additive manufacturing(AM), while ensuring manufacturability. Specifically, parts fabricated through Laser Powder Bed Fusion (LPBF) are prone to defects such as warpage or cracking due to high residual stress values generated from the steep thermal gradients produced during the build process. PATO is based on the a priori discovery of crack-free designs, so that the optimized part can be built defect-free at the outset. To ensure that the design is crack free, producibility is explicitly encoded within the standard formulation of TO, using maximum shear strain index (MSSI) as a crack index. Simulating the build process, in order to estimate MSSI, is a coupled, multi-physics, time-complex computation and incorporating it in the TO loop can be computationally prohibitive. Current advances in deep convolutional neural networks (DCNN) are leveraged to develop a high-fidelity surrogate model based on an Attention-based U-Net architecture to predict the MSSI values as a spatially varying field over the part's domain. Further, automatic differentiation is employed to directly compute the gradient of maximum MSSI with respect to the input design variables and augment it with the performance-based sensitivity field to optimize the design while considering the trade-off between weight, manufacturability, and functionality. The effectiveness of the proposed method is demonstrated through benchmark studies in 3D and experimental validation. [ABSTRACT FROM AUTHOR]

Optimized parallel implementations on GPU or CPU have dramatically enhanced the fidelity, resolution and accuracy of physical simulations and mesh-based algorithms. However, attaining optimal performance requires expert knowledge and might demand complex code and memory layout optimizations. This adds to the fact that physical simulation algorithms require the implementation of derivatives, which can be a tedious and error-prone process. In recent years, researchers and practitioners have investigated the concept of designing systems that allow for a more expressive definition of mesh-based simulation code. These systems leverage domain-specific languages (DSL), automatic differentiation or symbolic computing to enhance readability of implementations without compromising performance. We follow this line of work and propose a symbolic code generation approach tailored to mesh-based computations on parallel devices. Our system extends related work by incorporating collision handling and a data access synchronization approach, enabling rapid sparse matrix assembly. [ABSTRACT FROM AUTHOR]

Procedural implicit surfaces are a popular representation for shape modeling. They provide a simple framework for complex geometric operations such as Booleans, blending and deformations. However, their editability remains a challenging task: as the definition of the shape is purely implicit, direct manipulation of the shape cannot be performed. Thus, parameters of the model are often exposed through abstract sliders, which have to be nontrivially created by the user and understood by others for each individual model to modify. Further, each of these sliders needs to be set one by one to achieve the desired appearance. To circumvent this laborious process while preserving editability, we propose to directly manipulate the implicit surface in the viewport. We let the user naturally interact with the output shape, leveraging points on a co-parameterization we design specifically for implicit surfaces, to guide the parameter updates and reach the desired appearance faster. We leverage our automatic differentiation of the procedural implicit surface to propagate interactions made by the user in the viewport to the shape parameters themselves. We further design a solver that uses such information to guide an intuitive and smooth user workflow. We demonstrate different editing processes across multiple implicit shapes and parameters that would be tedious by tuning sliders. [ABSTRACT FROM AUTHOR]

Quasi-Monte Carlo integration is at the core of rendering. This technique estimates the value of an integral by evaluating the integrand at well-chosen sample locations. These sample points are designed to cover the domain as uniformly as possible to achieve better convergence rates than purely random points. Deterministic low-discrepancy sequences have been shown to outperform many competitors by guaranteeing good uniformity as measured by the so-called discrepancy metric, and, indirectly, by an integer t value relating the number of points falling into each domain stratum with the stratum area (lower t is better). To achieve randomness, scrambling techniques produce multiple realizations preserving the t value, making the construction stochastic. Among them, Owen scrambling is a popular approach that recursively permutes intervals for each dimension. However, relying on permutation trees makes it incompatible with smooth optimization frameworks. We present a differentiable Owen scrambling that regularizes permutations. We show that it can effectively be used with automatic differentiation tools for optimizing low-discrepancy sequences to improve metrics such as optimal transport uniformity, integration error, designed power spectra or projective properties, while maintaining their initial t-value as guaranteed by Owen scrambling. In some rendering settings, we show that our optimized sequences improve the rendering error. [ABSTRACT FROM AUTHOR]

Recent machine learning (ML)‐based weather forecasting models have improved the accuracy and efficiency of forecasts while minimizing computational resources, yet still depend on traditional data assimilation (DA) systems to generate analysis fields. Four dimensional variational data assimilation (4DVar) enhances model states, relying on the prediction model to propagate observation to the initial field. Consequently, the initial fields from traditional DA are not optimal for ML‐based models, necessitating a customized DA system. This paper introduces an ensemble 4DVar system integrated with the FuXi model (FuXi‐En4DVar), which can independently generate accurate analysis fields. It utilizes automatic differentiation to compute gradients, and demonstrates the equivalence of these gradients with those derived from adjoint models. Experimental results indicate that this system preserves the physical balance of the analysis field and exhibits flow‐dependent characteristics. These features enhance the propagation and assimilation of observation into the initial analysis field, thereby improving the accuracy of the analysis fields. Plain Language Summary: Machine learning (ML)‐based weather forecasting models have made significant progress, offering fast and accurate weather predictions. However, a critical limitation of these models is their dependence on externally provided initial fields, which they are unable to generate independently. This study addresses this limitation by developing a data assimilation (DA) system with FuXi, a state‐of‐the‐art ML‐based weather forecasting model, enabling it to generate these initial fields. Experimental results confirm the rationality and effectiveness of this system. Key Points: The FuXi‐En4Dvar employ automatic differentiation to compute gradients eliminating the need for tangent linear models and adjoint modelsUsing the rapid ensemble generation capabilities of ML‐based weather forecasting model to construct the background error covariance matrixThe FuXi‐En4DVar demonstrates flow‐dependent characteristics, constraining analysis increments that adhere to physical balance relationships [ABSTRACT FROM AUTHOR]

A backward stable numerical calculation of a function with condition number κ$$ \kappa $$ will have a relative accuracy of κϵmachine$$ \kappa {\epsilon}_{\mathrm{machine}} $$. Standard formulations and software implementations of finite‐strain elastic materials models make use of the deformation gradient F=I+∂u/∂X$$ \boldsymbol{F}=I+\partial \boldsymbol{u}/\partial \boldsymbol{X} $$ and Cauchy‐Green tensors. These formulations are not numerically stable, leading to loss of several digits of accuracy when used in the small strain regime, and often precluding the use of single precision floating point arithmetic. We trace the source of this instability to specific points of numerical cancellation, interpretable as ill‐conditioned steps. We show how to compute various strain measures in a stable way and how to transform common constitutive models to their stable representations, formulated in either initial or current configuration. The stable formulations all provide accuracy of order ϵmachine$$ {\epsilon}_{\mathrm{machine}} $$. In many cases, the stable formulations have elegant representations in terms of appropriate strain measures and offer geometric intuition that is lacking in their standard representation. We show that algorithmic differentiation can stably compute stresses so long as the strain energy is expressed stably, and give principles for stable computation that can be applied to inelastic materials. [ABSTRACT FROM AUTHOR]

Sensitivity, which is useful for evaluating the contributions of input variations to output changes, favors designers exploiting the most sensitive design parameters and completing design optimization in aerospace. This study introduces the turbulence adjoint method due to its low computational cost in calculating sensitivities with high accuracy and then solves the adjoint partial differential equations with respect to both the Reynolds-averaged Navier–Stokes and turbulence model equations with the assistance of an automatic differentiation tool. By utilizing the turbulence adjoint method, the sensitivities of aerodynamic performance to the blade profile modifications of two-dimensional and three-dimensional compressor cascades are calculated, allowing for the investigations of the impact of geometric variations on the changes in the flow field. Ultimately, aerodynamic optimization of the compressor cascades is conducted. After optimization, the adiabatic efficiency of the two-dimensional cascade increases by 3.11%, and it increases by 1.15% for the three-dimensional cascade. The variations in flow fields of both the original and optimized cascades are illustrated to discover the origins of performance improvements. The changes in blade profile are almost consistent with those forecasted from sensitivity analysis, demonstrating the potential superiority of the adjoint method for aerodynamic design. [ABSTRACT FROM AUTHOR]

Automatic differentiation is a popular technique for computing derivatives of computer programs. While automatic differentiation has been successfully used in countless engineering, science, and machine learning applications, it can sometimes nevertheless produce surprising results. In this paper, we categorize problematic usages of automatic differentiation, and illustrate each category with examples such as chaos, time‐averages, discretizations, fixed‐point loops, lookup tables, linear solvers, and probabilistic programs, in the hope that readers may more easily avoid or detect such pitfalls. We also review debugging techniques and their effectiveness in these situations. This article is categorized under:Technologies > Machine Learning [ABSTRACT FROM AUTHOR]

The article offers a general account of the genesis of the absolute differential calculus (ADC), paying special attention to its links with the history of differential geometry. In relatively recent times, several historians have described the development of the ADC as a direct outgrowth either of the theory of algebraic and differential invariants or as a product of analytical investigations, thus minimizing the role of Riemann's geometry in the process leading to its discovery. Our principal aim consists in challenging this historiographical tenet and analyzing the intimate connection between the development of Riemannian geometry and the birth of tensor calculus. [ABSTRACT FROM AUTHOR]

Although database systems perform well in data access and manipulation, their relational model hinders data scientists from formulating machine learning algorithms in SQL. Nevertheless, we argue that modern database systems perform well for machine learning algorithms expressed in relational algebra. To overcome the barrier of the relational model, this paper shows how to transform data into a coordinate relational representation for training neural networks in SQL: We first describe building blocks for data transformation, model training and inference in SQL-92 and their counterparts using an extended array data type. Then, we compare the implementation for model training and inference using array data types to the one using a coordinate relational representation in SQL-92 only. The evaluation in terms of runtime and memory consumption proves the suitability of modern database systems for matrix algebra, although specialised array data types perform better than matrices in coordinate relational representation. [ABSTRACT FROM AUTHOR]

This paper is dedicated to the acoustic inversion of the vertical sound speed profiles (SSPs) in the underwater marine environment. The method of automatic differentiation is applied for the first time in this context. Representing the finite-difference Padé approximation of the propagation operator as a computational graph allows for the analytical computation of the gradient with respect to the SSP directly within the numerical scheme. The availability of the gradient, along with the high computational efficiency of the numerical method used, enables rapid inversion of the SSP based on acoustic measurements from a hydrophone array. It is demonstrated that local optimization methods can be effectively used for real-time sound speed inversion. Comparative analysis with existing methods shows the significant superiority of the proposed method in terms of computation speed. [ABSTRACT FROM AUTHOR]

Theano, TensorFlow, Keras, Torch, PyTorch, and other software frameworks have remarkably stimulated the popularity of deep learning (DL). Apart from all the good they achieve, the danger of such frameworks is that they unintentionally spur a black-box attitude. Some practitioners play around with building blocks offered by frameworks and rely on them, having a superficial understanding of the internal mechanics. This paper constitutes a concise tutorial that elucidates the flows of signals and gradients in deep neural networks, enabling readers to successfully implement a deep network from scratch. By "from scratch", we mean with access to a programming language and numerical libraries but without any components that hide DL computations underneath. To achieve this goal, the following five topics need to be well understood: (1) automatic differentiation, (2) the initialization of weights, (3) learning algorithms, (4) regularization, and (5) the organization of computations. We cover all of these topics in the paper. From a tutorial perspective, the key contributions include the following: (a) proposition of R and S operators for tensors—rashape and stack, respectively—that facilitate algebraic notation of computations involved in convolutional, pooling, and flattening layers; (b) a Python project named hmdl ("home-made deep learning"); and (c) consistent notation across all mathematical contexts involved. The hmdl project serves as a practical example of implementation and a reference. It was built using NumPy and Numba modules with JIT and CUDA amenities applied. In the experimental section, we compare hmdl implementation to Keras (backed with TensorFlow). Finally, we point out the consistency of the two in terms of convergence and accuracy, and we observe the superiority of the latter in terms of efficiency. [ABSTRACT FROM AUTHOR]

Over the past decade, the development of computer-aided detection tools for medical image analysis has seen significant advancements. However, tasks such as the automatic differentiation of tissues or regions in medical images remain challenging. Magnetic resonance imaging (MRI) has proven valuable for early diagnosis, particularly in conditions like prostate cancer, yet it often struggles to produce high-resolution images with clearly defined boundaries. In this article, we propose a novel segmentation approach based on minimum cross-entropy thresholding using the equilibrium optimizer (MCE-EO) to enhance the visual differentiation of tissues in prostate MRI scans. To validate our method, we conducted two experiments. The first evaluated the overall performance of MCE-EO using standard grayscale benchmark images, while the second focused on a set of transaxial-cut prostate MRI scans. MCE-EO's performance was compared against six stochastic optimization techniques. Statistical analysis of the results demonstrates that MCE-EO offers superior performance for prostate MRI segmentation, providing a more effective tool for distinguishing between various tissue types. [ABSTRACT FROM AUTHOR]

We develop hierarchical models and methods in a fully parametric approach to generalized linear mixed models for any patterned covariance matrix. The Laplace approximation is used to marginally estimate covariance parameters by integrating over all fixed and latent random effects. The Laplace approximation relies on Newton–Raphson updates, which also leads to predictions for the latent random effects. We develop methodology for complete marginal inference, from estimating covariance parameters and fixed effects to making predictions for unobserved data. The marginal likelihood is developed for six distributions that are often used for binary, count, and positive continuous data, and our framework is easily extended to other distributions. We compare our methods to fully Bayesian methods, automatic differentiation, and integrated nested Laplace approximations (INLA) for bias, mean‐squared (prediction) error, and interval coverage, and all methods yield very similar results. However, our methods are much faster than Bayesian methods, and more general than INLA. Examples with binary and proportional data, count data, and positive‐continuous data are used to illustrate all six distributions with a variety of patterned covariance structures that include spatial models (both geostatistical and areal models), time series models, and mixtures with typical random intercepts based on grouping. [ABSTRACT FROM AUTHOR]

For the numerical solution of problems arising in various fields of mathematics and mechanics, it is often necessary to determine the values of derivatives included in the model. Currently, numerical values of derivatives can be obtained using automatic differentiation libraries in many programming languages. This paper discusses the use of the Python programming language, which is widely used in the scientific community. It should be noted that the principles of automatic differentiation are not related to numerical or symbolic differentiation methods. The work consists of three parts. The introduction reviews the historical development of the general theory of complex numbers and the use of simple complex, double and dual numbers, which are a subset of the set of general complex numbers, in various fields of mathematics. The second part is devoted to the algebra of dual and hyperdual numbers and their properties. This section presents tables of the basis element of elementary functions with dual and hyperdual arguments, based on multiplication rules. Two important formulas for finding the numerical values of a complex function's first and second derivatives by expanding functions with dual and hyperdual arguments in the Taylor series are also obtained. A simple test function was used to verify the correctness of these formulas, the results of which were checked analytically as well as through implementation in a programming language. The third part of the paper focuses on practical applications and the implementation of these methods in Python. It includes detailed examples of case studies demonstrating the effectiveness of using hyperdual numbers in automatic differentiation. The results highlight the accuracy and computational efficiency of these methods, making them valuable tools for researchers and engineers. This comprehensive approach not only validates the theoretical aspects but also showcases the practical utility of dual and hyperdual numbers in solving complex mathematical and mechanical problems.

Abstract Over the past few decades, extensive research has been devoted to the design of artificial reverberation algorithms aimed at emulating the room acoustics of physical environments. Despite significant advancements, automatic parameter tuning of delay-network models remains an open challenge. We introduce a novel method for finding the parameters of a feedback delay network (FDN) such that its output renders target attributes of a measured room impulse response. The proposed approach involves the implementation of a differentiable FDN with trainable delay lines, which, for the first time, allows us to simultaneously learn each and every delay-network parameter via backpropagation. The iterative optimization process seeks to minimize a perceptually motivated time-domain loss function incorporating differentiable terms accounting for energy decay and echo density. Through experimental validation, we show that the proposed method yields time-invariant frequency-independent FDNs capable of closely matching the desired acoustical characteristics and outperforms existing methods based on genetic algorithms and analytical FDN design.

The Griewank test function for global unconstrained optimization has multiple local minima clustered around the global minimum at the origin. A new version of this test function is proposed that has a similar structure, but whose behavior at the local minima and maxima is non-smooth. This piecewise smooth version of the Griewank function represents an abs-factorable test case of objective functions for global non-smooth optimization as, for example, observed in the training of neural networks. [ABSTRACT FROM AUTHOR]

Four-dimensional (4D) constellations with up to 131 072 points (17 bit/4D-sym) are designed for the first time using geometric shaping. The constellations are optimized in terms of mutual information (MI) and generalized MI (GMI) for the additive white Gaussian noise (AWGN) channel, targeting a forward error correction (FEC) rate of 0.8 at finite signal-to-noise ratios. The presented 15–17 bit constellations are currently the highest-performing constellations in the literature, having a gap to the AWGN capacity as low as 0.17 dB (MI) and 0.45 dB (GMI) at 17 bit/4D-sym. For lower cardinalities, our constellations match or closely approach the performance of previously published optimized constellations. We also show that (GMI-)optimized constellations with a symmetry constraint, optimized for a FEC rate of 0.8, perform nearly identical to their unconstrained counterparts for cardinalities above 8 bit/4D-sym. A symmetry constraint for MI-optimized constellations is shown to have a negative impact in general. The proposed procedure relies on a Monte-Carlo-based approach for evaluating performance and is extendable to other (nonlinear) channels. Stochastic gradient descent is used for the optimization algorithm for which the gradients are computed using automatic differentiation. This article is part of the theme issue 'Celebrating the 15th anniversary of the Royal Society Newton International Fellowship'. [ABSTRACT FROM AUTHOR]

Determining Jacobian sparsity structure is an important step in the efficient computation of sparse Jacobians. We introduce a new method for determining Jacobian sparsity patterns by combining bit vector probing with Bloom filters. We further refine Bloom filter probing by combining it with hierarchical probing to yield a highly effective strategy for Jacobian sparsity pattern determination. [ABSTRACT FROM AUTHOR]

Over the past few decades, extensive research has been devoted to the design of artificial reverberation algorithms aimed at emulating the room acoustics of physical environments. Despite significant advancements, automatic parameter tuning of delay-network models remains an open challenge. We introduce a novel method for finding the parameters of a feedback delay network (FDN) such that its output renders target attributes of a measured room impulse response. The proposed approach involves the implementation of a differentiable FDN with trainable delay lines, which, for the first time, allows us to simultaneously learn each and every delay-network parameter via backpropagation. The iterative optimization process seeks to minimize a perceptually motivated time-domain loss function incorporating differentiable terms accounting for energy decay and echo density. Through experimental validation, we show that the proposed method yields time-invariant frequency-independent FDNs capable of closely matching the desired acoustical characteristics and outperforms existing methods based on genetic algorithms and analytical FDN design. [ABSTRACT FROM AUTHOR]

For the numerical solution of problems arising in various fields of mathematics and mechanics, it is often necessary to determine the values of derivatives included in the model. Currently, numerical values of derivatives can be obtained using automatic differentiation Libraries in many programming Languages. This paper discusses the use of the Python programming language, which is widely used in the scientific community. It should be noted that the principles of automatic differentiation are not related to numerical or symbolic differentiation methods. The work consists of three parts. The introduction reviews the historical development of the general theory of complex numbers and the use of simple complex, double and dual numbers, which are a subset of the set of general complex numbers, in various fields of mathematics. The second part is devoted to the algebra of dual and hyperdual numbers and their properties. This section presents tables of the basis element of elementary functions with dual and hyperdual arguments, based on multiplication rules. Two important formulas for finding the numerical values of a complex function's first and second derivatives by expanding functions with dual and hyperdual arguments in the Taylor series are also obtained. A simple test function was used to verify the correctness of these formulas, the results of which were checked analytically as well as through implementation in a programming language. The third part of the paper focuses on practical applications and the implementation of these methods in Python. It includes detailed examples of case studies demonstrating the effectiveness of using hyperdual numbers in automatic differentiation. The results highlight the accuracy and computational efficiency of these methods, making them valuable tools for researchers and engineers. This comprehensive approach not only validates the theoretical aspects but also showcases the practical utility of dual and hyperdual numbers in solving complex mathematical and mechanical problems. [ABSTRACT FROM AUTHOR]

Physics‐informed neural networks (PINNs) are gaining attention as an alternative approach to solve scientific problems governed by differential equations. This work aims at assessing the effectiveness of PINNs to solve a set of partial differential equations for which this method has never been considered, namely the augmented shallow water equations (SWEs) with topography. Differently from traditional SWEs, the bed elevation is considered as an additional conserved variable, and therefore one more equation expressing the fixed‐bed condition is included in the system. This approach allows the PINN model to leverage automatic differentiation to compute the bed slopes by learning the topographical information during training. PINNs are here tested for different one‐dimensional cases with non‐flat topography, and results are compared with analytical solutions. Though some limitations can be highlighted, PINNs show a good accuracy for the depth and velocity predictions even in the presence of non‐horizontal bottom. The solution of the augmented system of SWEs can therefore be regarded as a suitable alternative strategy to deal with flows over complex topography using PINNs, also in view of future extensions to realistic problems. Key Points: Physics‐informed neural networks (PINNs) are applied to solve the augmented shallow water equations with topographyApplications to one‐dimensional cases of free‐surface flows over non‐flat bottom show a good solution accuracySolving the augmented system is an alternative way to deal with non‐flat topography [ABSTRACT FROM AUTHOR]

Mesh processing algorithms are often communicated via concise mathematical notation (e.g., summation over mesh neighborhoods). However, conversion of notation into working code remains a time-consuming and error-prone process, which requires arcane knowledge of low-level data structures and libraries—impeding rapid exploration of high-level algorithms. We address this problem by introducing a domain-specific language (DSL) for mesh processing called I MESH, which resembles notation commonly used in visual and geometric computing and automates the process of converting notation into code. The centerpiece of our language is a flexible notation for specifying and manipulating neighborhoods of a cell complex, internally represented via standard operations on sparse boundary matrices. This layered design enables natural expression of algorithms while minimizing demands on a code generation backend. In particular, by integrating I MESH with the linear algebra features of the I LA DSL and adding support for automatic differentiation, we can rapidly implement a rich variety of algorithms on point clouds, surface meshes, and volume meshes. [ABSTRACT FROM AUTHOR]

In this paper, we introduced the gradient-enhanced fractional physics-informed neural networks (gfPINNs) for solving the forward and inverse problems of the multiterm time-fractional Burger-type equation. The gfPINNs leverage gradient information derived from the residual of the fractional partial differential equation and embed the gradient into the loss function. Since the standard chain rule in integer calculus is invalid in fractional calculus, the automatic differentiation of neural networks does not apply to fractional operators. The automatic differentiation for the integer order operators and the finite difference discretization for the fractional operators were used to construct the residual in the loss function. The numerical results demonstrate the effectiveness of gfPINNs in solving the multiterm time-fractional Burger-type equation. By comparing the experimental results of fractional physics-informed neural networks (fPINNs) and gfPINNs, it can be seen that the training performance of gfPINNs is better than fPINNs. [ABSTRACT FROM AUTHOR]

Shape optimization approaches to inverse design offer low‐dimensional, physically‐guided parameterizations of structures by representing them as combinations of primitives. However, on fixed grids, computing the gradient of a user objective via the adjoint variables method requires a product of forward/adjoint field solutions and the Jacobian of the simulation material distribution with respect to the structural shape parameters. Shape parameters often perturb global parts of the simulation grid resulting in many non‐zero Jacobian entries. These are often computed by finite‐difference (FD) in practice, and hence can be non‐trivial. In this work, the gradient calculation is accelerated by invoking automatic differentiation (AD) in instantiations of structural material distributions, enabled by the development of extensible differentiable feature‐mappings from parameters to primitives and differentiable effective logic operations (denoted AutoDiffGeo or ADG). ADG can also be used to accelerate FD‐based shape optimization by efficient boundary selection. AD‐enhanced shape optimization is demonstrated using three integrated photonic examples: a blazed grating coupler, a waveguide transition taper, and a polarization‐splitting grating coupler. The accelerations of the gradient calculation by AD relative to FD with boundary selection exceed 10×$\times$, resulting in total optimization wall time accelerations of 1.4×$1.4\times$–3.8×$3.8\times$ on the same hardware with no compromise to device figure‐of‐merit. [ABSTRACT FROM AUTHOR]

As a widely used equation in electrostatics, the Poisson equation has significant research value in numerical solution. The basic principle of existing methods is to divide the solution domain into various grids and solve the numerical solutions at each grid node. Therefore, the accuracy of the solution is strongly correlated with the grid density divided. Based on this, this paper proposes a grid-free numerical calculation method that requires far fewer model parameters than traditional methods, and can ignore the order of the equation to solve high-dimensional Poisson equations. Given a Poisson equation, which has a certain type of boundary condition. A certain number of coordinate points are selected on the solution space and its boundary to construct a dataset. Using automatic differentiation technique to fit the differential operator in the equation, a loss function is constructed by incorporating the given boundary conditions or initial conditions, and the final numerical solution is obtained through iterative optimization algorithms. In the numerical experiment section, the algorithm proposed in this paper was used to solve the two-dimensional and three-dimensional Poisson equations with given exact solutions. The relative errors between the numerical solution and the true solution were 8. 5 3 e − 4 and 6. 4 0 e − 2 , which are within the acceptable range. This proves that the proposed algorithm is feasible for solving the two-dimensional and three-dimensional Poisson equations with precise solutions. Secondly, the proposed algorithm is used to solve the four-dimensional Poisson equation with first-type boundary conditions, and the relative error range of the solution was within [0,0.56], which successfully extends the algorithm to solve high-dimensional Poisson equations and verifies its feasibility and efficiency in solving high-dimensional Poisson equations regardless of the dimension restriction. [ABSTRACT FROM AUTHOR]