Search

Unsteady Aerodynamic Shape Optimization presents new challenges in terms of sensitivity analysis of time-dependent objective functions. In this work, we consider periodic unsteady flows governed by the URANS equations. Hence, the resulting output functions acting as objective or constraint functions of the optimization are themselves periodic with unknown period length, that may depend on the design parameter of said optimization. Sensitivity Analysis on the time-average of a function with these properties turns out to be difficult. Therefore, we explore methods to regularize the time average of such a function with the so called windowing-approach. Furthermore, we embed these regularizers into the discrete adjoint solver for the URANS equations of the multi-physics and optimization software SU2. Finally, we exhibit a comparison study between the classical non regularized optimization procedure and the ones enhanced with regularizers of different smoothness and show that the latter result in a more robust optimization.

Low-Rank Adaptation (LoRA) has become a widely used method for parameter-efficient fine-tuning of large-scale, pre-trained neural networks. However, LoRA and its extensions face several challenges, including the need for rank adaptivity, robustness, and computational efficiency during the fine-tuning process. We introduce GeoLoRA, a novel approach that addresses these limitations by leveraging dynamical low-rank approximation theory. GeoLoRA requires only a single backpropagation pass over the small-rank adapters, significantly reducing computational cost as compared to similar dynamical low-rank training methods and making it faster than popular baselines such as AdaLoRA. This allows GeoLoRA to efficiently adapt the allocated parameter budget across the model, achieving smaller low-rank adapters compared to heuristic methods like AdaLoRA and LoRA, while maintaining critical convergence, descent, and error-bound theoretical guarantees. The resulting method is not only more efficient but also more robust to varying hyperparameter settings. We demonstrate the effectiveness of GeoLoRA on several state-of-the-art benchmarks, showing that it outperforms existing methods in both accuracy and computational efficiency.

In this work, we propose a federated dynamical low-rank training (FeDLRT) scheme to reduce client compute and communication costs - two significant performance bottlenecks in horizontal federated learning. Our method builds upon dynamical low-rank splitting schemes for manifold-constrained optimization to create a global low-rank basis of network weights, which enables client training on a small coefficient matrix. A consistent global low-rank basis allows us to incorporate a variance correction scheme and prove global loss descent and convergence to a stationary point. Dynamic augmentation and truncation of the low-rank bases automatically optimizes computing and communication resource utilization. We demonstrate the efficiency of FeDLRT in an array of computer vision benchmarks and show a reduction of client compute and communication costs by up to an order of magnitude with minimal impacts on global accuracy.

The main challenge of large-scale numerical simulation of radiation transport is the high memory and computation time requirements of discretization methods for kinetic equations. In this work, we derive and investigate a neural network-based approximation to the entropy closure method to accurately compute the solution of the multi-dimensional moment system with a low memory footprint and competitive computational time. We extend methods developed for the standard entropy-based closure to the context of regularized entropy-based closures. The main idea is to interpret structure-preserving neural network approximations of the regularized entropy closure as a two-stage approximation to the original entropy closure. We conduct a numerical analysis of this approximation and investigate optimal parameter choices. Our numerical experiments demonstrate that the method has a much lower memory footprint than traditional methods with competitive computation times and simulation accuracy.

This work explores the application of deep operator learning principles to a problem in statistical physics. Specifically, we consider the linear kinetic equation, consisting of a differential advection operator and an integral collision operator, which is a powerful yet expensive mathematical model for interacting particle systems with ample applications, e.g., in radiation transport. We investigate the capabilities of the Deep Operator network (DeepONet) approach to modelling the high dimensional collision operator of the linear kinetic equation. This integral operator has crucial analytical structures that a surrogate model, e.g., a DeepONet, needs to preserve to enable meaningful physical simulation. We propose several DeepONet modifications to encapsulate essential structural properties of this integral operator in a DeepONet model. To be precise, we adapt the architecture of the trunk-net so the DeepONet has the same collision invariants as the theoretical kinetic collision operator, thus preserving conserved quantities, e.g., mass, of the modeled many-particle system. Further, we propose an entropy-inspired data-sampling method tailored to train the modified DeepONet surrogates without requiring an excessive expensive simulation-based data generation., Comment: 12 pages, 8 figures

Numerical simulations of kinetic problems can become prohibitively expensive due to their large memory footprint and computational costs. A method that has proven to successfully reduce these costs is the dynamical low-rank approximation (DLRA). One key question when using DLRA methods is the construction of robust time integrators that preserve the invariances and associated conservation laws of the original problem. In this work, we demonstrate that the augmented basis update & Galerkin integrator (BUG) preserves solution invariances and the associated conservation laws when using a conservative truncation step and an appropriate time and space discretization. We present numerical comparisons to existing conservative integrators and discuss advantages and disadvantages

Reducing parameter redundancies in neural network architectures is crucial for achieving feasible computational and memory requirements during training and inference phases. Given its easy implementation and flexibility, one promising approach is layer factorization, which reshapes weight tensors into a matrix format and parameterizes them as the product of two small rank matrices. However, this approach typically requires an initial full-model warm-up phase, prior knowledge of a feasible rank, and it is sensitive to parameter initialization. In this work, we introduce a novel approach to train the factors of a Tucker decomposition of the weight tensors. Our training proposal proves to be optimal in locally approximating the original unfactorized dynamics independently of the initialization. Furthermore, the rank of each mode is dynamically updated during training. We provide a theoretical analysis of the algorithm, showing convergence, approximation and local descent guarantees. The method's performance is further illustrated through a variety of experiments, showing remarkable training compression rates and comparable or even better performance than the full baseline and alternative layer factorization strategies.

Neural networks have achieved tremendous success in a large variety of applications. However, their memory footprint and computational demand can render them impractical in application settings with limited hardware or energy resources. In this work, we propose a novel algorithm to find efficient low-rank subnetworks. Remarkably, these subnetworks are determined and adapted already during the training phase and the overall time and memory resources required by both training and evaluating them are significantly reduced. The main idea is to restrict the weight matrices to a low-rank manifold and to update the low-rank factors rather than the full matrix during training. To derive training updates that are restricted to the prescribed manifold, we employ techniques from dynamic model order reduction for matrix differential equations. This allows us to provide approximation, stability, and descent guarantees. Moreover, our method automatically and dynamically adapts the ranks during training to achieve the desired approximation accuracy. The efficiency of the proposed method is demonstrated through a variety of numerical experiments on fully-connected and convolutional networks.

In this paper we present KiT-RT (Kinetic Transport Solver for Radiation Therapy), an open-source C++ based framework for solving kinetic equations in radiation therapy applications. The aim of this code framework is to provide a collection of classical deterministic solvers for unstructured meshes that allow for easy extendability. Therefore, KiT-RT is a convenient base to test new numerical methods in various applications and compare them against conventional solvers. The implementation includes spherical-harmonics, minimal entropy, neural minimal entropy and discrete ordinates methods. Solution characteristics and efficiency are presented through several test cases ranging from radiation transport to electron radiation therapy. Due to the variety of included numerical methods and easy extendability, the presented open source code is attractive for both developers, who want a basis to build their own numerical solvers and users or application engineers, who want to gain experimental insights without directly interfering with the codebase., Comment: 28 pages, 15 figures, journal submission

The multi-scale nature of gaseous flows poses tremendous difficulties for theoretical and numerical analysis. The Boltzmann equation, while possessing a wider applicability than hydrodynamic equations, requires significantly more computational resources due to the increased degrees of freedom in the model. The success of a hybrid fluid-kinetic flow solver for the study of multi-scale flows relies on accurate prediction of flow regimes. In this paper, we draw on binary classification in machine learning and propose the first neural network classifier to detect near-equilibrium and non-equilibrium flow regimes based on local flow conditions. Compared with classical semi-empirical criteria of continuum breakdown, the current method provides a data-driven alternative where the parameterized implicit function is trained by solutions of the Boltzmann equation. The ground-truth labels are derived rigorously from the deviation of particle distribution functions and the approximations based on the Chapman-Enskog ansatz. Therefore, no tunable parameter is needed in the criterion. Following the entropy closure of the Boltzmann moment system, a data generation strategy is developed to produce training and test sets. Numerical analysis shows its superiority over simulation-based samplings. A hybrid Boltzmann-Navier-Stokes flow solver is built correspondingly with adaptive partition of local flow regimes. Numerical experiments including one-dimensional Riemann problem, shear flow layer and hypersonic flow around circular cylinder are presented to validate the current scheme for simulating cross-scale and non-equilibrium flow physics. The quantitative comparison with a semi-empirical criterion and benchmark results demonstrates the capability of the current neural classifier to accurately predict continuum breakdown., Comment: 35 pages, 21figures

This work presents neural network based minimal entropy closures for the moment system of the Boltzmann equation, that preserve the inherent structure of the system of partial differential equations, such as entropy dissipation and hyperbolicity. The described method embeds convexity of the moment to entropy map in the neural network approximation to preserve the structure of the minimal entropy closure. Two techniques are used to implement the methods. The first approach approximates the map between moments and the minimal entropy of the moment system and is convex by design. The second approach approximates the map between moments and Lagrange multipliers of the dual of the minimal entropy optimization problem, which present the gradients of the entropy with respect to the moments, and is enforced to be monotonic by introduction of a penalty function. We derive an error bound for the generalization gap of convex neural networks which are trained in Sobolev norm and use the results to construct data sampling methods for neural network training. Numerical experiments are conducted, which show that neural network-based entropy closures provide a significant speedup for kinetic solvers while maintaining a sufficient level of accuracy. The code for the described implementations can be found in the Github repositories., Comment: 28 pages, 15 figures. arXiv admin note: text overlap with arXiv:2106.09445

Direct simulation of physical processes on a kinetic level is prohibitively expensive in aerospace applications due to the extremely high dimension of the solution spaces. In this paper, we consider the moment system of the Boltzmann equation, which projects the kinetic physics onto the hydrodynamic scale. The unclosed moment system can be solved in conjunction with the entropy closure strategy. Using an entropy closure provides structural benefits to the physical system of partial differential equations. Usually computing such closure of the system spends the majority of the total computational cost, since one needs to solve an ill-conditioned constrained optimization problem. Therefore, we build a neural network surrogate model to close the moment system, which preserves the structural properties of the system by design, but reduces the computational cost significantly. Numerical experiments are conducted to illustrate the performance of the current method in comparison to the traditional closure., Comment: 17 pages, 6 figures