Search

Computing the numerical solution to high-dimensional tensor differential equations can lead to prohibitive computational costs and memory requirements. To reduce the memory and computational footprint, dynamical low-rank approximation (DLRA) has proven to be a promising approach. DLRA represents the solution as a low-rank tensor factorization and evolves the resulting low-rank factors in time. A central challenge in DLRA is to find time integration schemes that are robust to the arising small singular values. A robust parallel basis update & Galerkin integrator, which simultaneously evolves all low-rank factors, has recently been derived for matrix differential equations. This work extends the parallel low-rank matrix integrator to Tucker tensors and general tree tensor networks, yielding an algorithm in which all bases and connecting tensors are evolved in parallel over a time step. We formulate the algorithm, provide a robust error bound, and demonstrate the efficiency of the new integrators for problems in quantum many-body physics, uncertainty quantification, and radiative transfer.

Low-Rank Adaptation (LoRA) has become a widely used method for parameter-efficient fine-tuning of large-scale, pre-trained neural networks. However, LoRA and its extensions face several challenges, including the need for rank adaptivity, robustness, and computational efficiency during the fine-tuning process. We introduce GeoLoRA, a novel approach that addresses these limitations by leveraging dynamical low-rank approximation theory. GeoLoRA requires only a single backpropagation pass over the small-rank adapters, significantly reducing computational cost as compared to similar dynamical low-rank training methods and making it faster than popular baselines such as AdaLoRA. This allows GeoLoRA to efficiently adapt the allocated parameter budget across the model, achieving smaller low-rank adapters compared to heuristic methods like AdaLoRA and LoRA, while maintaining critical convergence, descent, and error-bound theoretical guarantees. The resulting method is not only more efficient but also more robust to varying hyperparameter settings. We demonstrate the effectiveness of GeoLoRA on several state-of-the-art benchmarks, showing that it outperforms existing methods in both accuracy and computational efficiency.

This work proposes and analyzes a new class of numerical integrators for computing low-rank approximations to solutions of matrix differential equation. We combine an explicit Runge-Kutta method with repeated randomized low-rank approximation to keep the rank of the stages limited. The so-called generalized Nystr\"om method is particularly well suited for this purpose; it builds low-rank approximations from random sketches of the discretized dynamics. In contrast, all existing dynamical low-rank approximation methods are deterministic and usually perform tangent space projections to limit rank growth. Using such tangential projections can result in larger error compared to approximating the dynamics directly. Moreover, sketching allows for increased flexibility and efficiency by choosing structured random matrices adapted to the structure of the matrix differential equation. Under suitable assumptions, we establish moment and tail bounds on the error of our randomized low-rank Runge-Kutta methods. When combining the classical Runge-Kutta method with generalized Nystr\"om, we obtain a method called Rand RK4, which exhibits fourth-order convergence numerically -- up to the low-rank approximation error. For a modified variant of Rand RK4, we also establish fourth-order convergence theoretically. Numerical experiments for a range of examples from the literature demonstrate that randomized low-rank Runge-Kutta methods compare favorably with two popular dynamical low-rank approximation methods, in terms of robustness and speed of convergence., Comment: 27 pages

Compactly representing and efficently applying linear operators are fundamental ingredients in tensor network methods for simulating quantum many-body problems and solving high-dimensional problems in scientific computing. In this work, we study such representations for tree tensor networks, the so called tree tensor network operators (TTNOs), paying particular attention to Hamiltonian operators that involve long-range pairwise interactions between particles. Generalizing the work by Lin, Tong, and others on matrix product operators, we establish a direct connection between the hierarchical low-rank structure of the interaction matrix and the TTNO property. This connection allows us to arrive at very compact TTNO representations by compressing the interaction matrix into a hierarchically semi-separable matrix. Numerical experiments for different quantum spin systems validate our results and highlight the potential advantages of TTNOs over matrix product operators.

Dynamical low-rank approximation has become a valuable tool to perform an on-the-fly model order reduction for prohibitively large matrix differential equations. A core ingredient is the construction of integrators that are robust to the presence of small singular values and the resulting large time derivatives of the orthogonal factors in the low-rank matrix representation. Recently, the robust basis-update & Galerkin (BUG) class of integrators has been introduced. These methods require no steps that evolve the solution backward in time, often have favourable structure-preserving properties, and allow for parallel time-updates of the low-rank factors. The BUG framework is flexible enough to allow for adaptations to these and further requirements. However, the BUG methods presented so far have only first-order robust error bounds. This work proposes a second-order BUG integrator for dynamical low-rank approximation based on the midpoint rule. The integrator first performs a half-step with a first-order BUG integrator, followed by a Galerkin update with a suitably augmented basis. We prove a robust second-order error bound which in addition shows an improved dependence on the normal component of the vector field. These rigorous results are illustrated and complemented by a number of numerical experiments.

The cluster structure of the neutron-rich isotope $^{10}$Be has been probed via the $(p,p\alpha)$ reaction at 150 MeV/nucleon in inverse kinematics and in quasifree conditions. The populated states of $^{6}$He residues were investigated through missing mass spectroscopy. The triple differential cross-section for the ground-state transition was extracted for quasifree angle pairs ($\theta_{p}$, $\theta_{\alpha}$) and compared to distorted-wave impulse approximation reaction calculations performed in a microscopic framework using successively the Tohsaki-Horiuchi-Schuck-R\"opke product wave-function and the wave-function deduced from Antisymmetrized Molecular Dynamics calculations. The remarkable agreement between calculated and measured cross-sections in both shape and magnitude validates the molecular structure description of the $^{10}$Be ground-state, configured as an $\alpha$-$\alpha$ core with two valence neutrons occupying $\pi$-type molecular orbitals., Comment: 7 pages, 3 figures, 1 tables

In algorithms for solving optimization problems constrained to a smooth manifold, retractions are a well-established tool to ensure that the iterates stay on the manifold. More recently, it has been demonstrated that retractions are a useful concept for other computational tasks on manifold as well, including interpolation tasks. In this work, we consider the application of retractions to the numerical integration of differential equations on fixed-rank matrix manifolds. This is closely related to dynamical low-rank approximation (DLRA) techniques. In fact, any retraction leads to a numerical integrator and, vice versa, certain DLRA techniques bear a direct relation with retractions. As an example for the latter, we introduce a new retraction, called KLS retraction, that is derived from the so-called unconventional integrator for DLRA. We also illustrate how retractions can be used to recover known DLRA techniques and to design new ones. In particular, this work introduces two novel numerical integration schemes that apply to differential equations on general manifolds: the accelerated forward Euler (AFE) method and the Projected Ralston--Hermite (PRH) method. Both methods build on retractions by using them as a tool for approximating curves on manifolds. The two methods are proven to have local truncation error of order three. Numerical experiments on classical DLRA examples highlight the advantages and shortcomings of these new methods.

With the growth of model and data sizes, a broad effort has been made to design pruning techniques that reduce the resource demand of deep learning pipelines, while retaining model performance. In order to reduce both inference and training costs, a prominent line of work uses low-rank matrix factorizations to represent the network weights. Although able to retain accuracy, we observe that low-rank methods tend to compromise model robustness against adversarial perturbations. By modeling robustness in terms of the condition number of the neural network, we argue that this loss of robustness is due to the exploding singular values of the low-rank weight matrices. Thus, we introduce a robust low-rank training algorithm that maintains the network's weights on the low-rank matrix manifold while simultaneously enforcing approximate orthonormal constraints. The resulting model reduces both training and inference costs while ensuring well-conditioning and thus better adversarial robustness, without compromising model accuracy. This is shown by extensive numerical evidence and by our main approximation theorem that shows the computed robust low-rank network well-approximates the ideal full model, provided a highly performing low-rank sub-network exists.

Reducing parameter redundancies in neural network architectures is crucial for achieving feasible computational and memory requirements during training and inference phases. Given its easy implementation and flexibility, one promising approach is layer factorization, which reshapes weight tensors into a matrix format and parameterizes them as the product of two small rank matrices. However, this approach typically requires an initial full-model warm-up phase, prior knowledge of a feasible rank, and it is sensitive to parameter initialization. In this work, we introduce a novel approach to train the factors of a Tucker decomposition of the weight tensors. Our training proposal proves to be optimal in locally approximating the original unfactorized dynamics independently of the initialization. Furthermore, the rank of each mode is dynamically updated during training. We provide a theoretical analysis of the algorithm, showing convergence, approximation and local descent guarantees. The method's performance is further illustrated through a variety of experiments, showing remarkable training compression rates and comparable or even better performance than the full baseline and alternative layer factorization strategies.

This work introduces a parallel and rank-adaptive matrix integrator for dynamical low-rank approximation. The method is related to the previously proposed rank-adaptive basis update & Galerkin (BUG) integrator but differs significantly in that all arising differential equations, both for the basis and the Galerkin coefficients, are solved in parallel. Moreover, this approach eliminates the need for a potentially costly coefficient update with augmented basis matrices. The integrator also incorporates a new step rejection strategy that enhances the robustness of both the parallel integrator and the BUG integrator. By construction, the parallel integrator inherits the robust error bound of the BUG and projector-splitting integrators. Comparisons of the parallel and BUG integrators are presented by a series of numerical experiments which demonstrate the efficiency of the proposed method, for problems from radiative transfer and radiation therapy.

Open quantum systems provide a conceptually simple setting for the exploration of collective behavior stemming from the competition between quantum effects, many-body interactions, and dissipative processes. They may display dynamics distinct from that of closed quantum systems or undergo nonequilibrium phase transitions which are not possible in classical settings. However, studying open quantum many-body dynamics is challenging, in particular in the presence of critical long-range correlations or long-range interactions. Here, we make progress in this direction and introduce a numerical method for open quantum systems, based on tree tensor networks. Such a structure is expected to improve the encoding of many-body correlations and we adopt an integration scheme suited for long-range interactions and applications to dissipative dynamics. We test the method using a dissipative Ising model with power-law decaying interactions and observe signatures of a first-order phase transition for power-law exponents smaller than one., Comment: 7+3 pages, 4 figures

With the start of a new Great Observatories era, there is renewed concern that the demand for these forefront facilities, through proposal pressure, will exceed conventional peer-review management's capacity for ensuring an unbiased and efficient selection. There is need for new methods, strategies, and tools to facilitate those reviews. Here, we describe PACMan2, an updated tool for proposal review management that utilizes machine learning models and techniques to topically categorize proposals and reviewers, to match proposals to reviewers, and to facilitate proposal assignments, mitigating some conflicts of interest. We find that the classifier has cross-validation accuracy of $80.0\pm2.2\%$ on proposals for time on the Hubble Space Telescope and the James Webb Space Telescope., Comment: 10 pages, 7 figures

Neural networks have achieved tremendous success in a large variety of applications. However, their memory footprint and computational demand can render them impractical in application settings with limited hardware or energy resources. In this work, we propose a novel algorithm to find efficient low-rank subnetworks. Remarkably, these subnetworks are determined and adapted already during the training phase and the overall time and memory resources required by both training and evaluating them are significantly reduced. The main idea is to restrict the weight matrices to a low-rank manifold and to update the low-rank factors rather than the full matrix during training. To derive training updates that are restricted to the prescribed manifold, we employ techniques from dynamic model order reduction for matrix differential equations. This allows us to provide approximation, stability, and descent guarantees. Moreover, our method automatically and dynamically adapts the ranks during training to achieve the desired approximation accuracy. The efficiency of the proposed method is demonstrated through a variety of numerical experiments on fully-connected and convolutional networks.

A rank-adaptive integrator for the approximate solution of high-order tensor differential equations by tree tensor networks is proposed and analyzed. In a recursion from the leaves to the root, the integrator updates bases and then evolves connection tensors by a Galerkin method in the augmented subspace spanned by the new and old bases. This is followed by rank truncation within a specified error tolerance. The memory requirements are linear in the order of the tensor and linear in the maximal mode dimension. The integrator is robust to small singular values of matricizations of the connection tensors. Up to the rank truncation error, which is controlled by the given error tolerance, the integrator preserves norm and energy for Schrodinger equations, and it dissipates the energy in gradient systems. Numerical experiments with a basic quantum spin system illustrate the behavior of the proposed algorithm.

The past few decades have seen major developments in the design and operation of cryogenic particle detectors. This technology offers an extremely good energy resolution - comparable to semiconductor detectors - and a wide choice of target materials, making low temperature calorimetric detectors ideal for a variety of particle physics applications. Rare event searches have continued to require ever greater exposures, which has driven them to ever larger cryogenic detectors, with the CUORE experiment being the first to reach a tonne-scale, mK-cooled, experimental mass. CUORE, designed to search for neutrinoless double beta decay, has been operational since 2017 at a temperature of about 10 mK. This result has been attained by the use of an unprecedentedly large cryogenic infrastructure called the CUORE cryostat: conceived, designed and commissioned for this purpose. In this article the main characteristics and features of the cryogenic facility developed for the CUORE experiment are highlighted. A brief introduction of the evolution of the field and of the past cryogenic facilities are given. The motivation behind the design and development of the CUORE cryogenic facility is detailed as are the steps taken toward realization, commissioning, and operation of the CUORE cryostat. The major challenges overcome by the collaboration and the solutions implemented throughout the building of the cryogenic facility will be discussed along with the potential improvements for future facilities. The success of CUORE has opened the door to a new generation of large-scale cryogenic facilities in numerous fields of science. Broader implications of the incredible feat achieved by the CUORE collaboration on the future cryogenic facilities in various fields ranging from neutrino and dark matter experiments to quantum computing will be examined., Comment: 45 pages, 14 figures

The dynamical low-rank approximation (DLRA) is used to treat high-dimensional problems that arise in such diverse fields as kinetic transport and uncertainty quantification. Even though it is well known that certain spatial and temporal discretizations when combined with the DLRA approach can result in numerical instability, this phenomenon is poorly understood. In this paper we perform a $L^2$ stability analysis for the corresponding nonlinear equations of motion. This reveals the source of the instability for the projector splitting integrator when first discretizing the equations and then applying the DLRA. Based on this we propose a projector splitting integrator, based on applying DLRA to the continuous system before performing the discretization, that recovers the classic CFL condition. We also show that the unconventional integrator has more favorable stability properties and explain why the projector splitting integrator performs better when approximating higher moments, while the unconventional integrator is generally superior for first order moments. Furthermore, an efficient and stable dynamical low-rank update for the scattering term in kinetic transport is proposed. Numerical experiments for kinetic transport and uncertainty quantification, which confirm the results of the stability analysis, are presented.