Search

Transformers are extremely successful machine learning models whose mathematical properties remain poorly understood. Here, we rigorously characterize the behavior of transformers with hardmax self-attention and normalization sublayers as the number of layers tends to infinity. By viewing such transformers as discrete-time dynamical systems describing the evolution of points in a Euclidean space, and thanks to a geometric interpretation of the self-attention mechanism based on hyperplane separation, we show that the transformer inputs asymptotically converge to a clustered equilibrium determined by special points called leaders. We then leverage this theoretical understanding to solve sentiment analysis problems from language processing using a fully interpretable transformer model, which effectively captures `context' by clustering meaningless words around leader words carrying the most meaning. Finally, we outline remaining challenges to bridge the gap between the mathematical analysis of transformers and their real-life implementation., Comment: 23 pages, 10 figures, 1 table. Funded by the European Union (Horizon Europe MSCA project ModConFlex, grant number 101073558). Accompanying code available at: https://github.com/DCN-FAU-AvH/clustering-hardmax-transformers

We consider an internally heated fluid between parallel plates with fixed thermal fluxes. For a large class of heat sources that vary in the direction of gravity, we prove that $\langle\delta T \rangle_h \geq \sigma R^{-1/3} - \mu$, where $\langle\delta T \rangle_h$ is the average temperature difference between the bottom and top plates, $R$ is a `flux' Rayleigh number and the constants $\sigma,\mu >0$ depend on the geometric properties of the internal heating. This result implies that mean downward conduction (for which $\langle\delta T \rangle_h< 0$) is impossible for a range of Rayleigh numbers smaller than a critical value $R_0$. The bound demonstrates that $R_0$ depends on the heating distribution and can be made arbitrarily large by concentrating the heating near the bottom plate. However, for any given fixed heating profile of the class we consider, the corresponding value of $R_0$ is always finite. This points to a fundamental difference between internally heated convection and its limiting case of Rayleigh-B\'enard convection with fixed flux boundary conditions, for which $\langle\delta T \rangle_h$ is known to be positive for all $R$., Comment: 13 pages, 4 figures

In this study, we conduct atomistic-level molecular dynamics simulations on fixed-sized silicon-germanium (Si$_{1-x}$Ge$_{x}$) crystals to elucidate the effects of dopant concentration and temperature on the crystalline inter-planar distances. Our calculations consider a range of Ge dopant concentrations between pure Si (0%) and 15%, and for both the optimised system state and a temperature of 300 K. We observe a linear relationship between Ge concentration and inter-planar distance and lattice constant, in line with the approximation of Vegard's Law, and other experimental and computational results. These findings will be employed in conjunction with future studies to establish precise tolerances for use in crystal growth, crucial for the manufacture of crystals intended for emerging gamma-ray crystal-based light source technologies.

Invariant measures encode the long-time behaviour of a dynamical system. In this work, we propose an optimization-based method to discover invariant measures directly from data gathered from a system. Our method does not require an explicit model for the dynamics and allows one to target specific invariant measures, such as physical and ergodic measures. Moreover, it applies to both deterministic and stochastic dynamics in either continuous or discrete time. We provide convergence results and illustrate the performance of our method on data from the logistic map and a stochastic double-well system, for which invariant measures can be found by other means. We then use our method to approximate the physical measure of the chaotic attractor of the R\"ossler system, and we extract unstable periodic orbits embedded in this attractor by identifying discrete-time periodic points of a suitably defined Poincar\'e map. This final example is truly data-driven and shows that our method can significantly outperform previous approaches based on model identification., Comment: Fix small mistake in analysis, extended examples (incl. comparison with alternative methods)

We consider finite element approximations to the optimal constant for the Hardy inequality with exponent $p=2$ in bounded domains of dimension $n=1$ or $n \geq 3$. For finite element spaces of piecewise linear and continuous functions on a mesh of size $h$, we prove that the approximate Hardy constant converges to the optimal Hardy constant at a rate proportional to $1/| \log h |^2$. This result holds in dimension $n=1$, in any dimension $n \geq 3$ if the domain is the unit ball and the finite element discretization exploits the rotational symmetry of the problem, and in dimension $n=3$ for general finite element discretizations of the unit ball. In the first two cases, our estimates show excellent quantitative agreement with values of the discrete Hardy constant obtained computationally., Comment: Review: Significantly improved estimates compared to the original version (23 pages, 6 figures)

We describe a `discretize-then-relax' strategy to globally minimize integral functionals over functions $u$ in a Sobolev space subject to Dirichlet boundary conditions. The strategy applies whenever the integral functional depends polynomially on $u$ and its derivatives, even if it is nonconvex. The `discretize' step uses a bounded finite element scheme to approximate the integral minimization problem with a convergent hierarchy of polynomial optimization problems over a compact feasible set, indexed by the decreasing size $h$ of the finite element mesh. The `relax' step employs sparse moment-sum-of-squares relaxations to approximate each polynomial optimization problem with a hierarchy of convex semidefinite programs, indexed by an increasing relaxation order $\omega$. We prove that, as $\omega\to\infty$ and $h\to 0$, solutions of such semidefinite programs provide approximate minimizers that converge in a suitable sense (including in certain $L^p$ norms) to the global minimizer of the original integral functional if it is unique. We also report computational experiments showing that our numerical strategy works well even when technical conditions required by our theoretical analysis are not satisfied., Comment: Improved main results, added section on computational complexity. 27 pages, 9 figures

The bistable deployable composite boom (Bi-DCB) can achieve bistable function by storing and releasing strain energy, which has a good application prospect in space field. For example, it serves as the main support section of deployable structures (e.g., solar arrays and antennas). This paper investigates the folding stable state of the Bi-DCB through the analytical method. Based on the classical Archimedes' helix, the geometrical model of the Bi-DCB was established. Using energy principle, an analytical model for predicting the folding stable state of the Bi-DCB was presented. The failure indices of six Bi-DCBs in the folding stable state were calculated using the Tsai-Hill criterion and the maximum stress criterion. To validate the analytical model proposed in this paper, the prediction results were compared with the results of two Finite Element Models (FEMs) and experimental results, and the four were in good agreement. Finally, the effect of geometric parameters (i.e., radius of cross-section, thickness and length) on the folding stable state of the Bi-DCB was further investigated with the aid of the analytical model. It is shown that geometric parameters are one of the key factors affecting the folding stable state of the Bi-DCB., Comment: arXiv admin note: substantial text overlap with arXiv:2305.05289

The bistable deployable composite boom (Bi-DCB) can realize the bistable function by storing and releasing strain energy, which has a good application prospect in the aerospace field. In this paper, the folding stable state of the Bi-DCB was investigated using experimental and numerical approaches. Using the vacuum bag method, six Bi-DCB specimens were prepared. Bistable experiments of Bi-DCB specimens were conducted and linear fitting with Archimedes' helix was performed to determine the folding stable configuration. In addition, two Finite Element Models (FEMs) were established for predicting the folding stable state of the Bi-DCB. Two classical failure criteria were utilized to analyze the stress level of the folding stable state of the Bi-DCB. Numerical results of two FEMs agreed well with experimental results, including the bistable deformation process and the folding stable state.

We present a flexible data-driven method for dynamical system analysis that does not require explicit model discovery. The method is rooted in well-established techniques for approximating the Koopman operator from data and is implemented as a semidefinite program that can be solved numerically. Furthermore, the method is agnostic of whether data is generated through a deterministic or stochastic process, so its implementation requires no prior adjustments by the user to accommodate these different scenarios. Rigorous convergence results justify the applicability of the method, while also extending and uniting similar results from across the literature. Examples on discovering Lyapunov functions, performing ergodic optimization, and bounding extrema over attractors for both deterministic and stochastic dynamics exemplify these convergence results and demonstrate the performance of the method., Comment: We have significantly expanded the presentation. This has improved the presentation and made the paper more readable. Comments welcome

We use Lyapunov-like functions and convex optimization to propagate uncertainty in the initial condition of nonlinear systems governed by ordinary differential equations. We consider the full nonlinear dynamics without approximation, producing rigorous bounds on the expected future value of a quantity of interest even when only limited statistics of the initial condition (e.g., mean and variance) are known. For dynamical systems evolving in compact sets, the best upper (lower) bound coincides with the largest (smallest) expectation among all initial state distributions consistent with the known statistics. For systems governed by polynomial equations and polynomial quantities of interest, one-sided estimates on the optimal bounds can be computed using tools from polynomial optimization and semidefinite programming. Moreover, these numerical bounds provably converge to the optimal ones in the compact case. We illustrate the approach on a van der Pol oscillator and on the Lorenz system in the chaotic regime., Comment: 6 pages, 2 figures, 2 tables. Accepted to American Control Conference 2023

Internally heated convection involves the transfer of heat by fluid motion between a distribution of sources and sinks. Focusing on the balanced case where the total heat added by the sources matches the heat taken away by the sinks, we obtain \emph{a priori} bounds on the minimum mean thermal dissipation $\langle |\nabla T|^2\rangle$ as a measure of the inefficiency of transport. In the advective limit, our bounds scale with the inverse mean kinetic energy of the flow. The constant in this scaling law depends on the source--sink distribution, as we explain both in a pair of examples involving oscillatory or concentrated heating and cooling, and via a general asymptotic variational principle for optimizing transport. Key to our analysis is the solution of a pure advection equation, which we do to find examples of extreme heat transfer by cellular and `pinching' flows. When the flow obeys a momentum equation, our bound is re-expressed in terms of a flux-based Rayleigh number $Ra$ yielding $\langle |\nabla T|^2\rangle\geq CRa^{-\alpha}$. The power $\alpha$ is $0, 2/3$ or $1$ depending on the arrangement of the sources and sinks relative to gravity., Comment: Minor revisions from review, figure added

We analyze two recently proposed methods to establish a priori lower bounds on the minimum of general integral variational problems. The methods, which involve either `occupation measures' or a `pointwise dual relaxation' procedure, are shown to produce the same lower bound under a coercivity hypothesis ensuring their strong duality. We then show by a minimax argument that the methods actually evaluate the minimum for classes of one-dimensional, scalar-valued, or convex multidimensional problems. For generic problems, however, these methods should fail to capture the minimum and produce non-sharp lower bounds. We demonstrate this using two examples, the first of which is one-dimensional and scalar-valued with a non-convex constraint, and the second of which is multidimensional and non-convex in a different way. The latter example emphasizes the existence in multiple dimensions of nonlinear constraints on gradient fields that are ignored by occupation measures, but are built into the finer theory of gradient Young measures., Comment: 24 pages, updated references, minor modifications to text

New bounds are proven on the mean vertical convective heat transport, $\overline{\langle wT \rangle}$, for uniform internally heated (IH) convection in the limit of infinite Prandtl number. For fluid in a horizontally-periodic layer between isothermal boundaries, we show that $\overline{\langle wT \rangle} \leq \frac12 - c R^{-2}$, where $R$ is a nondimensional `flux' Rayleigh number quantifying the strength of internal heating and $c = 216$. Then, $\overline{\langle wT \rangle} = 0$ corresponds to vertical heat transport by conduction alone, while $\overline{\langle wT \rangle} > 0$ represents the enhancement of vertical heat transport upwards due to convective motion. If, instead, the lower boundary is a thermal insulator, then we obtain $\overline{\langle wT \rangle} \leq \frac12 - c R^{-4}$, with $c\approx 0.0107$. This result implies that the Nusselt number $Nu$, defined as the ratio of the total-to-conductive heat transport, satisfies $Nu \lesssim R^{4}$. Both bounds are obtained by combining the background method with a minimum principle for the fluid's temperature and with Hardy--Rellich inequalities to exploit the link between the vertical velocity and temperature. In both cases, power-law dependence on $R$ improves the previously best-known bounds, which, although valid at both infinite and finite Prandtl numbers, approach the uniform bound exponentially with $R$., Comment: 28 pages, 5 figures

Naphthalene (C10H8) is the simplest polycyclic aromatic hydrocarbon (PAH) and an important component in a series of astrochemical reactions involving hydrocarbons. Its molecular charge state affects the stability of its isomeric structures, which is specially relevant in ionised astrophysical environments. We thus perform an extensive computational search for low-energy molecular structures of neutral, singly, and multiply charged naphthalene and its isomers with charge states +q = 0-4 and investigate their geometric properties and bonding situations. We find that isomerisation reactions should be frequent for higher charged states and that open chains dominate their low-energy structures. We compute both the scaled-harmonic and anharmonic infrared spectra of selected low-energy species and provide the calculated scaling factors for the naphthalene neutral, cation, and dication global minima. All simulated spectra reproduce satisfactorily the experimental data and, thus, are adequate for aiding observations. Moreover, the potential presence of these species in the emission spectra of the circumnuclear regions of active galactic nuclei (AGNs), with high energetic X-ray photon fluxes, is explored using the experimental value of the naphthalene photodissociation cross-section, \sigma_{ph-d}, to determine its half-life, t_{1/2}, at a photon energy of 2.5 keV in a set of relevant sources. Finally, we show that the computed IR bands of the triply and quadruply charged species are able to reproduce some features of the selected AGN sources., Comment: Accepted for publication in the Monthly Notices of the Royal Astronomical Society. 15 pages, 7 pages, supplementary information available online

Motivated by the application of Lyapunov methods to partial differential equations (PDEs), we study functional inequalities of the form $f(I_1(u),\ldots,I_k(u))\geq 0$ where $f$ is a polynomial, $u$ is any function satisfying prescribed constraints, and $I_1(u),\ldots,I_k(u)$ are integral functionals whose integrands are polynomial in $u$, its derivatives, and the integration variable. We show that such functional inequalities can be strengthened into sufficient polynomial inequalities, which in principle can be checked via semidefinite programming using standard techniques for polynomial optimization. These sufficient conditions can be used also to optimize functionals with affine dependence on tunable parameters whilst ensuring their nonnegativity. Our approach relies on a measure-theoretic lifting of the original functional inequality, which extends both a recent moment relaxation strategy for PDE analysis and a dual approach to inequalities for integral functionals., Comment: 8 pages. Submitted to IFAC for possible publication

We obtain an analytical bound on the mean vertical convective heat flux $\langle w T \rangle$ between two parallel boundaries driven by uniform internal heating. We consider two configurations, one with both boundaries held at the same constant temperature, and the other one with a top boundary held at constant temperature and a perfectly insulating bottom boundary. For the first configuration, Arslan et al. (J. Fluid Mech. 919:A15, 2021) recently provided numerical evidence that Rayleigh-number-dependent corrections to the only known rigorous bound $\langle w T \rangle \leq 1/2$ may be provable if the classical background method is augmented with a minimum principle stating that the fluid's temperature is no smaller than that of the top boundary. Here, we confirm this fact rigorously for both configurations by proving bounds on $\langle wT \rangle$ that approach $1/2$ exponentially from below as the Rayleigh number is increased. The key to obtaining these bounds are inner boundary layers in the background fields with a particular inverse-power scaling, which can be controlled in the spectral constraint using Hardy and Rellich inequalities. These allow for qualitative improvements in the analysis not available to standard constructions., Comment: 20 pages, 3 figures

We present a method for finding lower bounds on the global infima of integral variational problems, wherein $\int_\Omega f(x,u(x),\nabla u(x)){\rm d}x$ is minimized over functions $u\colon\Omega\subset\mathbb{R}^n\to\mathbb{R}^m$ satisfying given equality or inequality constraints. Each constraint may be imposed over $\Omega$ or its boundary, either pointwise or in an integral sense. These global minimizations are generally non-convex and intractable. We formulate a particular convex maximization, here called the pointwise dual relaxation (PDR), whose supremum is a lower bound on the infimum of the original problem. The PDR can be derived by dualizing and relaxing the original problem; its constraints are pointwise equalities or inequalities over finite-dimensional sets, rather than over infinite-dimensional function spaces. When the original minimization can be specified by polynomial functions of $(x,u,\nabla u)$, the PDR can be further relaxed by replacing pointwise inequalities with polynomial sum-of-squares (SOS) conditions. The resulting SOS program is computationally tractable when the dimensions $m,n$ and number of constraints are not too large. The framework presented here generalizes an approach of Valmorbida, Ahmadi, and Papachristodoulou (IEEE Trans. Automat. Contr., 61:1649--1654, 2016). We prove that the optimal lower bound given by the PDR is sharp for several classes of problems, whose special cases include leading eigenvalues of Sturm-Liouville problems and optimal constants of Poincar\'e inequalities. For these same classes, we prove that SOS relaxations of the PDR converge to the sharp lower bound as polynomial degrees are increased. Convergence of SOS computations in practice is illustrated for several examples.

The background method is a widely used technique to bound mean properties of turbulent flows rigorously. This work reviews recent advances in the theoretical formulation and numerical implementation of the method. First, we describe how the background method can be formulated systematically within a broader "auxiliary function" framework for bounding mean quantities, and explain how symmetries of the flow and constraints such as maximum principles can be exploited. All ideas are presented in a general setting and are illustrated on Rayleigh-B\'enard convection between stress-free isothermal plates. Second, we review a semidefinite programming approach and a timestepping approach to optimizing bounds computationally, revealing that they are related to each other through convex duality and low-rank matrix factorization. Open questions and promising directions for further numerical analysis of the background method are also outlined., Comment: Revised version: 29 pages, 3 figures, 2 tables. Added simple example to Section 1 and complexity estimates to Section 4. Improved discussion in Section 4

Chordal and factor-width decomposition methods for semidefinite programming and polynomial optimization have recently enabled the analysis and control of large-scale linear systems and medium-scale nonlinear systems. Chordal decomposition exploits the sparsity of semidefinite matrices in a semidefinite program (SDP), in order to formulate an equivalent SDP with smaller semidefinite constraints that can be solved more efficiently. Factor-width decompositions, instead, relax or strengthen SDPs with dense semidefinite matrices into more tractable problems, trading feasibility or optimality for lower computational complexity. This article reviews recent advances in large-scale semidefinite and polynomial optimization enabled by these two types of decomposition, highlighting connections and differences between them. We also demonstrate that chordal and factor-width decompositions allow for significant computational savings on a range of classical problems from control theory, and on more recent problems from machine learning. Finally, we outline possible directions for future research that have the potential to facilitate the efficient optimization-based study of increasingly complex large-scale dynamical systems., Comment: 49 pages, 21 figures, 4 tables