Search

Investigation of the generalized trigonometric and hyperbolic functions containing two parameters has been a very active research area over the last decade. We believe, however, that their monotonicity and convexity properties with respect to parameters have not been thoroughly studied. In this paper, we make an attempt to fill this gap. Our results are not complete; for some functions, we manage to establish (log)-convexity/concavity in parameters, while for others, we only managed the prove monotonicity, in which case we present necessary and sufficient conditions for convexity/concavity. In the course of the investigation, we found two hypergeometric representations for the generalized cosine and hyperbolic cosine functions which appear to be new. In the last section of the paper, we present four explicit integral evaluations of combinations of generalized trigonometric/hyperbolic functions in terms of hypergeometric functions., Comment: 20 pages; 1 figure

Given the increasing scale of model sizes, novel training strategies like gradual stacking [Gong et al., 2019, Reddi et al., 2023] have garnered interest. Stacking enables efficient training by gradually growing the depth of a model in stages and using layers from a smaller model in an earlier stage to initialize the next stage. Although efficient for training, the model biases induced by such growing approaches are largely unexplored. In this work, we examine this fundamental aspect of gradual stacking, going beyond its efficiency benefits. We propose a variant of gradual stacking called MIDAS that can speed up language model training by up to 40%. Furthermore we discover an intriguing phenomenon: MIDAS is not only training-efficient but surprisingly also has an inductive bias towards improving downstream tasks, especially tasks that require reasoning abilities like reading comprehension and math problems, despite having similar or slightly worse perplexity compared to baseline training. To further analyze this inductive bias, we construct reasoning primitives -- simple synthetic tasks that are building blocks for reasoning -- and find that a model pretrained with stacking is significantly better than standard pretraining on these primitives, with and without fine-tuning. This provides stronger and more robust evidence for this inductive bias towards reasoning. These findings of training efficiency and inductive bias towards reasoning are verified at 1B, 2B and 8B parameter language models. Finally, we conjecture the underlying reason for this inductive bias by exploring the connection of stacking to looped models and provide strong supporting empirical analysis., Comment: Accepted at NeurIPS 2024

An elementary, but very useful lemma due to Biernacki and Krzy\.{z} (1955) asserts that the ratio of two power series inherits monotonicity from that of the sequence of ratios of their respective coefficients. Over the last two decades it has been realized that, under some additional assumptions, similar claims hold for more general series ratios as well as for unimodality in place of monotonicity. This paper continues this line of research: we consider ratios of general functional series and integral transforms and furnish natural sufficiency conditions for preservation of unimodality by such ratios. Numerous series and integral transforms appearing in applications satisfy our sufficiency conditions, including Dirichlet, factorial and inverse factorial series, Laplace, Mellin and generalized Stieltjes transforms, among many others. Finally, we illustrate our general results by exhibiting certain statements on monotonicity patterns for ratios of some special functions. The key role in our considerations is played by the notion of sign regularity., Comment: added theree references and some definitions; corrected various typos and inaccuracies; 18 pages, no figures

Gradient aggregation has long been identified as a major bottleneck in today's large-scale distributed machine learning training systems. One promising solution to mitigate such bottlenecks is gradient compression, directly reducing communicated gradient data volume. However, in practice, many gradient compression schemes do not achieve acceleration of the training process while also preserving accuracy. In this work, we identify common issues in previous gradient compression systems and evaluation methodologies. These include excessive computational overheads; incompatibility with all-reduce; and insufficient evaluation methods, such as not using an end-to-end metric or using a 32-bit baseline instead of the stronger 16-bit baseline. We revisit common compression approaches (sparsification, quantization, and low-rank decomposition) and demonstrate how considering the above issues can lead to minor but strategic design changes, resulting in notably better performance. Our goal is to raise awareness of the need for design and evaluation standards that naturally translate to the end-to-end utility of gradient compression., Comment: Accepted at ACM HotNets 2024. 9 pages, 3 figures

INTRODUCTION: The pharmacological treatment of Major Depressive Disorder (MDD) relies on a trial-and-error approach. We introduce an artificial intelligence (AI) model aiming to personalize treatment and improve outcomes, which was deployed in the Artificial Intelligence in Depression Medication Enhancement (AIDME) Study. OBJECTIVES: 1) Develop a model capable of predicting probabilities of remission across multiple pharmacological treatments for adults with at least moderate major depression. 2) Validate model predictions and examine them for amplification of harmful biases. METHODS: Data from previous clinical trials of antidepressant medications were standardized into a common framework and included 9,042 adults with moderate to severe major depression. Feature selection retained 25 clinical and demographic variables. Using Bayesian optimization, a deep learning model was trained on the training set, refined using the validation set, and tested once on the held-out test set. RESULTS: In the evaluation on the held-out test set, the model demonstrated achieved an AUC of 0.65. The model outperformed a null model on the test set (p = 0.01). The model demonstrated clinical utility, achieving an absolute improvement in population remission rate in hypothetical and actual improvement testing. While the model did identify one drug (escitalopram) as generally outperforming the other drugs (consistent with the input data), there was otherwise significant variation in drug rankings. On bias testing, the model did not amplify potentially harmful biases. CONCLUSIONS: We demonstrate the first model capable of predicting outcomes for 10 different treatment options for patients with MDD, intended to be used at or near the start of treatment to personalize treatment. The model was put into clinical practice during the AIDME randomized controlled trial whose results are reported separately.

Recently, there has been increasing interest in efficient pretraining paradigms for training Transformer-based models. Several recent approaches use smaller models to initialize larger models in order to save computation (e.g., stacking and fusion). In this work, we study the fundamental question of how to select the best growing strategy from a given pool of growing strategies. Prior works have extensively focused on loss- and/or function-preserving behavior at initialization or simply performance at the end of training. Instead, we identify that behavior at initialization can be misleading as a predictor of final performance and present an alternative perspective based on early training dynamics, which we call "landscape-aware growing (LAG)". We perform extensive analysis of correlation of the final performance with performance in the initial steps of training and find early and more accurate predictions of the optimal growing strategy (i.e., with only a small "lag" after initialization). This perspective also motivates an adaptive strategy for gradual stacking.

A quasi-exponential is an entire function of the form $e^{cu}p(u)$, where $p(u)$ is a polynomial and $c \in \mathbb{C}$. Let $V = \langle e^{h_1u}p_1(u), \dots, e^{h_Nu}p_N(u) \rangle$ be a vector space with a basis of quasi-exponentials. We show that if $h_1, \dots, h_N$ are nonnegative and all of the complex zeros of the Wronskian $\operatorname{Wr}(V)$ are real, then $V$ is totally nonnegative in the sense that all of its Grassmann-Pl\"{u}cker coordinates defined by the Taylor expansion about $u=t$ are nonnegative, for any real $t$ greater than all of the zeros of $\operatorname{Wr}(V)$. Our proof proceeds by showing that the higher Gaudin Hamiltonians $T_\lambda^G(t)$ introduced in [ALTZ14] are universal Pl\"ucker coordinates about $u=t$ for the Wronski map on spaces of quasi-exponentials. The result that $V$ is totally nonnegative follows from the fact that $T_\lambda^G(t)$ is positive semidefinite, which we establish using partial traces. We also show that if $h_1 = \cdots = h_N = 0$ then $T_\lambda^G(t)$ equals $\beta^\lambda(t)$, which is the universal Pl\"ucker coordinate for the Wronski map on spaces of polynomials introduced in [KP23]., Comment: 24 pages

The never-ending computational demand from simulations of turbulence makes computational fluid dynamics (CFD) a prime application use case for current and future exascale systems. High-order finite element methods, such as the spectral element method, have been gaining traction as they offer high performance on both multicore CPUs and modern GPU-based accelerators. In this work, we assess how high-fidelity CFD using the spectral element method can exploit the modular supercomputing architecture at scale through domain partitioning, where the computational domain is split between a Booster module powered by GPUs and a Cluster module with conventional CPU nodes. We investigate several different flow cases and computer systems based on the modular supercomputing architecture (MSA). We observe that for our simulations, the communication overhead and load balancing issues incurred by incorporating different computing architectures are seldom worthwhile, especially when I/O is also considered, but when the simulation at hand requires more than the combined global memory on the GPUs, utilizing additional CPUs to increase the available memory can be fruitful. We support our results with a simple performance model to assess when running across modules might be beneficial. As MSA is becoming more widespread and efforts to increase system utilization are growing more important our results give insight into when and how a monolithic application can utilize and spread out to more than one module and obtain a faster time to solution., Comment: 13 pages, 5 figures, 3 tables, preprint

As supercomputers' complexity has grown, the traditional boundaries between processor, memory, network, and accelerators have blurred, making a homogeneous computer model, in which the overall computer system is modeled as a continuous medium with homogeneously distributed computational power, memory, and data movement transfer capabilities, an intriguing and powerful abstraction. By applying a homogeneous computer model to algorithms with a given I/O complexity, we recover from first principles, other discrete computer models, such as the roofline model, parallel computing laws, such as Amdahl's and Gustafson's laws, and phenomenological observations, such as super-linear speedup. One of the homogeneous computer model's distinctive advantages is the capability of directly linking the performance limits of an application to the physical properties of a classical computer system. Applying the homogeneous computer model to supercomputers, such as Frontier, Fugaku, and the Nvidia DGX GH200, shows that applications, such as Conjugate Gradient (CG) and Fast Fourier Transforms (FFT), are rapidly approaching the fundamental classical computational limits, where the performance of even denser systems in terms of compute and memory are fundamentally limited by the speed of light., Comment: 10 pages, 8 figures, 3 tables

Vision tasks are characterized by the properties of locality and translation invariance. The superior performance of convolutional neural networks (CNNs) on these tasks is widely attributed to the inductive bias of locality and weight sharing baked into their architecture. Existing attempts to quantify the statistical benefits of these biases in CNNs over locally connected convolutional neural networks (LCNs) and fully connected neural networks (FCNs) fall into one of the following categories: either they disregard the optimizer and only provide uniform convergence upper bounds with no separating lower bounds, or they consider simplistic tasks that do not truly mirror the locality and translation invariance as found in real-world vision tasks. To address these deficiencies, we introduce the Dynamic Signal Distribution (DSD) classification task that models an image as consisting of $k$ patches, each of dimension $d$, and the label is determined by a $d$-sparse signal vector that can freely appear in any one of the $k$ patches. On this task, for any orthogonally equivariant algorithm like gradient descent, we prove that CNNs require $\tilde{O}(k+d)$ samples, whereas LCNs require $\Omega(kd)$ samples, establishing the statistical advantages of weight sharing in translation invariant tasks. Furthermore, LCNs need $\tilde{O}(k(k+d))$ samples, compared to $\Omega(k^2d)$ samples for FCNs, showcasing the benefits of locality in local tasks. Additionally, we develop information theoretic tools for analyzing randomized algorithms, which may be of interest for statistical research., Comment: 40 pages, 4 figures, Accepted to ICLR 2024, Spotlight

By replacing the Euler gamma function by the Barnes double gamma function in the definition of the Meijer $G$-function, we introduce a new family of special functions, which we call $K$-functions. This is a very general class of functions, which includes as special cases Meijer $G$-functions (thus also all hypergeometric functions ${}_p F_q$) as well as several new functions that appeared recently in the literature. Our goal is to define the $K$-function, study its analytic and transformation properties and relate it to several functions that appeared recently in the study of random processes and the fractional Laplacian. We further introduce a generalization of the Kilbas-Saigo function and show that it is a special case of $K$-function., Comment: 27 pages, 2 figures

We present for the first time an experimental investigation of electrohydrodynamic (EHD) flows within a neutrally buoyant drop with initial radius of 2.25 mm. Utilizing particle image velocimetry (PIV) and high-speed shadowgraphy, we measure the internal circulation and reported velocity profiles in the bulk and at the interface of the drop. Two leaky dielectric liquids, Silicone and Castor oils, are employed as the drop and external phase, allowing for the analysis of two shape configurations: oblate and prolate. The strength of the applied uniform electric field, $E_o$, spans from 0.125 to 1.75 kV/cm, enabling the analysis covering the small-deformation limit, where the leaky dielectric model (LDM) is applicable. Drops with larger deformations, for which no analytical velocity field is available, are also investigated. Our measurements show a good agreement with the LDM theory for the small-deformation cases. The flows begin at the interface as a result of jump in the electric stresses, leading then to four counter-rotating vortices inside the drop. At permanent regime, the analytical solutions adequately predicts the radial and tangential velocity components both in the bulk and at the interface of the drop. However, a nuanced behavior is noticed for larger deformations, where the LDM theory underpredicts the internal circulation. Moreover, due to the increased deformation, a non-uniform azimuthal profile is observed for the velocity at the interface. Transient measurements of this velocity component enlighten the dynamic response of the EHD flows of the drop. Following the currently available analytical solutions, the dynamic response is governed by the time-scale of its deformation. We propose a critical value of electric capillary number of roughly 0.1 below which the LDM adequately describes the velocity field in both quasi steady-state and transitory regimes., Comment: 12 pages, 11 figures

Although statistical learning theory provides a robust framework to understand supervised learning, many theoretical aspects of deep learning remain unclear, in particular how different architectures may lead to inductive bias when trained using gradient based methods. The goal of these lectures is to provide an overview of some of the main questions that arise when attempting to understand deep learning from a learning theory perspective. After a brief reminder on statistical learning theory and stochastic optimization, we discuss implicit bias in the context of benign overfitting. We then move to a general description of the mirror descent algorithm, showing how we may go back and forth between a parameter space and the corresponding function space for a given learning problem, as well as how the geometry of the learning problem may be represented by a metric tensor. Building on this framework, we provide a detailed study of the implicit bias of gradient descent on linear diagonal networks for various regression tasks, showing how the loss function, scale of parameters at initialization and depth of the network may lead to various forms of implicit bias, in particular transitioning between kernel or feature learning., Comment: 66 pages, 20 figures

Given a $d$-dimensional vector space $V \subset \mathbb{C}[u]$ of polynomials, its Wronskian is the polynomial $(u + z_1) \cdots (u + z_n)$ whose zeros $-z_i$ are the points of $\mathbb{C}$ such that $V$ contains a nonzero polynomial with a zero of order at least $d$ at $-z_i$. Equivalently, $V$ is a solution to the Schubert problem defined by osculating planes to the moment curve at $z_1, \dots, z_n$. The inverse Wronski problem involves finding all $V$ with a given Wronskian $(u + z_1) \cdots (u + z_n)$. We solve this problem by providing explicit formulas for the Grassmann-Pl\"ucker coordinates of the general solution $V$, as commuting operators in the group algebra $\mathbb{C}[\mathfrak{S}_n]$ of the symmetric group. The Pl\"ucker coordinates of individual solutions over $\mathbb{C}$ are obtained by restricting to an eigenspace and replacing each operator by its eigenvalue. This generalizes work of Mukhin, Tarasov, and Varchenko (2013) and of Purbhoo (2022), which give formulas in $\mathbb{C}[\mathfrak{S}_n]$ for the differential equation satisfied by $V$. Moreover, if $z_1, \dots, z_n$ are real and nonnegative, then our operators are positive semidefinite, implying that the Pl\"ucker coordinates of $V$ are all real and nonnegative. This verifies several outstanding conjectures in real Schubert calculus, including the positivity conjectures of Mukhin and Tarasov (2017) and of Karp (2021), the disconjugacy conjecture of Eremenko (2015), and the divisor form of the secant conjecture of Sottile (2003). The proofs involve the representation theory of $\mathfrak{S}_n$, symmetric functions, and $\tau$-functions of the KP hierarchy., Comment: 70 pages