Search

Deep neural networks (DNNs) exhibit a remarkable ability to automatically learn data representations, finding appropriate features without human input. Here we present a method for analysing feature learning by decomposing DNNs into 1) a forward feature-map $\Phi$ that maps the input dataspace to the post-activations of the penultimate layer, and 2) a final linear layer that classifies the data. We diagonalize $\Phi$ with respect to the gradient descent operator and track feature learning by measuring how the eigenfunctions and eigenvalues of $\Phi$ change during training. Across many popular architectures and classification datasets, we find that DNNs converge, after just a few epochs, to a minimal feature (MF) regime dominated by a number of eigenfunctions equal to the number of classes. This behaviour resembles the neural collapse phenomenon studied at longer training times. For other DNN-data combinations, such as a fully connected network on CIFAR10, we find an extended feature (EF) regime where significantly more features are used. Optimal generalisation performance upon hyperparameter tuning typically coincides with the MF regime, but we also find examples of poor performance within the MF regime. Finally, we recast the phenomenon of neural collapse into a kernel picture which can be extended to broader tasks such as regression.

In this paper, we introduce and study the Dirichlet series enumerating (proper) equivalence classes of full rank subforms/sublattices of a given quadratic form/lattice, focusing on the positive definite binary case. We obtain formulas linking this Dirichlet series with Dirichlet series counting ideal classes of the imaginary quadratic field associated with the quadratic form. Utilizing the result, we provide explicit formulas of the Dirichlet series for several lattices, including square lattice and hexagonal lattice. Moreover, we investigate some analytic properties of this Dirichlet series., Comment: 37 pages

Given a totally real number field $F$, we show that there are only finitely many totally real extensions of $K$ of a fixed degree that admit a universal quadratic form defined over $F$. We further obtain several explicit classification results in the case of relative quadratic extensions., Comment: 17 pages

Deep learning models can exhibit what appears to be a sudden ability to solve a new problem as training time, training data, or model size increases, a phenomenon known as emergence. In this paper, we present a framework where each new ability (a skill) is represented as a basis function. We solve a simple multi-linear model in this skill-basis, finding analytic expressions for the emergence of new skills, as well as for scaling laws of the loss with training time, data size, model size, and optimal compute. We compare our detailed calculations to direct simulations of a two-layer neural network trained on multitask sparse parity, where the tasks in the dataset are distributed according to a power-law. Our simple model captures, using a single fit parameter, the sigmoidal emergence of multiple new skills as training time, data size or model size increases in the neural network., Comment: Accepted at NeurIPS 2024 Conference

Lifting problem for universal quadratic forms asks for totally real number fields $K$ that admit a positive definite quadratic form with coefficients in $\mathbb{Z}$ that is universal over the ring of integers of $K$. In this paper, we show that $K=\mathbb{Q}(\zeta_7+\zeta_7^{-1})$ is the only such totally real cubic field. Moreover, we show that there is no such biquadratic field., Comment: 12 pages

Given $c,$ a positive integer, we give an explicit formula and an asymptotic formula for \[ \sum\chi(c)|L(1,\,\chi)|^{2}, \] where $\chi$ is the non-trivial Dirichlet character mod $f$ with $f>c.$

We prove a scheme-theoretic version of Mnev-Sturmfels Universality, suitable to be used in the proof of Murphy's Law in Algebraic Geometry. Somewhat more precisely, we show that any singularity type of finite type over Z appears on some incidence scheme of points and lines, subject to some particular further constraints., Comment: 14 pages, 5 figures; revised thanks to suggestions of referees and other readers