Search

We introduce a geometric model of shallow multiplicative exponential linear logic (MELL) using the Hilbert scheme. Building on previous work interpreting multiplicative linear logic proofs as systems of linear equations, we show that shallow MELL proofs can be modeled by locally projective schemes. The key insight is that while multiplicative linear logic proofs correspond to equations between formulas, the exponential fragment of shallow proofs corresponds to equations between these equations. We prove that the model is invariant under cut-elimination by constructing explicit isomorphisms between the schemes associated to proofs related by cut-reduction steps. A key technical tool is the interpretation of the exponential modality using the Hilbert scheme, which parameterizes closed subschemes of projective space. We demonstrate the model through detailed examples, including an analysis of Church numerals that reveals how the Hilbert scheme captures the geometric content of promoted formulas. This work establishes new connections between proof theory and algebraic geometry, suggesting broader relationships between computation and scheme theory.

In this position paper, we argue that understanding the relation between structure in the data distribution and structure in trained models is central to AI alignment. First, we discuss how two neural networks can have equivalent performance on the training set but compute their outputs in essentially different ways and thus generalise differently. For this reason, standard testing and evaluation are insufficient for obtaining assurances of safety for widely deployed generally intelligent systems. We argue that to progress beyond evaluation to a robust mathematical science of AI alignment, we need to develop statistical foundations for an understanding of the relation between structure in the data distribution, internal structure in models, and how these structures underlie generalisation.

Modern deep neural networks display striking examples of rich internal computational structure. Uncovering principles governing the development of such structure is a priority for the science of deep learning. In this paper, we explore the transient ridge phenomenon: when transformers are trained on in-context linear regression tasks with intermediate task diversity, they initially behave like ridge regression before specializing to the tasks in their training distribution. This transition from a general solution to a specialized solution is revealed by joint trajectory principal component analysis. Further, we draw on the theory of Bayesian internal model selection to suggest a general explanation for the phenomena of transient structure in transformers, based on an evolving tradeoff between loss and complexity. We empirically validate this explanation by measuring the model complexity of our transformers as defined by the local learning coefficient., Comment: 37 pages, 27 figures

Mechanistic interpretability aims to understand the computational mechanisms underlying neural networks' capabilities in order to accomplish concrete scientific and engineering goals. Progress in this field thus promises to provide greater assurance over AI system behavior and shed light on exciting scientific questions about the nature of intelligence. Despite recent progress toward these goals, there are many open problems in the field that require solutions before many scientific and practical benefits can be realized: Our methods require both conceptual and practical improvements to reveal deeper insights; we must figure out how best to apply our methods in pursuit of specific goals; and the field must grapple with socio-technical challenges that influence and are influenced by our work. This forward-facing review discusses the current frontier of mechanistic interpretability and the open problems that the field may benefit from prioritizing.

We introduce refined variants of the Local Learning Coefficient (LLC), a measure of model complexity grounded in singular learning theory, to study the development of internal structure in transformer language models during training. By applying these \textit{refined LLCs} (rLLCs) to individual components of a two-layer attention-only transformer, we gain novel insights into the progressive differentiation and specialization of attention heads. Our methodology reveals how attention heads differentiate into distinct functional roles over the course of training, analyzes the types of data these heads specialize to process, and discovers a previously unidentified multigram circuit. These findings demonstrate that rLLCs provide a principled, quantitative toolkit for \textit{developmental interpretability}, which aims to understand models through their evolution across the learning process. More broadly, this work takes a step towards establishing the correspondence between data distributional structure, geometric properties of the loss landscape, learning dynamics, and emergent computational structures in neural networks.

We develop a point of view on reduction of multiplicative proof nets based on quantum error-correcting codes. To each proof net we associate a code, in such a way that cut-elimination corresponds to error correction.

Deep learning involves navigating a high-dimensional loss landscape over the neural network parameter space. Over the course of training, complex computational structures form and re-form inside the neural network, leading to shifts in input/output behavior. It is a priority for the science of deep learning to uncover principles governing the development of neural network structure and behavior. Drawing on the framework of singular learning theory, we propose that model development is deeply linked to degeneracy in the local geometry of the loss landscape. We investigate this link by monitoring loss landscape degeneracy throughout training, as quantified by the local learning coefficient, for a transformer language model and an in-context linear regression transformer. We show that training can be divided into distinct periods of change in loss landscape degeneracy, and that these changes in degeneracy coincide with significant changes in the internal computational structure and the input/output behavior of the transformers. This finding underscores the potential of a degeneracy-based perspective for understanding modern deep learning., Comment: Material on essential dynamics from v1 of this preprint has been removed from v2 and developed in arXiv:2501.17745

We investigate phase transitions in a Toy Model of Superposition (TMS) using Singular Learning Theory (SLT). We derive a closed formula for the theoretical loss and, in the case of two hidden dimensions, discover that regular $k$-gons are critical points. We present supporting theory indicating that the local learning coefficient (a geometric invariant) of these $k$-gons determines phase transitions in the Bayesian posterior as a function of training sample size. We then show empirically that the same $k$-gon critical points also determine the behavior of SGD training. The picture that emerges adds evidence to the conjecture that the SGD learning trajectory is subject to a sequential learning mechanism. Specifically, we find that the learning process in TMS, be it through SGD or Bayesian learning, can be characterized by a journey through parameter space from regions of high loss and low complexity to regions of low loss and high complexity.

The Local Learning Coefficient (LLC) is introduced as a novel complexity measure for deep neural networks (DNNs). Recognizing the limitations of traditional complexity measures, the LLC leverages Singular Learning Theory (SLT), which has long recognized the significance of singularities in the loss landscape geometry. This paper provides an extensive exploration of the LLC's theoretical underpinnings, offering both a clear definition and intuitive insights into its application. Moreover, we propose a new scalable estimator for the LLC, which is then effectively applied across diverse architectures including deep linear networks up to 100M parameters, ResNet image models, and transformer language models. Empirical evidence suggests that the LLC provides valuable insights into how training heuristics might influence the effective complexity of DNNs. Ultimately, the LLC emerges as a crucial tool for reconciling the apparent contradiction between deep learning's complexity and the principle of parsimony., Comment: This version contains new empirical results and merged content from a related paper (arXiv:2402.03698) to provide a more comprehensive study

We associate to every proof structure in multiplicative linear logic an ideal which represents the logical content of the proof as polynomial equations. We show how cut-elimination in multiplicative proof nets corresponds to instances of the Buchberger algorithm for computing Gr\"obner bases in elimination theory.

We re-evaluate universal computation based on the synthesis of Turing machines. This leads to a view of programs as singularities of analytic varieties or, equivalently, as phases of the Bayesian posterior of a synthesis problem. This new point of view reveals unexplored directions of research in program synthesis, of which neural networks are a subset, for example in relation to phase transitions, complexity and generalisation. We also lay the empirical foundations for these new directions by reporting on our implementation in code of some simple experiments., Comment: 16 pages, 7 figures

In singular models, the optimal set of parameters forms an analytic set with singularities and classical statistical inference cannot be applied to such models. This is significant for deep learning as neural networks are singular and thus "dividing" by the determinant of the Hessian or employing the Laplace approximation are not appropriate. Despite its potential for addressing fundamental issues in deep learning, singular learning theory appears to have made little inroads into the developing canon of deep learning theory. Via a mix of theory and experiment, we present an invitation to singular learning theory as a vehicle for understanding deep learning and suggest important future work to make singular learning theory directly applicable to how deep learning is performed in practice.

The Curry-Howard correspondence is often described as relating proofs (in intutionistic natural deduction) to programs (terms in simply-typed lambda calculus). However this narrative is hardly a perfect fit, due to the computational content of cut-elimination and the logical origins of lambda calculus. We revisit Howard's work and interpret it as an isomorphism between a category of proofs in intuitionistic sequent calculus and a category of terms in simply-typed lambda calculus. In our telling of the story the fundamental duality is not between proofs and programs but between local (sequent calculus) and global (lambda calculus or natural deduction) points of view on a common logico-computational mathematical structure.

We introduce the $2$-simplicial Transformer, an extension of the Transformer which includes a form of higher-dimensional attention generalising the dot-product attention, and uses this attention to update entity representations with tensor products of value vectors. We show that this architecture is a useful inductive bias for logical reasoning in the context of deep reinforcement learning.

We study constructive $A_\infty$-models of the DG-category of matrix factorisations of a potential over a commutative $\mathbb{Q}$-algebra $k$, consisting of a Hom-finite $A_\infty$-category equipped with an $A_\infty$-idempotent functor.

We calculate denotations under the Sweedler semantics of the Ehrhard-Regnier derivatives of various encodings of Turing machines into linear logic. We show that these derivatives calculate the rate of change of probabilities naturally arising in the Sweedler semantics of linear logic proofs. The resulting theory is applied to the problem of synthesising Turing machines by gradient descent., Comment: 62 pages, moved the section on naive Bayesian observers earlier (Section 6.2) with slight changes to notation, references added in the introduction to Section 7 and related work in Remark 7.16

We give several different encodings of the step function of a Turing machine in intuitionistic linear logic, and calculate the denotations of these encodings in the Sweedler semantics., Comment: 50 pages

We prove that the semantics of intuitionistic linear logic in vector spaces which uses cofree coalgebras is also a model of differential linear logic, and that the Cartesian closed category of cofree coalgebras is a model of the simply-typed differential lambda calculus., Comment: New introduction to the cofree coalgebra and relevant geometric intuition, and a discussion of the differential lambda calculus

We give an introduction to logic tailored for algebraists, explaining how proofs in linear logic can be viewed as algorithms for constructing morphisms in symmetric closed monoidal categories with additional structure. This is made explicit by showing how to represent proofs in linear logic as linear maps between vector spaces. The interesting part of this vector space semantics is based on the cofree cocommutative coalgebra of Sweedler., Comment: v2: the article has been substantially rewritten to improve the exposition, some false statements about cut-elimination were corrected, a new section about second-order linear logic was added, and the material on geometry of interaction has been removed (to be published elsewhere), v3: fixed typos, added references

We give a direct proof of a result of Sweedler describing the cofree cocommutative coalgebra over a vector space, and use our approach to give an explicit construction of liftings of maps into this universal coalgebra. The basic ingredients in our approach are local cohomology and residues., Comment: v2: improved some notation and fixed typos. Published in J. Pure and Applied Algebra

The bicategory $\mathcal{LG}$ of Landau-Ginzburg models has polynomials as objects and matrix factorisations as $1$-morphisms. The composition of these $1$-morphisms produces infinite rank matrix factorisations, which is a nuisance. In this paper we define a bicategory which is equivalent to $\mathcal{LG}$ in which composition of $1$-morphisms produces finite rank matrix factorisations equipped with the action of a Clifford algebra. This amounts to a finite model of composition in $\mathcal{LG}$., Comment: Third version, minor corrections

We review the results of arXiv:1208.1481 on orientation reversal and duality for defects in topological Landau-Ginzburg models, with the intention of providing an easily accessible toolkit for computations. As an example we include a proof of the main result on adjunctions in a special case, using Pauli matrices. We also explain how to compute arbitrary correlators of defect-decorated planar worldsheets, and briefly discuss the relation to generalised orbifolds., Comment: 11 pages, contribution to the String-Math 2012 proceedings

We study the bicategory of Landau-Ginzburg models, which has potentials as objects and matrix factorisations as 1-morphisms. Our main result is the existence of adjoints in this bicategory and a description of evaluation and coevaluation maps in terms of Atiyah classes and homological perturbation. The bicategorical perspective offers a unified approach to Landau-Ginzburg models: we show how to compute arbitrary correlators and recover the full structure of open/closed TFT, including the Kapustin-Li disk correlator and a simple proof of the Cardy condition, in terms of defect operators which in turn are directly computable from the adjunctions., Comment: 58 pages; v2: Fixed typos and references, removed comments about graded matrix factorisations; v3: exposition improved and shortened, main result now holds over an arbitrary ring k; v4: many improvements to exposition

We compute the categorified sl(N) link invariants as defined by Khovanov and Rozansky, for various links and values of N. This is made tractable by an algorithm for reducing tensor products of matrix factorisations to finite rank, which we implement in the computer algebra package Singular., Comment: 35 pages; v2: exposition shortened and reorganised; v3: typos in Section 4 corrected

We describe the pushforward of a matrix factorisation along a ring morphism in terms of an idempotent defined using relative Atiyah classes, and use this construction to study the convolution of kernels defining integral functors between categories of matrix factorisations. We give an elementary proof of a formula for the Chern character of the convolution generalising the Hirzebruch-Riemann-Roch formula of Polishchuk and Vaintrob., Comment: 43 pages, comments welcome

We provide a new perspective on the Kapustin-Li formula for the duality pairing on the morphism complexes in the matrix factorization category of an isolated hypersurface singularity. In our context, the formula arises as an explicit description of a local duality isomorphism, obtained by using the basic perturbation lemma and Grothendieck residues. The non-degeneracy of the pairing becomes apparent in this setting. Further, we show that the pairing lifts to a Calabi-Yau structure on the matrix factorization category. This allows us to define topological quantum field theories with matrix factorizations as boundary conditions., Comment: 28 pages, 3 figures, comments welcome

We study Serre duality in the singularity category of an isolated Gorenstein singularity and find an explicit formula for the duality pairing in terms of generalised fractions and residues. For hypersurfaces we recover the residue formula of the string theorists Kapustin and Li. These results are obtained from an explicit construction of complete injective resolutions of maximal Cohen-Macaulay modules., Comment: 28 pages, the exposition has been simplified and shortened

We study cocoverings of triangulated categories, in the sense of Rouquier, and prove that for any regular cardinal $\alpha$ the condition of $\alpha$-compactness, in the sense of Neeman, is local with respect to such cocoverings. This was established for ordinary compactness by Rouquier. Our result yields a new technique for proving that a given triangulated category is well-generated. As an application we describe the $\alpha$-compact objects in the unbounded derived category of a quasi-compact semi-separated scheme., Comment: 18 pages, comments welcome

We define a notion of total acyclicity for complexes of flat quasi-coherent sheaves over a semi-separated noetherian scheme, generalising complete flat resolutions over a ring. By studying these complexes as objects of the pure derived category of flat sheaves we extend several results about totally acyclic complexes of projective modules to schemes; for example, we prove that a scheme is Gorenstein if and only if every acyclic complex of flat sheaves is totally acyclic. Our formalism also removes the need for a dualising complex in several known results for rings, including Jorgensen's proof of the existence of Gorenstein projective precovers., Comment: 37 pages, comments welcome

In the recent paper "Mutation in triangulated categories and rigid Cohen-Macaulay modules" Iyama and Yoshino consider two interesting examples of isolated singularities over which it is possible to classify the indecomposable maximal Cohen-Macaulay modules in terms of linear algebra data. In this paper we present two new approaches to these examples. In the first approach we give a relation with cluster categories. In the second approach we use Orlov's result on the graded singularity category. We obtain some new results on the singularity category of isolated singularities which may be interesting in their own right., Comment: The first name of the first author was misspelled in the arXiv abstract. There are no changes in the paper