Search

Aims: The relative roles of the physical mechanisms involved in quenching galaxy star formation are still unclear. We tackle this fundamental problem with our cosmological semi-empirical model DECODE (Discrete statistical sEmi-empiriCal mODEl), designed to predict galaxy stellar mass assembly histories, from minimal input assumptions. Methods: Specifically, in this work the star formation history of each galaxy is calculated along its progenitor dark matter halo by assigning at each redshift a star formation rate extracted from a monotonic star formation rate-halo accretion rate (SFR-HAR) relation derived from abundance matching between the (observed) SFR function and the (numerically predicted) HAR function, a relation that is also predicted by the TNG100 simulation. SFRs are integrated across cosmic time to build up the mass of galaxies, which may halt their star formation following input physical quenching recipes. Results: In this work we test the popular halo quenching scenario and we find that: 1) the assumption of a monotonic relation between SFR and HAR allows to reproduce the number densities of the bulk of star-forming galaxies in the local Universe; 2) the halo quenching is sufficient to reproduce the statistics of the quenched galaxies and flat (steep) high-mass end of the SMHM relation (SMF); and 3) to align with the observed steep (flat) low-mass end of the SMHM (SMF) additional quenching processes in the least massive haloes are needed. Conclusions: DECODE is an invaluable tool and will pave the way to investigate the origin of newly observed high-redshift objects from the latest ongoing facilities such as JWST and Euclid., Comment: 15 pages, 13 figures, accepted for publication in Astronomy & Astrophysics

Chemical functionalization of diamond surfaces by hydrogen is an important method for controlling the charge state of near-surface fluorescent color centers, an essential process in fabricating devices such as diamond field-effect transistors and chemical sensors, and a required first step for realizing families of more complex terminations through subsequent chemical processing. In all these cases, termination is typically achieved using hydrogen plasma sources which can etch or damage the diamond as well as deposited materials or embedded colour centers. This work explores alternative methods for lower-damage hydrogenation of diamond surfaces, specifically the annealing of diamond samples in high-purity, non-explosive mixtures of nitrogen and hydrogen gas, and the exposure of samples to microwave hydrogen plasmas in the absence of intentional stage heating. The effectiveness of these methods are characterized by x-ray photoelectron spectroscopy, and comparison of the results to density-functional modelling of the surface hydrogenation energetics implicates surface oxygen ligands as the primary factor limiting the termination quality of annealed samples. Finally, photoluminescence spectroscopy is used to verify that both the annealing and reduced sample temperature plasma methods are non-destructive to near-surface ensembles of nitrogen-vacancy centers, in stark contrast to plasma treatments which use heated sample stages., Comment: 24 pages, 4 figures

The proliferation of generative models, combined with pretraining on web-scale data, raises a timely question: what happens when these models are trained on their own generated outputs? Recent investigations into model-data feedback loops proposed that such loops would lead to a phenomenon termed model collapse, under which performance progressively degrades with each model-data feedback iteration until fitted models become useless. However, those studies largely assumed that new data replace old data over time, where an arguably more realistic assumption is that data accumulate over time. In this paper, we ask: what effect does accumulating data have on model collapse? We empirically study this question by pretraining sequences of language models on text corpora. We confirm that replacing the original real data by each generation's synthetic data does indeed tend towards model collapse, then demonstrate that accumulating the successive generations of synthetic data alongside the original real data avoids model collapse; these results hold across a range of model sizes, architectures, and hyperparameters. We obtain similar results for deep generative models on other types of real data: diffusion models for molecule conformation generation and variational autoencoders for image generation. To understand why accumulating data can avoid model collapse, we use an analytically tractable framework introduced by prior work in which a sequence of linear models are fit to the previous models' outputs. Previous work used this framework to show that if data are replaced, the test error increases with the number of model-fitting iterations; we extend this argument to prove that if data instead accumulate, the test error has a finite upper bound independent of the number of iterations, meaning model collapse no longer occurs.

We empirically study a simple layer-pruning strategy for popular families of open-weight pretrained LLMs, finding minimal degradation of performance on different question-answering benchmarks until after a large fraction (up to half) of the layers are removed. To prune these models, we identify the optimal block of layers to prune by considering similarity across layers; then, to "heal" the damage, we perform a small amount of finetuning. In particular, we use parameter-efficient finetuning (PEFT) methods, specifically quantization and Low Rank Adapters (QLoRA), such that each of our experiments can be performed on a single A100 GPU. From a practical perspective, these results suggest that layer pruning methods can complement other PEFT strategies to further reduce computational resources of finetuning on the one hand, and can improve the memory and latency of inference on the other hand. From a scientific perspective, the robustness of these LLMs to the deletion of layers implies either that current pretraining methods are not properly leveraging the parameters in the deeper layers of the network or that the shallow layers play a critical role in storing knowledge., Comment: 12 + 10 pages, 5 + 4 figures

Nearly a hundred progenitor-less, thin stellar streams have been discovered in the Milky Way, thanks to Gaia and related surveys. Most streams are believed to have formed from star clusters and it was recently proposed that extended star clusters -- rich in stellar-mass black holes (BHs) -- are efficient in creating streams. To understand the nature of stream progenitors better, we quantify the differences between streams originating from star clusters with and without BHs using direct $N$-body models and a new model for the density profiles of streams based on time-dependent escape rates from clusters. The QSG (Quantifying Stream Growth) model facilitates the rapid exploration of parameter space and provides an analytic framework to understand the impact of different star cluster properties and escape conditions on the structure of streams. Using these models it is found that, compared to streams from BH-free clusters on the same orbit, streams of BH-rich clusters: (1) are approximately five times more massive; (2) have a peak density three times closer to the cluster 1 Gyr post-dissolution (for orbits of Galactocentric radius > 10 kpc), and (3) have narrower peaks and more extended wings in their density profile. We discuss other observable stream properties that are affected by the presence of BHs in their progenitor cluster, namely the width of the stream, its radial offset from the orbit, and the properties of the gap at the progenitor's location. Our results provide a step towards using stellar streams to constrain the BH content of dissolved (globular) star clusters., Comment: 13 pages, 8 figures, accepted for MNRAS

A boron-oxide termination of the diamond (100) surface has been formed by depositing molecular boron oxide $\rm{B_2O_3}$ onto the hydrogen-terminated (100) diamond surface under ultrahigh vacuum conditions and annealing to $\rm{950^{\circ} C}$. The resulting termination was highly oriented and chemically homogeneous, although further optimisation is required to increase the surface coverage beyond the 0.4 ML achieved here. This work demonstrates the possibility of using molecular deposition under ultrahigh vacuum conditions for complex surface engineering of the diamond surface, and may be a first step in an alternative approach to fabricating boron doped delta layers in diamond., Comment: Main text 10 pages, 4 figures. Supp Info 3 pages

Fully-connected deep neural networks with weights initialized from independent Gaussian distributions can be tuned to criticality, which prevents the exponential growth or decay of signals propagating through the network. However, such networks still exhibit fluctuations that grow linearly with the depth of the network, which may impair the training of networks with width comparable to depth. We show analytically that rectangular networks with tanh activations and weights initialized from the ensemble of orthogonal matrices have corresponding preactivation fluctuations which are independent of depth, to leading order in inverse width. Moreover, we demonstrate numerically that, at initialization, all correlators involving the neural tangent kernel (NTK) and its descendants at leading order in inverse width -- which govern the evolution of observables during training -- saturate at a depth of $\sim 20$, rather than growing without bound as in the case of Gaussian initializations. We speculate that this structure preserves finite-width feature learning while reducing overall noise, thus improving both generalization and training speed in deep networks with depth comparable to width. We provide some experimental justification by relating empirical measurements of the NTK to the superior performance of deep nonlinear orthogonal networks trained under full-batch gradient descent on the MNIST and CIFAR-10 classification tasks., Comment: v2: numerical experiments updated with more data, plots updated to match, conclusions unchanged. 30+12 pages, 20 figures

Large language models with a huge number of parameters, when trained on near internet-sized number of tokens, have been empirically shown to obey neural scaling laws: specifically, their performance behaves predictably as a power law in either parameters or dataset size until bottlenecked by the other resource. To understand this better, we first identify the necessary properties allowing such scaling laws to arise and then propose a statistical model -- a joint generative data model and random feature model -- that captures this neural scaling phenomenology. By solving this model in the dual limit of large training set size and large number of parameters, we gain insight into (i) the statistical structure of datasets and tasks that lead to scaling laws, (ii) the way nonlinear feature maps, such as those provided by neural networks, enable scaling laws when trained on these datasets, (iii) the optimality of the equiparameterization scaling of training sets and parameters, and (iv) whether such scaling laws can break down and how they behave when they do. Key findings are the manner in which the power laws that occur in the statistics of natural datasets are extended by nonlinear random feature maps and then translated into power-law scalings of the test loss and how the finite extent of the data's spectral power law causes the model's performance to plateau., Comment: 73 + 23 pages, 14 + 5 figures

This book develops an effective theory approach to understanding deep neural networks of practical relevance. Beginning from a first-principles component-level picture of networks, we explain how to determine an accurate description of the output of trained networks by solving layer-to-layer iteration equations and nonlinear learning dynamics. A main result is that the predictions of networks are described by nearly-Gaussian distributions, with the depth-to-width aspect ratio of the network controlling the deviations from the infinite-width Gaussian description. We explain how these effectively-deep networks learn nontrivial representations from training and more broadly analyze the mechanism of representation learning for nonlinear models. From a nearly-kernel-methods perspective, we find that the dependence of such models' predictions on the underlying learning algorithm can be expressed in a simple and universal way. To obtain these results, we develop the notion of representation group flow (RG flow) to characterize the propagation of signals through the network. By tuning networks to criticality, we give a practical solution to the exploding and vanishing gradient problem. We further explain how RG flow leads to near-universal behavior and lets us categorize networks built from different activation functions into universality classes. Altogether, we show that the depth-to-width ratio governs the effective model complexity of the ensemble of trained networks. By using information-theoretic techniques, we estimate the optimal aspect ratio at which we expect the network to be practically most useful and show how residual connections can be used to push this scale to arbitrary depths. With these tools, we can learn in detail about the inductive bias of architectures, hyperparameters, and optimizers., Comment: 471 pages, to be published by Cambridge University Press; v2: hyperlinks fixed, index added

We derive a simple and model-independent formula for the change in the generalization gap due to a gradient descent update. We then compare the change in the test error for stochastic gradient descent to the change in test error from an equivalent number of gradient descent updates and show explicitly that stochastic gradient descent acts to regularize generalization error by decorrelating nearby updates. These calculations depends on the details of the model only through the mean and covariance of the gradient distribution, which may be readily measured for particular models of interest. We discuss further improvements to these calculations and comment on possible implications for stochastic optimization., Comment: First appeared at the "Workshop on Integration of Deep Learning Theories" at NeurIPS in 2018 and has been available since then at https://research.fb.com/publications/sgd-implicitly-regularizes-generalization-error/

We discuss why AI is hard and why physics is simple. We discuss how physical intuition and the approach of theoretical physics can be brought to bear on the field of artificial intelligence and specifically machine learning. We suggest that the underlying project of machine learning and the underlying project of physics are strongly coupled through the principle of sparsity, and we call upon theoretical physicists to work on AI as physicists. As a first step in that direction, we discuss an upcoming book on the principles of deep learning theory that attempts to realize this approach., Comment: written for a special issue of Machine Learning: Science and Technology as an invited perspective piece

Autoencoders have been proposed as a powerful tool for model-independent anomaly detection in high-energy physics. The operating principle is that events which do not belong to the space of training data will be reconstructed poorly, thus flagging them as anomalies. We point out that in a variety of examples of interest, the connection between large reconstruction error and anomalies is not so clear. In particular, for data sets with nontrivial topology, there will always be points that erroneously seem anomalous due to global issues. Conversely, neural networks typically have an inductive bias or prior to locally interpolate such that undersampled or rare events may be reconstructed with small error, despite actually being the desired anomalies. Taken together, these facts are in tension with the simple picture of the autoencoder as an anomaly detector. Using a series of illustrative low-dimensional examples, we show explicitly how the intrinsic and extrinsic topology of the dataset affects the behavior of an autoencoder and how this topology is manifested in the latent space representation during training. We ground this analysis in the discussion of a mock "bump hunt" in which the autoencoder fails to identify an anomalous "signal" for reasons tied to the intrinsic topology of $n$-particle phase space., Comment: 24 + 20 pages, 26 figures; no autoencoders were harmed in the making of this project. v2: JHEP published version

Design of reliable systems must guarantee stability against input perturbations. In machine learning, such guarantee entails preventing overfitting and ensuring robustness of models against corruption of input data. In order to maximize stability, we analyze and develop a computationally efficient implementation of Jacobian regularization that increases classification margins of neural networks. The stabilizing effect of the Jacobian regularizer leads to significant improvements in robustness, as measured against both random and adversarial input perturbations, without severely degrading generalization properties on clean data., Comment: 21 pages, 10 figures

We show that in a variety of large-scale deep learning scenarios the gradient dynamically converges to a very small subspace after a short period of training. The subspace is spanned by a few top eigenvectors of the Hessian (equal to the number of classes in the dataset), and is mostly preserved over long periods of training. A simple argument then suggests that gradient descent may happen mostly in this subspace. We give an example of this effect in a solvable model of classification, and we comment on possible implications for optimization and learning., Comment: 9 pages + appendices, 12 figures

We discuss the probability distribution for the "size" of a time-evolving operator in the SYK model. Scrambling is related to the fact that as time passes, the distribution shifts towards larger operators. Initially, the rate is exponential and determined by the infinite-temperature chaos exponent. We evaluate the size distribution numerically for $N = 30$, and show how to compute it in the large-$N$ theory using the dressed fermion propagator. We then evaluate the distribution explicitly at leading nontrivial order in the large-$q$ expansion., Comment: 18 pages, 2 official figures, many unofficial figures

We study the relationship between quantum chaos and pseudorandomness by developing probes of unitary design. A natural probe of randomness is the "frame potential," which is minimized by unitary $k$-designs and measures the $2$-norm distance between the Haar random unitary ensemble and another ensemble. A natural probe of quantum chaos is out-of-time-order (OTO) four-point correlation functions. We show that the norm squared of a generalization of out-of-time-order $2k$-point correlators is proportional to the $k$th frame potential, providing a quantitative connection between chaos and pseudorandomness. Additionally, we prove that these $2k$-point correlators for Pauli operators completely determine the $k$-fold channel of an ensemble of unitary operators. Finally, we use a counting argument to obtain a lower bound on the quantum circuit complexity in terms of the frame potential. This provides a direct link between chaos, complexity, and randomness., Comment: 46+many pages, and all the figures too. v2: the director's cut -- more jokes, less typos

As experiments are increasingly able to probe the quantum dynamics of systems with many degrees of freedom, it is interesting to probe fundamental bounds on the dynamics of quantum information. We elaborate on the relationship between one such bound---the Lieb-Robinson bound---and the butterfly effect in strongly-coupled quantum systems. The butterfly effect implies the ballistic growth of local operators in time, which can be quantified with the "butterfly" velocity $v_B$. Similarly, the Lieb-Robinson velocity places a state independent ballistic upper bound on the size of time evolved operators in non-relativistic lattice models. Here, we argue that $v_B$ is a state-dependent effective Lieb-Robinson velocity. We study the butterfly velocity in a wide variety of quantum field theories using holography and compare with free particle computations to understand the role of strong coupling. We find that, depending on the way length and time scale, $v_B$ acquires a temperature dependence and decreases towards the IR. We also comment on experimental prospects and on the relationship between the butterfly velocity and signaling., Comment: 5+5 pages, 0 figures. v2: updated references and additional clarifications