Search

Models that rely on subword tokenization have significant drawbacks, such as sensitivity to character-level noise like spelling errors and inconsistent compression rates across different languages and scripts. While character- or byte-level models like ByT5 attempt to address these concerns, they have not gained widespread adoption -- processing raw byte streams without tokenization results in significantly longer sequence lengths, making training and inference inefficient. This work introduces MrT5 (MergeT5), a more efficient variant of ByT5 that integrates a token deletion mechanism in its encoder to dynamically shorten the input sequence length. After processing through a fixed number of encoder layers, a learnt delete gate determines which tokens are to be removed and which are to be retained for subsequent layers. MrT5 effectively ``merges'' critical information from deleted tokens into a more compact sequence, leveraging contextual information from the remaining tokens. In continued pre-training experiments, we find that MrT5 can achieve significant gains in inference runtime with minimal effect on performance. When trained on English text, MrT5 demonstrates the capability to transfer its deletion feature zero-shot across several languages, with significant additional improvements following multilingual training. Furthermore, MrT5 shows comparable accuracy to ByT5 on downstream evaluations such as XNLI and character-level tasks while reducing sequence lengths by up to 80%. Our approach presents a solution to the practical limitations of existing byte-level models.

Query Autocomplete (QAC) is a critical feature in modern search engines, facilitating user interaction by predicting search queries based on input prefixes. Despite its widespread adoption, the absence of large-scale, realistic datasets has hindered advancements in QAC system development. This paper addresses this gap by introducing AmazonQAC, a new QAC dataset sourced from Amazon Search logs, comprising 395M samples. The dataset includes actual sequences of user-typed prefixes leading to final search terms, as well as session IDs and timestamps that support modeling the context-dependent aspects of QAC. We assess Prefix Trees, semantic retrieval, and Large Language Models (LLMs) with and without finetuning. We find that finetuned LLMs perform best, particularly when incorporating contextual information. However, even our best system achieves only half of what we calculate is theoretically possible on our test data, which implies QAC is a challenging problem that is far from solved with existing systems. This contribution aims to stimulate further research on QAC systems to better serve user needs in diverse environments. We open-source this data on Hugging Face at https://huggingface.co/datasets/amazon/AmazonQAC., Comment: EMNLP 2024

In-context learning (ICL) is a powerful technique for getting language models to perform complex tasks with no training updates. Prior work has established strong correlations between the number of in-context examples provided and the accuracy of the model's predictions. In this paper, we seek to explain this correlation by showing that ICL approximates a Bayesian learner. This perspective gives rise to a family of novel Bayesian scaling laws for ICL. In experiments with \mbox{GPT-2} models of different sizes, our scaling laws exceed or match existing scaling laws in accuracy while also offering interpretable terms for task priors, learning efficiency, and per-example probabilities. To illustrate the analytic power that such interpretable scaling laws provide, we report on controlled synthetic dataset experiments designed to inform real-world studies of safety alignment. In our experimental protocol, we use SFT to suppress an unwanted existing model capability and then use ICL to try to bring that capability back (many-shot jailbreaking). We then experiment on real-world instruction-tuned LLMs using capabilities benchmarks as well as a new many-shot jailbreaking dataset. In all cases, Bayesian scaling laws accurately predict the conditions under which ICL will cause the suppressed behavior to reemerge, which sheds light on the ineffectiveness of post-training at increasing LLM safety., Comment: 10 pages main text, 26 pages total

Semiconductor double quantum dots (DQD) coupled to superconducting microwave resonators offer a promising platform for the detection of single microwave photons. In previous works, the photodetection was studied for a monochromatic source of microwave photons. Here, we theoretically analyze the photodetection of single microwave pulses. The photodetection in this case can be seen as a non-linear filtering process of an incoming signal, the pulse, to an outgoing one, the photocurrent. This analogy to signal processing motivated the derivation of a Wigner-function formalism which provides a compelling visualization of the time and frequency properties of the photodetector for low intensities. We find a trade-off between detecting the time and the frequency of the incoming photons in agreement with the time-energy uncertainty relation. As the intensity of the source increases, the photodetection is influenced by coherent Rabi oscillations of the DQD. Our findings give insight into the time-dependent properties of microwave photons interacting with electrons in a DQD-resonator hybrid system and provide guidance for experiments on single microwave pulse detection., Comment: 19 pages, 9 figures

The rapid introduction of new brand names into everyday language poses a unique challenge for e-commerce spelling correction services, which must distinguish genuine misspellings from novel brand names that use unconventional spelling. We seek to address this challenge via Retrieval Augmented Generation (RAG). On this approach, product names are retrieved from a catalog and incorporated into the context used by a large language model (LLM) that has been fine-tuned to do contextual spelling correction. Through quantitative evaluation and qualitative error analyses, we find improvements in spelling correction utilizing the RAG framework beyond a stand-alone LLM. We also demonstrate the value of additional finetuning of the LLM to incorporate retrieved context.

Quantum acoustics is an emerging platform for hybrid quantum technologies enabling quantum coherent control of mechanical vibrations. High-overtone bulk acoustic resonators (HBARs) represent an attractive mechanical implementation of quantum acoustics due to their potential for exceptionally high mechanical coherence. Here, we demonstrate an implementation of high-coherence HBAR quantum acoustics integrated with a planar superconducting qubit architecture, demonstrating an acoustically-induced-transparency regime of high cooperativity and weak coupling, analogous to the electrically-induced transparency in atomic physics. Demonstrating high-coherence quantum acoustics with planar superconducting devices enables new applications for acoustic resonators in quantum technologies., Comment: 11 pages, 7 figures, 1 table

Aggregation-diffusion equations are foundational tools for modelling biological aggregations. Their principal use is to link the collective movement mechanisms of organisms to their emergent space use patterns in a rigorous, non-speculative way. However, most existing studies implicitly assume that organism movement is not affected by the underlying environment. In reality, the environment is a key determinant of emergent space use patterns, albeit in combination with collective aspects of motion. This work studies aggregation-diffusion equations in a heterogeneous environment in one spatial dimension. Under certain assumptions, it is possible to find exact analytic expressions for the steady-state solutions to the equation when diffusion is quadratic. Minimising the associated energy functional across these solutions provides a rapid way of determining the likely emergent space use pattern, which can be verified via numerics. This energy-minimisation procedure is applied to a simple test case, where the environment consists of a single clump of attractive resources. Here, self-attraction and resource-attraction combine to shape the emergent aggregation. Two counter-intuitive results emerge from the analytic results: (a) a non-monotonic dependence of clump width on the aggregation width, (b) a positive correlation between self-attraction strength and aggregation width when the resource attraction is strong. These are verified through numerical simulations. Overall, the study shows rigorously how environment and collective behaviour combine to shape organism space use, sometimes in counter-intuitive ways.

Quality pretraining data is often seen as the key to high-performance language models. However, progress in understanding pretraining data has been slow due to the costly pretraining runs required for data selection experiments. We present a framework that avoids these costs and selects high-quality pretraining data without any LLM training of our own. Our work is based on a simple observation: LLM losses on many pretraining texts are correlated with downstream benchmark performance, and selecting high-correlation documents is an effective pretraining data selection method. We build a new statistical framework for data selection centered around estimates of perplexity-benchmark correlations and perform data selection using a sample of 90 LLMs taken from the Open LLM Leaderboard on texts from tens of thousands of web domains. In controlled pretraining experiments at the 160M parameter scale on 8 benchmarks, our approach outperforms DSIR on every benchmark, while matching the best data selector found in DataComp-LM, a hand-engineered bigram classifier.

Periodic laser driving, known as Floquet engineering, is a powerful tool to manipulate the properties of quantum materials. Using circularly polarized light, artificial magnetic fields, called Berry curvature, can be created in the photon-dressed Floquet-Bloch states that form. This mechanism, when applied to 3D Dirac and Weyl systems, is predicted to lead to photon-dressed movement of Weyl nodes which should be detectable in the transport sector. The transport response of such a topological light-matter hybrid, however, remains experimentally unknown. Here, we report on the transport properties of the type-II Weyl semimetal T$\mathrm{_d}$-MoTe$_\mathrm{2}$ illuminated by a femtosecond pulse of circularly polarized light. Using an ultrafast optoelectronic device architecture, we observed injection currents and a helicity-dependent anomalous Hall effect whose scaling with laser field strongly deviate from the perturbative laws of nonlinear optics. We show using Floquet theory that this discovery corresponds to the formation of a magnetic Floquet-Weyl semimetal state. Numerical ab initio simulations support this interpretation, indicating that the light-induced motion of the Weyl nodes contributes substantially to the measured transport signals. This work demonstrates the ability to generate large effective magnetic fields ($>$ 30T) with light, which can be used to manipulate the magnetic and topological properties of a range of quantum materials.

The Linear Representation Hypothesis (LRH) states that neural networks learn to encode concepts as directions in activation space, and a strong version of the LRH states that models learn only such encodings. In this paper, we present a counterexample to this strong LRH: when trained to repeat an input token sequence, gated recurrent neural networks (RNNs) learn to represent the token at each position with a particular order of magnitude, rather than a direction. These representations have layered features that are impossible to locate in distinct linear subspaces. To show this, we train interventions to predict and manipulate tokens by learning the scaling factor corresponding to each sequence position. These interventions indicate that the smallest RNNs find only this magnitude-based solution, while larger RNNs have linear representations. These findings strongly indicate that interpretability research should not be confined by the LRH.

Large Language Models (LLMs) are often aligned using contrastive alignment objectives and preference pair datasets. The interaction between model, paired data, and objective makes alignment a complicated procedure, sometimes producing subpar results. We study this and find that (i) preference data gives a better learning signal when the underlying responses are contrastive, and (ii) alignment objectives lead to better performance when they specify more control over the model during training. Based on these insights, we introduce Contrastive Learning from AI Revisions (CLAIR), a data-creation method which leads to more contrastive preference pairs, and Anchored Preference Optimization (APO), a controllable and more stable alignment objective. We align Llama-3-8B-Instruct using various comparable datasets and alignment objectives and measure MixEval-Hard scores, which correlate highly with human judgments. The CLAIR preferences lead to the strongest performance out of all datasets, and APO consistently outperforms less controllable objectives. Our best model, trained on 32K CLAIR preferences with APO, improves Llama-3-8B-Instruct by 7.65%, closing the gap with GPT4-turbo by 45%. Our code is available at https://github.com/ContextualAI/CLAIR_and_APO.

Steady-state quantum thermal machines are typically characterized by a continuous flow of heat between different reservoirs. However, at the level of discrete stochastic realizations, heat flow is unraveled as a series of abrupt quantum jumps, each representing an exchange of finite quanta with the environment. In this work, we present a framework that resolves the dynamics of quantum thermal machines into cycles that are classified as engine-like, cooling-like, or idle. We explore the statistics of each cycle type and its duration, enabling us to determine both the fraction of cycles useful for thermodynamic tasks and the average waiting time between cycles of the same type. Our framework presents a novel approach in characterizing thermal machines, with significant relevance to modern experiments such as mesoscopic transport using quantum dots., Comment: 6 + 12 pages; 3 + 2 figures

Large Language Models (LLMs) frequently memorize long sequences verbatim, often with serious legal and privacy implications. Much prior work has studied such verbatim memorization using observational data. To complement such work, we develop a framework to study verbatim memorization in a controlled setting by continuing pre-training from Pythia checkpoints with injected sequences. We find that (1) non-trivial amounts of repetition are necessary for verbatim memorization to happen; (2) later (and presumably better) checkpoints are more likely to verbatim memorize sequences, even for out-of-distribution sequences; (3) the generation of memorized sequences is triggered by distributed model states that encode high-level features and makes important use of general language modeling capabilities. Guided by these insights, we develop stress tests to evaluate unlearning methods and find they often fail to remove the verbatim memorized information, while also degrading the LM. Overall, these findings challenge the hypothesis that verbatim memorization stems from specific model weights or mechanisms. Rather, verbatim memorization is intertwined with the LM's general capabilities and thus will be very difficult to isolate and suppress without degrading model quality.

The fast reconstruction of a bandlimited function from its sample data is an essential problem in signal processing. In this paper, we consider the widely used Gaussian regularized Shannon sampling formula in comparison to regularized Shannon sampling formulas employing alternative window functions, including the modified Gaussian function, the sinh-type window function, and the continuous Kaiser-Bessel window function. It is shown that the approximation errors of these regularized Shannon sampling formulas possess an exponential decay with respect to the truncation parameter. The main focus of this paper is to identify the optimal variance of the (modified) Gaussian function as well as the optimal shape parameters of the sinh-type window function and the continuous Kaiser-Bessel window function, with the aim of achieving the fastest exponential decay of the approximation error. In doing so, we demonstrate that the decay rate of the sinh-type regularized Shannon sampling formula is considerably superior to that of the Gaussian regularized Shannon sampling formula. Additionally, numerical experiments illustrate the theoretical results.

The Kinetic Uncertainty Relation (KUR) bounds the signal-to-noise ratio of stochastic currents in terms of the number of transitions per unit time, known as the dynamical activity. This bound was derived in a classical context, and can be violated in the quantum regime due to coherent effects. However, the precise connection between KUR violations and quantum coherence has so far remained elusive, despite significant investigation. In this work, we solve this problem by deriving a modified bound that exactly pinpoints how, and when, coherence might lead to KUR violations. Our bound is sensitive to the specific kind of unraveling of the quantum master equation. It therefore allows one to compare quantum jumps and quantum diffusion, and understand, in each case, how quantum coherence affects fluctuations. We illustrate our result on a double quantum dot, where the electron current is monitored either by electron jump detection or with continuous diffusive charge measurement.

Predictive estimation, which comprises model calibration, model prediction, and validation, is a common objective when performing inverse uncertainty quantification (UQ) in diverse scientific applications. These techniques typically require thousands to millions of realisations of the forward model, leading to high computational costs. Surrogate models are often used to approximate these simulations. However, many surrogate models suffer from the fundamental limitation of being unable to estimate plausible high-dimensional outputs, inevitably compromising their use in the UQ framework. To address this challenge, this study introduces an efficient surrogate modelling workflow tailored for high-dimensional outputs. Specifically, a two-step approach is developed: (1) a dimensionality reduction technique is used for extracting data features and mapping the original output space into a reduced space; and (2) a multivariate surrogate model is constructed directly on the reduced space. The combined approach is shown to improve the accuracy of the surrogate model while retaining the computational efficiency required for UQ inversion. The proposed surrogate method, combined with Bayesian inference, is evaluated for a civil engineering application by performing inverse analyses on a laterally loaded pile problem. The results demonstrate the superiority of the proposed framework over traditional surrogate methods in dealing with high-dimensional outputs for sequential inversion analysis., Comment: 19 pages, 10 figures

Natural Language Processing (NLP) systems are increasingly taking the form of sophisticated modular pipelines, e.g., Retrieval Augmented Generation (RAG), where each module may involve a distinct Language Model (LM) and an associated prompt template. These compound systems often lack intermediate labels or gradient flow to optimize each module, making their end-to-end optimization challenging. Here we seek strategies to optimize both the module-level LM weights and the associated prompt templates of such systems to maximize a downstream task metric. We propose for the first time combining the weight and prompt optimization strategies to optimize a modular LM pipeline by alternating between the two to get the same LM to teach itself. In experiments with multi-hop QA, mathematical reasoning, and feature-based classification using mistral-7b, llama-2-7b, and llama-3-8b, these BetterTogether strategies optimizing the weights and prompts of a pipeline together outperform directly optimizing weights alone and prompts alone by up to 60% and 6%, respectively, on average across LMs and tasks. BetterTogether optimizer is released in DSPy at http://dspy.ai, Comment: EMNLP 2024

Parametric coupling between harmonic oscillators has enabled exquisite measurement precision and control of linear resonators, being extensively studied, for example, in cavity optomechanics. This level of control has been made possible by using strong sideband drives, enhancing the coupling rate while also linearizing the interaction. In this article, we demonstrate a new paradigm of parametrically coupled microwave circuits replacing one linear microwave cavity with a superconducting transmon qubit. Our system utilizes photon-pressure coupling between the transmon qubit and a highly linear microwave resonator, a microwave analog of the radiation-pressure interaction. Applying a strong sideband drive results in an on-demand, non-linear Jaynes-Cummings interaction with the linear resonator. We also observe a single-photon coupling rate an order of magnitude larger than all decay rates, placing the device in the single-photon strong coupling regime. This demonstration of photon-pressure Jaynes-Cummings interactions paves the way for developing novel photon-pressure quantum information processing hardware and will enable exotic tests of quantum gravity in the future by interfacing this new platform with mechanical resonators., Comment: 18 pages, 10 figures

Despite the evidence that supermassive black holes (SMBHs) co-evolve with their host galaxy, and that most of the growth of these SMBHs occurs via merger-free processes, the underlying mechanisms which drive this secular co-evolution are poorly understood. We investigate the role that both strong and weak large-scale galactic bars play in mediating this relationship. Using 72,940 disc galaxies in a volume-limited sample from Galaxy Zoo DESI, we analyse the active galactic nucleus (AGN) fraction in strongly barred, weakly barred, and unbarred galaxies up to z = 0.1 over a range of stellar masses and colours. After controlling for stellar mass and colour, we find that the optically selected AGN fraction is 31.6 +/- 0.9 per cent in strongly barred galaxies, 23.3 +/- 0.8 per cent in weakly barred galaxies, and 14.2 +/- 0.6 per cent in unbarred disc galaxies. These are highly statistically robust results, strengthening the tantalising results in earlier works. Strongly barred galaxies have a higher fraction of AGNs than weakly barred galaxies, which in turn have a higher fraction than unbarred galaxies. Thus, while bars are not required in order to grow a SMBH in a disc galaxy, large-scale galactic bars appear to facilitate AGN fuelling, and the presence of a strong bar makes a disc galaxy more than twice as likely to host an AGN than an unbarred galaxy at all galaxy stellar masses and colours., Comment: 11 pages, 8 figures, accepted for publication in MNRAS

Time estimation is a fundamental task that underpins precision measurement, global navigation systems, financial markets, and the organisation of everyday life. Many biological processes also depend on time estimation by nanoscale clocks, whose performance can be significantly impacted by random fluctuations. In this work, we formulate the problem of optimal time estimation for Markovian stochastic processes, and present its general solution in the asymptotic (long-time) limit. Specifically, we obtain a tight upper bound on the precision of any time estimate constructed from sustained observations of a classical, Markovian jump process. This bound is controlled by the mean residual time, i.e. the expected wait before the first jump is observed. As a consequence, we obtain a universal bound on the signal-to-noise ratio of arbitrary currents and counting observables in the steady state. This bound is similar in spirit to the kinetic uncertainty relation but provably tighter, and we explicitly construct the counting observables that saturate it. Our results establish ultimate precision limits for an important class of observables in non-equilibrium systems, and demonstrate that the mean residual time, not the dynamical activity, is the measure of freneticity that tightly constrains fluctuations far from equilibrium., Comment: 13+16 pages, 9 figures. Comments welcome!

We report an estimation of the injected mass composition of ultra-high energy cosmic rays (UHECRs) at energies higher than 10 EeV. The composition is inferred from an energy-dependent sky distribution of UHECR events observed by the Telescope Array surface detector by comparing it to the Large Scale Structure of the local Universe. In the case of negligible extra-galactic magnetic fields the results are consistent with a relatively heavy injected composition at E ~ 10 EeV that becomes lighter up to E ~ 100 EeV, while the composition at E > 100 EeV is very heavy. The latter is true even in the presence of highest experimentally allowed extra-galactic magnetic fields, while the composition at lower energies can be light if a strong EGMF is present. The effect of the uncertainty in the galactic magnetic field on these results is subdominant., Comment: 8 pages, 3 figures, accepted for publication in PRL

We use a new method to estimate the injected mass composition of ultrahigh cosmic rays (UHECRs) at energies higher than 10 EeV. The method is based on comparison of the energy-dependent distribution of cosmic ray arrival directions as measured by the Telescope Array experiment (TA) with that calculated in a given putative model of UHECR under the assumption that sources trace the large-scale structure (LSS) of the Universe. As we report in the companion letter, the TA data show large deflections with respect to the LSS which can be explained, assuming small extra-galactic magnetic fields (EGMF), by an intermediate composition changing to a heavy one (iron) in the highest energy bin. Here we show that these results are robust to uncertainties in UHECR injection spectra, the energy scale of the experiment and galactic magnetic fields (GMF). The assumption of weak EGMF, however, strongly affects this interpretation at all but the highest energies E > 100 EeV, where the remarkable isotropy of the data implies a heavy injected composition even in the case of strong EGMF. This result also holds if UHECR sources are as rare as $2 \times 10^{-5}$ Mpc$^{-3}$, that is the conservative lower limit for the source number density., Comment: 18 pages, 11 figures, accepted for publication in PRD

The theory of quantum thermodynamics investigates how the concepts of heat, work, and temperature can be carried over to the quantum realm, where fluctuations and randomness are fundamentally unavoidable. These lecture notes provide an introduction to the thermodynamics of small quantum systems. It is illustrated how the laws of thermodynamics emerge from quantum theory and how open quantum systems can be modeled by Markovian master equations. Quantum systems that are designed to perform a certain task, such as cooling or generating entanglement are considered. Finally, the effect of fluctuations on the thermodynamic description is discussed., Comment: Submission to SciPost Phys. Lect. Notes

Decentralised Autonomous Organisations (DAOs) are a new type of digital organisation that uses blockchain infrastructure (e.g. smart contracts, tokens) to coordinate a group of people around a shared mission. Like all organisations, DAOs must attract sources of funding and other resources, and discover and retain a talented community and workforce. To do this, they must signal their true quality. Yet the characteristics of the environment that DAOs operate in (pseudonymous actors, global scale, permissionless entry and exit) makes this difficult. We apply costly signalling theory to explore the information asymmetry problem in DAOs and some of the strategies (behaviours and investments) and institutional solutions (including better signalling mechanisms) that have evolved to solve this problem., Comment: European DAO Workshop 24

We investigate experimentally the quantum coherence of an electronic two-level system in a double quantum dot under continuous charge detection. The charge-state of the two-level system is monitored by a capacitively coupled single quantum dot detector that imposes a back-action effect to the system. The measured back-action is well described by an additional decoherence rate, approximately linearly proportional to the detector electron tunneling rate. We provide a model for the decoherence rate arising due to level detuning fluctuations induced by detector charge fluctuations. The theory predicts a factor of two lower decoherence rate than observed in the experiment, suggesting the need for a more elaborate theory accounting for additional sources of decoherence., Comment: 5 pages, 3 figures

We develop a method for training small-scale (under 100M parameter) neural information retrieval models with as few as 10 gold relevance labels. The method depends on generating synthetic queries for documents using a language model (LM), and the key step is that we automatically optimize the LM prompt that is used to generate these queries based on training quality. In experiments with the BIRCO benchmark, we find that models trained with our method outperform RankZephyr and are competitive with RankLLama, both of which are 7B parameter models trained on over 100K labels. These findings point to the power of automatic prompt optimization for synthetic dataset generation.

Language Model Programs, i.e. sophisticated pipelines of modular language model (LM) calls, are increasingly advancing NLP tasks, but they require crafting prompts that are jointly effective for all modules. We study prompt optimization for LM programs, i.e. how to update these prompts to maximize a downstream metric without access to module-level labels or gradients. To make this tractable, we factorize our problem into optimizing the free-form instructions and few-shot demonstrations of every module and introduce several strategies to craft task-grounded instructions and navigate credit assignment across modules. Our strategies include (i) program- and data-aware techniques for proposing effective instructions, (ii) a stochastic mini-batch evaluation function for learning a surrogate model of our objective, and (iii) a meta-optimization procedure in which we refine how LMs construct proposals over time. Using these insights we develop MIPRO, a novel algorithm for optimizing LM programs. MIPRO outperforms baseline optimizers on five of seven diverse multi-stage LM programs using a best-in-class open-source model (Llama-3-8B), by as high as 13% accuracy. We have released our new optimizers and benchmark in DSPy at http://dspy.ai, Comment: EMNLP 2024. Krista and Michael contributed equally to this work

We report on an observation of the difference between northern and southern skies of the ultrahigh energy cosmic ray energy spectrum with a significance of ${\sim}8\sigma$. We use measurements from the two largest experiments$\unicode{x2014}$the Telescope Array observing the northern hemisphere and the Pierre Auger Observatory viewing the southern hemisphere. Since the comparison of two measurements from different observatories introduces the issue of possible systematic differences between detectors and analyses, we validate the methodology of the comparison by examining the region of the sky where the apertures of the two observatories overlap. Although the spectra differ in this region, we find that there is only a $1.8\sigma$ difference between the spectrum measurements when anisotropic regions are removed and a fiducial cut in the aperture is applied., Comment: 8 pages, 6 figures

We developed a modular silicon photomultiplier camera to detect Earth-skimming PeV to EeV tau neutrinos with the imaging atmospheric Cherenkov technique. We built two cameras, a 256-pixel camera with S14161-6050HS SiPMs for the Trinity Demonstrator located on Frisco Peak, Utah, and a 512-pixel camera with S14521-6050AN SiPMs for the EUSO-SPB2 Cherenkov Telescope. The front-end electronics are based on the eMUSIC ASIC, and the camera signals are sampled and digitized with the 100MS/s and 12-bit AGET system. Both cameras are liquid-cooled. We detail the camera concept and the results from characterizing the SiPMs, bench testing, and calibrating the two cameras., Comment: Submitted to Nuclear Instruments and Methods in Physics Research A

Although CLIPScore is a powerful generic metric that captures the similarity between a text and an image, it fails to distinguish between a caption that is meant to complement the information in an image and a description that is meant to replace an image entirely, e.g., for accessibility. We address this shortcoming by updating the CLIP model with the Concadia dataset to assign higher scores to descriptions than captions using parameter efficient fine-tuning and a loss objective derived from work on causal interpretability. This model correlates with the judgements of blind and low-vision people while preserving transfer capabilities and has interpretable structure that sheds light on the caption--description distinction.

We use the construction of the relative bar resolution via differential graded structures to obtain the minimal graded free resolution of $\text{Der}_{R \mid k}$, where $R$ is a determinantal ring defined by the maximal minors of an $n \times (n+1)$ generic matrix and $k$ is its coefficient field. Along the way, we compute an explicit action of the Hilbert-Burch differential graded algebra on a differential graded module resolving the cokernel of the Jacobian matrix whose kernel is $\text{Der}_{R \mid k}$. As a consequence of the minimality of the resulting relative bar resolution, we get a minimal generating set for $\text{Der}_{R \mid k}$ as an $R$-module, which, while already known, has not been obtained via our methods.

We present a dimension-incremental method for function approximation in bounded orthonormal product bases to learn the solutions of various differential equations. Therefore, we deconstruct the source function of the differential equation into parameters like Fourier or Spline coefficients and treat the solution of the differential equation as a high-dimensional function w.r.t. the spatial variables, these parameters and also further possible parameters from the differential equation itself. Finally, we learn this function in the sense of sparse approximation in a suitable function space by detecting coefficients of the basis expansion with largest absolute value. Investigating the corresponding indices of the basis coefficients yields further insights on the structure of the solution as well as its dependency on the parameters and their interactions and allows for a reasonable generalization to even higher dimensions and therefore better resolutions of the deconstructed source function.

We present a superconducting cavity-coupled double quantum dot (DQD) photodiode that achieves a maximum photon-to-electron conversion efficiency of 25% in the microwave domain. With a higher-quality-factor cavity and improved device design to prevent photon leakages through unwanted pathways, our device measures microwave signals down to 100 aW power level and achieves sensitivity to probe microwave signals with one photon at a time in the cavity. We analyze the photodiode operation using Jaynes-Cummings input-output theory, identifying the key improvements of stronger cavity-DQD coupling needed to achieve near-unity photodetection efficiency. The results presented in this work represent a crucial advancement toward near unity microwave photodetection efficiency with single cavity-photon sensitivity for studies of photon statistics in the microwave range and applications related to quantum information processing., Comment: 8 pages, 4 figures

We study how sharp signatures of fractionalization emerge in nonlinear spectroscopy experiments on spin liquids with separated energy scales. Our model is that of dipolar-octupolar rare earth pyrochlore materials, prime candidates for realising quantum spin ice. This family of three dimensional quantum spin liquids exhibits fractionalization of spin degrees of freedom into spinons charged under an emergent $U(1)$ gauge field. We show that the technique of two dimensional coherent spectroscopy (2DCS) can identify clear signatures of fractionalised spinon dynamics in dipolar-octupolar quantum spin ices. However, at intermediate temperatures, spinon dynamics are heavily constrained in the presence of an incoherent spin background, leading to a broad 2DCS response. At lower temperatures, a sharp signal emerges as the system enters a coherent spin liquid state. This lower temperature signal can in turn distinguish between zero-flux and $\pi$-flux forms of quantum spin ice., Comment: 6 pages, 4 figures

Previous work on Universal Transformers (UTs) has demonstrated the importance of parameter sharing across layers. By allowing recurrence in depth, UTs have advantages over standard Transformers in learning compositional generalizations, but layer-sharing comes with a practical limitation of parameter-compute ratio: it drastically reduces the parameter count compared to the non-shared model with the same dimensionality. Naively scaling up the layer size to compensate for the loss of parameters makes its computational resource requirements prohibitive. In practice, no previous work has succeeded in proposing a shared-layer Transformer design that is competitive in parameter count-dominated tasks such as language modeling. Here we propose MoEUT (pronounced "moot"), an effective mixture-of-experts (MoE)-based shared-layer Transformer architecture, which combines several recent advances in MoEs for both feedforward and attention layers of standard Transformers together with novel layer-normalization and grouping schemes that are specific and crucial to UTs. The resulting UT model, for the first time, slightly outperforms standard Transformers on language modeling tasks such as BLiMP and PIQA, while using significantly less compute and memory., Comment: Accepted to NeurIPS 2024

In scenarios coined Maxwell's demon, information on microscopic degrees of freedom is used to seemingly violate the second law of thermodynamics. This has been studied in the classical as well as the quantum domain. In this paper, we study an implementation of Maxwell's demon that can operate in both domains. In particular, we investigate information-to-work conversion over the quantum-to-classical transition. The demon continuously measures the charge state of a double quantum dot, and uses this information to guide electrons against a voltage bias by tuning the on-site energies of the dots. Coherent tunneling between the dots allows for the buildup of quantum coherence in the system. Under strong measurements, the coherence is suppressed, and the system is well-described by a classical model. As the measurement strength is further increased, the Zeno effect prohibits interdot tunneling. A Zeno-like effect is also observed for weak measurements, where measurement errors lead to fluctuations in the on-site energies, dephasing the system. We anticipate similar behaviors in other quantum systems under continuous measurement and feedback control, making our results relevant for implementations in quantum technology and quantum control.

Quantum technologies and experiments often require preparing systems in low-temperature states. Here, we investigate cooling schemes using feedback protocols modeled with a Quantum Fokker-Planck Master Equation (QFPME) recently derived by Annby-Andersson et. al. (Phys. Rev. Lett. 129, 050401, 2022). This equation describes systems under continuous weak measurements, with feedback based on the outcome of these measurements. We apply this formalism to study the cooling and trapping of a harmonic oscillator for several protocols based on position and/or momentum measurements. We find that the protocols can cool the oscillator down to, or close to, the ground state for suitable choices of parameters. Our analysis provides an analytically solvable case study of quantum measurement and feedback and illustrates the application of the QFPME to continuous quantum systems., Comment: 18 pages, 3 figures

Parameter-efficient finetuning (PEFT) methods seek to adapt large neural models via updates to a small number of weights. However, much prior interpretability work has shown that representations encode rich semantic information, suggesting that editing representations might be a more powerful alternative. We pursue this hypothesis by developing a family of Representation Finetuning (ReFT) methods. ReFT methods operate on a frozen base model and learn task-specific interventions on hidden representations. We define a strong instance of the ReFT family, Low-rank Linear Subspace ReFT (LoReFT), and we identify an ablation of this method that trades some performance for increased efficiency. Both are drop-in replacements for existing PEFTs and learn interventions that are 15x--65x more parameter-efficient than LoRA. We showcase LoReFT on eight commonsense reasoning tasks, four arithmetic reasoning tasks, instruction-tuning, and GLUE. In all these evaluations, our ReFTs deliver the best balance of efficiency and performance, and almost always outperform state-of-the-art PEFTs. We release a generic ReFT training library publicly at https://github.com/stanfordnlp/pyreft., Comment: preprint

We propose two algorithms for boosting random Fourier feature models for approximating high-dimensional functions. These methods utilize the classical and generalized analysis of variance (ANOVA) decomposition to learn low-order functions, where there are few interactions between the variables. Our algorithms are able to find an index set of important input variables and variable interactions reliably. Furthermore, we generalize already existing random Fourier feature models to an ANOVA setting, where terms of different order can be used. Our algorithms have the advantage of interpretability, meaning that the influence of every input variable is known in the learned model, even for dependent input variables. We give theoretical as well as numerical results that our algorithms perform well for sensitivity analysis. The ANOVA-boosting step reduces the approximation error of existing methods significantly.

Scientific publishing lays the foundation of science by disseminating research findings, fostering collaboration, encouraging reproducibility, and ensuring that scientific knowledge is accessible, verifiable, and built upon over time. Recently, there has been immense speculation about how many people are using large language models (LLMs) like ChatGPT in their academic writing, and to what extent this tool might have an effect on global scientific practices. However, we lack a precise measure of the proportion of academic writing substantially modified or produced by LLMs. To address this gap, we conduct the first systematic, large-scale analysis across 950,965 papers published between January 2020 and February 2024 on the arXiv, bioRxiv, and Nature portfolio journals, using a population-level statistical framework to measure the prevalence of LLM-modified content over time. Our statistical estimation operates on the corpus level and is more robust than inference on individual instances. Our findings reveal a steady increase in LLM usage, with the largest and fastest growth observed in Computer Science papers (up to 17.5%). In comparison, Mathematics papers and the Nature portfolio showed the least LLM modification (up to 6.3%). Moreover, at an aggregate level, our analysis reveals that higher levels of LLM-modification are associated with papers whose first authors post preprints more frequently, papers in more crowded research areas, and papers of shorter lengths. Our findings suggests that LLMs are being broadly used in scientific writings.

Van der Waals (vdW) heterostructures host many-body quantum phenomena that can be tuned in situ using electrostatic gates. These gates are often microstructured graphite flakes that naturally form plasmonic cavities, confining light in discrete standing waves of current density due to their finite size. Their resonances typically lie in the GHz - THz range, corresponding to the same $\mu$eV - meV energy scale characteristic of many quantum effects in the materials they electrically control. This raises the possibility that built-in cavity modes could be relevant for shaping the low-energy physics of vdW heterostructures. However, capturing this light-matter interaction remains elusive as devices are significantly smaller than the diffraction limit at these wavelengths, hindering far-field spectroscopic tools. Here, we report on the sub-wavelength cavity electrodynamics of graphene embedded in a vdW heterostructure plasmonic microcavity. Using on-chip THz spectroscopy, we observed spectral weight transfer and an avoided crossing between the graphite cavity and graphene plasmon modes as the graphene carrier density was tuned, revealing their ultrastrong coupling. Our findings show that intrinsic cavity modes of metallic gates can sense and manipulate the low-energy electrodynamics of vdW heterostructures. This opens a pathway for deeper understanding of emergent phases in these materials and new functionality through cavity control.

Line detection is a classic and essential problem in image processing, computer vision and machine intelligence. Line detection has many important applications, including image vectorization (e.g., document recognition and art design), indoor mapping, and important societal challenges (e.g., sea ice fracture line extraction from satellite imagery). Many line detection algorithms and methods have been developed, but robust and intuitive methods are still lacking. In this paper, we proposed and implemented a topological graph-guided algorithm, named TGGLinesPlus, for line detection. Our experiments on images from a wide range of domains have demonstrated the flexibility of our TGGLinesPlus algorithm. We also benchmarked our algorithm with five classic and state-of-the-art line detection methods and the results demonstrate the robustness of TGGLinesPlus. We hope our open-source implementation of TGGLinesPlus will inspire and pave the way for many applications where spatial science matters., Comment: Our TGGLinesPlus Python implementation is open source. 27 pages, 8 figures and 4 tables

In cavity magnomechanical systems, magnetic excitations couple simultaneously with mechanical vibrations and microwaves, incorporating the tunability of magnetism and the long lifetimes of mechanical modes. Applications of such systems, such as thermometry and sensing, require precise measurement of the mechanical degree-of-freedom. In this paper, we propose a scheme for realizing backaction evading measurements of the mechanical vibrations in cavity magnomechanics. Our proposal involves driving the microwave cavity with two tones separated by twice the phonon frequency and with amplitudes satisfying a balance relation. We show that the minimum added imprecision noise is obtained for drives centered around the lower frequency magnon-microwave polaritons, which can beat the standard quantum limit at modest drive amplitudes. Our scheme is a simple and flexible way of engineering backaction evasion measurements that can be further generalized to other multimode systems., Comment: Revised version. 9 Pages, 8 Figures plus supplementary information (11 pages, 2 figures)

Interventions on model-internal states are fundamental operations in many areas of AI, including model editing, steering, robustness, and interpretability. To facilitate such research, we introduce $\textbf{pyvene}$, an open-source Python library that supports customizable interventions on a range of different PyTorch modules. $\textbf{pyvene}$ supports complex intervention schemes with an intuitive configuration format, and its interventions can be static or include trainable parameters. We show how $\textbf{pyvene}$ provides a unified and extensible framework for performing interventions on neural models and sharing the intervened upon models with others. We illustrate the power of the library via interpretability analyses using causal abstraction and knowledge localization. We publish our library through Python Package Index (PyPI) and provide code, documentation, and tutorials at https://github.com/stanfordnlp/pyvene., Comment: 8 pages, 3 figures