Search

AI models might use deceptive strategies as part of scheming or misaligned behaviour. Monitoring outputs alone is insufficient, since the AI might produce seemingly benign outputs while their internal reasoning is misaligned. We thus evaluate if linear probes can robustly detect deception by monitoring model activations. We test two probe-training datasets, one with contrasting instructions to be honest or deceptive (following Zou et al., 2023) and one of responses to simple roleplaying scenarios. We test whether these probes generalize to realistic settings where Llama-3.3-70B-Instruct behaves deceptively, such as concealing insider trading (Scheurer et al., 2023) and purposely underperforming on safety evaluations (Benton et al., 2024). We find that our probe distinguishes honest and deceptive responses with AUROCs between 0.96 and 0.999 on our evaluation datasets. If we set the decision threshold to have a 1% false positive rate on chat data not related to deception, our probe catches 95-99% of the deceptive responses. Overall we think white-box probes are promising for future monitoring systems, but current performance is insufficient as a robust defence against deception. Our probes' outputs can be viewed at data.apolloresearch.ai/dd and our code at github.com/ApolloResearch/deception-detection., Comment: Website: http://data.apolloresearch.ai/dd/ Code: http://www.github.com/ApolloResearch/deception-detection/

Mechanistic interpretability aims to understand the computational mechanisms underlying neural networks' capabilities in order to accomplish concrete scientific and engineering goals. Progress in this field thus promises to provide greater assurance over AI system behavior and shed light on exciting scientific questions about the nature of intelligence. Despite recent progress toward these goals, there are many open problems in the field that require solutions before many scientific and practical benefits can be realized: Our methods require both conceptual and practical improvements to reveal deeper insights; we must figure out how best to apply our methods in pursuit of specific goals; and the field must grapple with socio-technical challenges that influence and are influenced by our work. This forward-facing review discusses the current frontier of mechanistic interpretability and the open problems that the field may benefit from prioritizing.

Mechanistic interpretability aims to understand the internal mechanisms learned by neural networks. Despite recent progress toward this goal, it remains unclear how best to decompose neural network parameters into mechanistic components. We introduce Attribution-based Parameter Decomposition (APD), a method that directly decomposes a neural network's parameters into components that (i) are faithful to the parameters of the original network, (ii) require a minimal number of components to process any input, and (iii) are maximally simple. Our approach thus optimizes for a minimal length description of the network's mechanisms. We demonstrate APD's effectiveness by successfully identifying ground truth mechanisms in multiple toy experimental settings: Recovering features from superposition; separating compressed computations; and identifying cross-layer distributed representations. While challenges remain to scaling APD to non-toy models, our results suggest solutions to several open problems in mechanistic interpretability, including identifying minimal circuits in superposition, offering a conceptual foundation for 'features', and providing an architecture-agnostic framework for neural network decomposition.

Sensitive directions experiments attempt to understand the computational features of Language Models (LMs) by measuring how much the next token prediction probabilities change by perturbing activations along specific directions. We extend the sensitive directions work by introducing an improved baseline for perturbation directions. We demonstrate that KL divergence for Sparse Autoencoder (SAE) reconstruction errors are no longer pathologically high compared to the improved baseline. We also show that feature directions uncovered by SAEs have varying impacts on model outputs depending on the SAE's sparsity, with lower L0 SAE feature directions exerting a greater influence. Additionally, we find that end-to-end SAE features do not exhibit stronger effects on model outputs compared to traditional SAEs., Comment: Presented at the Attributing Model Behavior at Scale (ATTRIB) and Scientific Methods for Understanding Deep Learning (SciForDL) workshops at NeurIPS 2024

Sparse Autoencoders for transformer-based language models are typically defined independently per layer. In this work we analyze statistical relationships between features in adjacent layers to understand how features evolve through a forward pass. We provide a graph visualization interface for features and their most similar next-layer neighbors (https://stefanhex.com/spar-2024/feature-browser/), and build communities of related features across layers. We find that a considerable amount of features are passed through from a previous layer, some features can be expressed as quasi-boolean combinations of previous features, and some features become more specialized in later layers., Comment: Presented at the Attributing Model Behavior at Scale (ATTRIB) workshop at NeurIPS 2024

We identify stable regions in the residual stream of Transformers, where the model's output remains insensitive to small activation changes, but exhibits high sensitivity at region boundaries. These regions emerge during training and become more defined as training progresses or model size increases. The regions appear to be much larger than previously studied polytopes. Our analysis suggests that these stable regions align with semantic distinctions, where similar prompts cluster within regions, and activations from the same region lead to similar next token predictions. This work provides a promising research direction for understanding the complexity of neural networks, shedding light on training dynamics, and advancing interpretability., Comment: Presented at the Scientific Methods for Understanding Deep Learning (SciForDL) workshop at NeurIPS 2024

Sparse Auto-Encoders (SAEs) are commonly employed in mechanistic interpretability to decompose the residual stream into monosemantic SAE latents. Recent work demonstrates that perturbing a model's activations at an early layer results in a step-function-like change in the model's final layer activations. Furthermore, the model's sensitivity to this perturbation differs between model-generated (real) activations and random activations. In our study, we assess model sensitivity in order to compare real activations to synthetic activations composed of SAE latents. Our findings indicate that synthetic activations closely resemble real activations when we control for the sparsity and cosine similarity of the constituent SAE latents. This suggests that real activations cannot be explained by a simple "bag of SAE latents" lacking internal structure, and instead suggests that SAE latents possess significant geometric and statistical properties. Notably, we observe that our synthetic activations exhibit less pronounced activation plateaus compared to those typically surrounding real activations., Comment: Presented at the Attributing Model Behavior at Scale (ATTRIB) workshop at NeurIPS 2024

The LayerNorm (LN) layer in GPT-style transformer models has long been a hindrance to mechanistic interpretability. LN is a crucial component required to stabilize the training of large language models, and LN or the similar RMSNorm have been used in practically all large language models based on the transformer architecture. The non-linear nature of the LN layers is a hindrance for mechanistic interpretability as it hinders interpretation of the residual stream, and makes it difficult to decompose the model into circuits. Some researchers have gone so far as to name "reasons interpretability researchers hate layer norm." In this paper we show that it is possible to remove the LN layers from a pre-trained GPT2-small model by fine-tuning on a fraction (500M tokens) of the training data. We demonstrate that this LN-free model achieves similar performance to the original model on the OpenWebText and ThePile datasets (-0.05 cross-entropy loss), and the Hellaswag benchmark (-0.5% accuracy). We provide our implementation at https://github.com/ApolloResearch/gpt2_noLN, and fine-tuned GPT2-small models at https://huggingface.co/apollo-research/gpt2_noLN. Our work not only provides a simplified model for mechanistic interpretability research, but also provides evidence that the LN layers, at inference time, do not play a crucial role in transformer models., Comment: Presented at the Attributing Model Behavior at Scale (ATTRIB) and Interpretable AI: Past, Present, and Future workshops at NeurIPS 2024

Mechanistic interpretability aims to understand the behavior of neural networks by reverse-engineering their internal computations. However, current methods struggle to find clear interpretations of neural network activations because a decomposition of activations into computational features is missing. Individual neurons or model components do not cleanly correspond to distinct features or functions. We present a novel interpretability method that aims to overcome this limitation by transforming the activations of the network into a new basis - the Local Interaction Basis (LIB). LIB aims to identify computational features by removing irrelevant activations and interactions. Our method drops irrelevant activation directions and aligns the basis with the singular vectors of the Jacobian matrix between adjacent layers. It also scales features based on their importance for downstream computation, producing an interaction graph that shows all computationally-relevant features and interactions in a model. We evaluate the effectiveness of LIB on modular addition and CIFAR-10 models, finding that it identifies more computationally-relevant features that interact more sparsely, compared to principal component analysis. However, LIB does not yield substantial improvements in interpretability or interaction sparsity when applied to language models. We conclude that LIB is a promising theory-driven approach for analyzing neural networks, but in its current form is not applicable to large language models.

Mechanistic Interpretability aims to reverse engineer the algorithms implemented by neural networks by studying their weights and activations. An obstacle to reverse engineering neural networks is that many of the parameters inside a network are not involved in the computation being implemented by the network. These degenerate parameters may obfuscate internal structure. Singular learning theory teaches us that neural network parameterizations are biased towards being more degenerate, and parameterizations with more degeneracy are likely to generalize further. We identify 3 ways that network parameters can be degenerate: linear dependence between activations in a layer; linear dependence between gradients passed back to a layer; ReLUs which fire on the same subset of datapoints. We also present a heuristic argument that modular networks are likely to be more degenerate, and we develop a metric for identifying modules in a network that is based on this argument. We propose that if we can represent a neural network in a way that is invariant to reparameterizations that exploit the degeneracies, then this representation is likely to be more interpretable, and we provide some evidence that such a representation is likely to have sparser interactions. We introduce the Interaction Basis, a tractable technique to obtain a representation that is invariant to degeneracies from linear dependence of activations or Jacobians.

Activation patching is a popular mechanistic interpretability technique, but has many subtleties regarding how it is applied and how one may interpret the results. We provide a summary of advice and best practices, based on our experience using this technique in practice. We include an overview of the different ways to apply activation patching and a discussion on how to interpret the results. We focus on what evidence patching experiments provide about circuits, and on the choice of metric and associated pitfalls., Comment: A tutorial on activation patching. 13 pages, 2 figures

The cosmological 21 cm signal is one of the most promising avenues to study the Epoch of Reionization. One class of experiments aiming to detect this signal is global signal experiments measuring the sky-averaged 21 cm brightness temperature as a function of frequency. A crucial step in the interpretation and analysis of such measurements is separating foreground contributions from the remainder of the signal, requiring accurate models for both components. Current models for the signal (non-foreground) component, which may contain cosmological and systematic contributions, are incomplete and unable to capture the full signal. We propose two new methods for extracting this component from the data: Firstly, we employ a foreground-orthogonal Gaussian Process to extract the part of the signal that cannot be explained by the foregrounds. Secondly, we use a FlexKnot parameterization to model the full signal component in a free-form manner, not assuming any particular shape or functional form. This method uses Bayesian model selection to find the simplest signal that can explain the data. We test our methods on both, synthetic data and publicly available EDGES low-band data. We find that the Gaussian Process can clearly capture the foreground-orthogonal signal component of both data sets. The FlexKnot method correctly recovers the full shape of the input signal used in the synthetic data and yields a multi-modal distribution of different signal shapes that can explain the EDGES observations., Comment: 18 pages, 17 figures, accepted for publication in MNRAS

Through considerable effort and intuition, several recent works have reverse-engineered nontrivial behaviors of transformer models. This paper systematizes the mechanistic interpretability process they followed. First, researchers choose a metric and dataset that elicit the desired model behavior. Then, they apply activation patching to find which abstract neural network units are involved in the behavior. By varying the dataset, metric, and units under investigation, researchers can understand the functionality of each component. We automate one of the process' steps: to identify the circuit that implements the specified behavior in the model's computational graph. We propose several algorithms and reproduce previous interpretability results to validate them. For example, the ACDC algorithm rediscovered 5/5 of the component types in a circuit in GPT-2 Small that computes the Greater-Than operation. ACDC selected 68 of the 32,000 edges in GPT-2 Small, all of which were manually found by previous work. Our code is available at https://github.com/ArthurConmy/Automatic-Circuit-Discovery., Comment: NeurIPS 2023 Spotlight

The first billion years of cosmic history remains largely unobserved. We demonstrate, using a novel machine learning technique, how combining upper limits on the spatial fluctuations in the 21-cm signal with observations of the sky-averaged 21-cm signal from neutral hydrogen can improve our understanding of this epoch. By jointly analysing data from SARAS3 (redshift $z\approx15-25$) and limits from HERA ($z\approx8$ and $10$), we show that such a synergetic analysis provides tighter constraints on the astrophysics of galaxies 200 million years after the Big Bang than can be achieved with the individual data sets. Although our constraints are weak, this is the first time data from a sky-averaged 21-cm experiment and power spectrum experiment have been analysed together. In synergy, the two experiments leave only $64.9^{+0.3}_{-0.1}$% of the explored broad theoretical parameter space to be consistent with the joint data set, in comparison to $92.3^{+0.3}_{-0.1}$% for SARAS3 and $79.0^{+0.5}_{-0.2}$% for HERA alone. We use the joint analysis to constrain star formation efficiency, minimum halo mass for star formation, X-ray luminosity of early emitters and the radio luminosity of early galaxies. The joint analysis disfavours at 68% confidence a combination of galaxies with X-ray emission that is $\lesssim 33$ and radio emission that is $\gtrsim 32$ times as efficient as present day galaxies. We disfavour at $95$% confidence scenarios in which power spectra are $\geq126$ mK$^{2}$ at $z=25$ and the sky-averaged signals are $\leq-277$ mK., Comment: Accepted for MNRAS

We report the most sensitive upper limits to date on the 21 cm epoch of reionization power spectrum using 94 nights of observing with Phase I of the Hydrogen Epoch of Reionization Array (HERA). Using similar analysis techniques as in previously reported limits (HERA Collaboration 2022a), we find at 95% confidence that $\Delta^2(k = 0.34$ $h$ Mpc$^{-1}$) $\leq 457$ mK$^2$ at $z = 7.9$ and that $\Delta^2 (k = 0.36$ $h$ Mpc$^{-1}) \leq 3,496$ mK$^2$ at $z = 10.4$, an improvement by a factor of 2.1 and 2.6 respectively. These limits are mostly consistent with thermal noise over a wide range of $k$ after our data quality cuts, despite performing a relatively conservative analysis designed to minimize signal loss. Our results are validated with both statistical tests on the data and end-to-end pipeline simulations. We also report updated constraints on the astrophysics of reionization and the cosmic dawn. Using multiple independent modeling and inference techniques previously employed by HERA Collaboration (2022b), we find that the intergalactic medium must have been heated above the adiabatic cooling limit at least as early as $z = 10.4$, ruling out a broad set of so-called "cold reionization" scenarios. If this heating is due to high-mass X-ray binaries during the cosmic dawn, as is generally believed, our result's 99% credible interval excludes the local relationship between soft X-ray luminosity and star formation and thus requires heating driven by evolved low-metallicity stars., Comment: 57 pages, 37 figures. Updated to match the accepted ApJ version. Corresponding author: Joshua S. Dillon

Fast Radio Bursts (FRBs) are recently discovered extra-galactic radio transients which are now used as novel cosmological probes. We show how the Bursts' Dispersion Measure can model-independently probe the history of Hydrogen reionization. Using a FlexKnot free-form parameterization to reconstruct the reionization history we predict an 11% accuracy constraint on the CMB optical depth, and 4% accuracy on the midpoint of reionization, to be achieved with 100 FRBs originating from redshifts z>5., Comment: 2 pages, 2 figures. Contribution to the 2022 Cosmology session of the 56th Rencontres de Moriond, based on arXiv:2107.14242 (Heimersheim, Sartorio, Fialkov, Lorimer)

I present our analysis of the 21cm power spectrum upper limits from the HERA radio interferometer, published in HERA Collaboration et al. 2022. We use the recent limits to constrain reionization and the properties of the IGM and galaxies in the early universe. I focus in particular on the possibility of a radio background in addition to the CMB (e.g. produced by early galaxies) which can lead to a stronger 21cm signal and is thus easier to constrain. I show what limits the HERA observations can put on these models and the IGM, and how this compares to existing constraints on the radio and X-ray background., Comment: 4 pages, 3 figures. Conference proceedings, talk based on arXiv:2108.07282 (HERA Collaboration)

Recently, the Hydrogen Epoch of Reionization Array (HERA) collaboration has produced the experiment's first upper limits on the power spectrum of 21-cm fluctuations at z~8 and 10. Here, we use several independent theoretical models to infer constraints on the intergalactic medium (IGM) and galaxies during the epoch of reionization (EoR) from these limits. We find that the IGM must have been heated above the adiabatic cooling threshold by z~8, independent of uncertainties about the IGM ionization state and the nature of the radio background. Combining HERA limits with galaxy and EoR observations constrains the spin temperature of the z~8 neutral IGM to 27 K < T_S < 630 K (2.3 K < T_S < 640 K) at 68% (95%) confidence. They therefore also place a lower bound on X-ray heating, a previously unconstrained aspects of early galaxies. For example, if the CMB dominates the z~8 radio background, the new HERA limits imply that the first galaxies produced X-rays more efficiently than local ones (with soft band X-ray luminosities per star formation rate constrained to L_X/SFR = { 10^40.2, 10^41.9 } erg/s/(M_sun/yr) at 68% confidence), consistent with expectations of X-ray binaries in low-metallicity environments. The z~10 limits require even earlier heating if dark-matter interactions (e.g., through millicharges) cool down the hydrogen gas. Using a model in which an extra radio background is produced by galaxies, we rule out (at 95% confidence) the combination of high radio and low X-ray luminosities of L_{r,\nu}/SFR > 3.9 x 10^24 W/Hz/(M_sun/yr) and L_X/SFR<10^40 erg/s/(M_sun/yr). The new HERA upper limits neither support nor disfavor a cosmological interpretation of the recent EDGES detection. The analysis framework described here provides a foundation for the interpretation of future HERA results., Comment: 40 pages, 19 figures, accepted to ApJ

Fast Radio Bursts (FRBs) are extra-galactic radio transients which exhibit a distance-dependent dispersion of their signal, and thus can be used as cosmological probes. In this article we, for the first time, apply a model-independent approach to measure reionization from synthetic FRB data assuming these signals are detected beyond redshift 5. This method allows us to constrain the full shape of the reionization history as well as the CMB optical depth $\tau$ while avoiding the problems of commonly used model-based techniques. 100 localized FRBs, originating from redshifts 5-15, could constrain (at 68% confidence level) the CMB optical depth to within 11%, and the midpoint of reionization to 4%, surpassing current state-of-the-art CMB bounds and quasar limits. Owing to the higher numbers of expected FRBs at lower redshifts, the $\tau$ constraints are asymmetric (+14%, -7%) providing a much stronger lower limit. Finally, we show that the independent constraints on reionization from FRBs will improve limits on other cosmological parameters such as the amplitude of the power spectrum of primordial fluctuations., Comment: 13 pages, 5 figures. Accepted by ApJ. Code and data available at https://zenodo.org/record/6542596

Many models of dark matter have been proposed in attempt to ease the $S_8$ tension between weak lensing and CMB experiments. One such exciting possibility is cannibalistic dark matter (CanDM), which has exothermal number-changing interactions allowing it to stay warm far into its non-relativistic regime. Here we investigate the cosmological implications of CanDM and how it impacts CMB anisotropies and the matter power spectrum, by implementing the model within a linear Einstein-Boltzmann solver. We show that CanDM suppresses the small scale matter power spectrum in a way very similar to light Warm Dark Matter or Hot Dark Matter. However, unlike in those models, the suppression may happen while the CanDM model still remains compatible with CMB constraints. We put strong constraints on the interaction strength of CanDM as a function of its abundance for both constant and temperature-dependent thermally-averaged cross sections. We find that the CanDM model can easily solve the $S_8$ tension (but has no impact on the Hubble tension). Indeed, it can accommodate values of $S_8$ of the order of 0.76 while being compatible with CMB+BAO data. However, as long as the $S_8$ tension remains moderate, the overall $\chi^2$ improvement is relatively small given the number of extra free parameters, and the CanDM model is not significantly preferred., Comment: 28 pages, 14 figures, 2 tables. Comments are welcome (we don't bite)