Search

Scaling data and compute is critical to the success of machine learning. However, scaling demands predictability: we want methods to not only perform well with more compute or data, but also have their performance be predictable from small-scale runs, without running the large-scale experiment. In this paper, we show that value-based off-policy RL methods are predictable despite community lore regarding their pathological behavior. First, we show that data and compute requirements to attain a given performance level lie on a Pareto frontier, controlled by the updates-to-data (UTD) ratio. By estimating this frontier, we can predict this data requirement when given more compute, and this compute requirement when given more data. Second, we determine the optimal allocation of a total resource budget across data and compute for a given performance and use it to determine hyperparameters that maximize performance for a given budget. Third, this scaling behavior is enabled by first estimating predictable relationships between hyperparameters, which is used to manage effects of overfitting and plasticity loss unique to RL. We validate our approach using three algorithms: SAC, BRO, and PQL on DeepMind Control, OpenAI gym, and IsaacGym, when extrapolating to higher levels of data, compute, budget, or performance.

We propose a novel framework, Meta Chain-of-Thought (Meta-CoT), which extends traditional Chain-of-Thought (CoT) by explicitly modeling the underlying reasoning required to arrive at a particular CoT. We present empirical evidence from state-of-the-art models exhibiting behaviors consistent with in-context search, and explore methods for producing Meta-CoT via process supervision, synthetic data generation, and search algorithms. Finally, we outline a concrete pipeline for training a model to produce Meta-CoTs, incorporating instruction tuning with linearized search traces and reinforcement learning post-training. Finally, we discuss open research questions, including scaling laws, verifier roles, and the potential for discovering novel reasoning algorithms. This work provides a theoretical and practical roadmap to enable Meta-CoT in LLMs, paving the way for more powerful and human-like reasoning in artificial intelligence.

A fundamental open challenge in modern LLM scaling is the lack of understanding around emergent capabilities. In particular, language model pretraining loss is known to be highly predictable as a function of compute. However, downstream capabilities are far less predictable -- sometimes even exhibiting emergent jumps -- which makes it challenging to anticipate the capabilities of future models. In this work, we first pose the task of emergence prediction: given access to current LLMs that have random few-shot accuracy on a task, can we predict whether future models (GPT-N+1) will have non-trivial accuracy on that task? We then discover a simple insight for this problem: finetuning LLMs on a given task can shift the point in scaling at which emergence occurs towards less capable models. To operationalize this insight, we can finetune LLMs with varying amounts of data and fit a parametric function that predicts when emergence will occur (i.e., "emergence laws"). We validate this approach using four standard NLP benchmarks where large-scale open-source LLMs already demonstrate emergence (MMLU, GSM8K, CommonsenseQA, and CoLA). Using only small-scale LLMs, we find that, in some cases, we can accurately predict whether models trained with up to 4x more compute have emerged. Finally, we present a case study of two realistic uses for emergence prediction.

A reliable deep learning system should be able to accurately express its confidence with respect to its predictions, a quality known as calibration. One of the most effective ways to produce reliable confidence estimates with a pre-trained model is by applying a post-hoc recalibration method. Popular recalibration methods like temperature scaling are typically fit on a small amount of data and work in the model's output space, as opposed to the more expressive feature embedding space, and thus usually have only one or a handful of parameters. However, the target distribution to which they are applied is often complex and difficult to fit well with such a function. To this end we propose \textit{selective recalibration}, where a selection model learns to reject some user-chosen proportion of the data in order to allow the recalibrator to focus on regions of the input space that can be well-captured by such a model. We provide theoretical analysis to motivate our algorithm, and test our method through comprehensive experiments on difficult medical imaging and zero-shot classification tasks. Our results show that selective recalibration consistently leads to significantly lower calibration error than a wide range of selection and recalibration baselines., Comment: Published in Transactions on Machine Learning Research (07/2024)

This report examines the success of students with lower levels of high school performance who began in transfer-level math as a result of Assembly Bill 705 (Irwin 2017). AB 705 is historic legislation that transformed placement and eventually ended developmental education in California's community colleges. Since community colleges are open access institutions, it is particularly important to monitor the impact of such reforms on students who are perceived to be underprepared--particularly in math, which has historically been a persistent barrier to academic progress for many students. In this report, the authors focus on students in the lowest placement band of the placement rules (i.e., default placement rules), that were developed by the Multiple Measures Assessment Project (MMAP) to support colleges in operationalizing AB 705. The authors compare the predicted versus actual success rates of students in the lowest placement band who were placed into, and began in, transfer-level math courses post-AB 705. If the predictions overestimated the success of these students, California community colleges may have grounds to revisit the efficacy of developmental education as a means for meeting AB 705's mandates.

Enabling LLMs to improve their outputs by using more test-time computation is a critical step towards building generally self-improving agents that can operate on open-ended natural language. In this paper, we study the scaling of inference-time computation in LLMs, with a focus on answering the question: if an LLM is allowed to use a fixed but non-trivial amount of inference-time compute, how much can it improve its performance on a challenging prompt? Answering this question has implications not only on the achievable performance of LLMs, but also on the future of LLM pretraining and how one should tradeoff inference-time and pre-training compute. Despite its importance, little research attempted to understand the scaling behaviors of various test-time inference methods. Moreover, current work largely provides negative results for a number of these strategies. In this work, we analyze two primary mechanisms to scale test-time computation: (1) searching against dense, process-based verifier reward models; and (2) updating the model's distribution over a response adaptively, given the prompt at test time. We find that in both cases, the effectiveness of different approaches to scaling test-time compute critically varies depending on the difficulty of the prompt. This observation motivates applying a "compute-optimal" scaling strategy, which acts to most effectively allocate test-time compute adaptively per prompt. Using this compute-optimal strategy, we can improve the efficiency of test-time compute scaling by more than 4x compared to a best-of-N baseline. Additionally, in a FLOPs-matched evaluation, we find that on problems where a smaller base model attains somewhat non-trivial success rates, test-time compute can be used to outperform a 14x larger model.

When developing a clinical prediction model, the sample size of the development dataset is a key consideration. Small sample sizes lead to greater concerns of overfitting, instability, poor performance and lack of fairness. Previous research has outlined minimum sample size calculations to minimise overfitting and precisely estimate the overall risk. However even when meeting these criteria, the uncertainty (instability) in individual-level risk estimates may be considerable. In this article we propose how to examine and calculate the sample size required for developing a model with acceptably precise individual-level risk estimates to inform decisions and improve fairness. We outline a five-step process to be used before data collection or when an existing dataset is available. It requires researchers to specify the overall risk in the target population, the (anticipated) distribution of key predictors in the model, and an assumed 'core model' either specified directly (i.e., a logistic regression equation is provided) or based on specified C-statistic and relative effects of (standardised) predictors. We produce closed-form solutions that decompose the variance of an individual's risk estimate into Fisher's unit information matrix, predictor values and total sample size; this allows researchers to quickly calculate and examine individual-level uncertainty interval widths and classification instability for specified sample sizes. Such information can be presented to key stakeholders (e.g., health professionals, patients, funders) using prediction and classification instability plots to help identify the (target) sample size required to improve trust, reliability and fairness in individual predictions. Our proposal is implemented in software module pmstabilityss. We provide real examples and emphasise the importance of clinical context including any risk thresholds for decision making., Comment: 36 pages, 6 figures, 1 table

Additive manufacturing (AM; 3D printing) in aluminium using laser powder bed fusion provides a new design space for lightweight mirror production. Printing layer-by-layer enables the use of intricate lattices for mass reduction, as well as organic shapes generated by topology optimisation, resulting in mirrors optimised for function as opposed to subtractive machining. However, porosity, a common AM defect, is present in printed aluminium and it is a result of the printing environment being either too hot or too cold, or gas entrapped bubbles within the aluminium powder. When present in an AM mirror substrates, porosity manifests as pits on the reflective surface, which increases micro-roughness and therefore scattered light. There are different strategies to reduce the impact of porosity: elimination during printing, coating the aluminium print in nickel phosphorous, or to apply a heat and pressure treatment to close the pores, commonly known as a hot isostatic press (HIP). This paper explores the application of HIP on printed aluminium substrates intended for mirror production using single point diamond turning (SPDT). The objective of the HIP is to reduce porosity whilst targeting a small grain growth within the aluminium, which is important in allowing the SPDT to generate surfaces with low micro-roughness. For this study, three disks, 50 mm diameter by 5 mm, were printed in AlSi10Mg at 0 deg, 45 deg, and 90 deg with respect to the build plate. X-ray computed tomography (XCT) was conducted before and after the HIP cycle to confirm the effectiveness of HIP to close porosity. The disks were SPDT and the micro-roughness evaluated. Mechanical testing and electron backscatter diffraction (EBSD) was used to quantify the mechanical strength and the grain size after HIP., Comment: 14 pages, 8 figures, submitted to SPIE Astronomical Telescopes & Instrumentation, Advances in Optical and Mechanical Technologies for Telescopes and Instrumentation VI (Conference 13100, Paper 141)

When evaluating the performance of a model for individualised risk prediction, the sample size needs to be large enough to precisely estimate the performance measures of interest. Current sample size guidance is based on precisely estimating calibration, discrimination, and net benefit, which should be the first stage of calculating the minimum required sample size. However, when a clinically important threshold is used for classification, other performance measures can also be used. We extend the previously published guidance to precisely estimate threshold-based performance measures. We have developed closed-form solutions to estimate the sample size required to target sufficiently precise estimates of accuracy, specificity, sensitivity, PPV, NPV, and F1-score in an external evaluation study of a prediction model with a binary outcome. This approach requires the user to pre-specify the target standard error and the expected value for each performance measure. We describe how the sample size formulae were derived and demonstrate their use in an example. Extension to time-to-event outcomes is also considered. In our examples, the minimum sample size required was lower than that required to precisely estimate the calibration slope, and we expect this would most often be the case. Our formulae, along with corresponding Python code and updated R and Stata commands (pmvalsampsize), enable researchers to calculate the minimum sample size needed to precisely estimate threshold-based performance measures in an external evaluation study. These criteria should be used alongside previously published criteria to precisely estimate the calibration, discrimination, and net-benefit., Comment: 27 pages, 1 figure

For California's community colleges, fall 2019 marked the deadline for implementing the state's historic developmental education legislation, Assembly Bill 705 (Irwin 2017). AB 705 drove transformative changes in community college math and English programs as colleges implemented research-based placement policies designed to maximize students' likelihood of completing transferable college-level courses in those disciplines. The result was a large increase in the proportion of students beginning in transfer-level coursework instead of developmental education. After the pandemic hit in the second semester of AB 705 implementation, throughput gains were maintained despite a 14-percentage point decline in community college enrollments. Despite the well-documented positive student outcomes associated with AB 705 reforms, it is one of several legislative mandates that some believe have had the unintended consequence of reducing California's community college enrollment. Any connection between AB 705 and enrollment is difficult to examine after fall 2019, given the enrollment declines driven by the pandemic. For this reason, this report analyzes enrollment in California's community colleges in the context of AB 705 with a focus on fall 2019, the AB 705 implementation deadline.

Humans rely on effective representations to learn from few examples and abstract useful information from sensory data. Inducing such representations in machine learning models has been shown to improve their performance on various benchmarks such as few-shot learning and robustness. However, finding effective training procedures to achieve that goal can be challenging as psychologically rich training data such as human similarity judgments are expensive to scale, and Bayesian models of human inductive biases are often intractable for complex, realistic domains. Here, we address this challenge by leveraging a Bayesian notion of generative similarity whereby two data points are considered similar if they are likely to have been sampled from the same distribution. This measure can be applied to complex generative processes, including probabilistic programs. We incorporate generative similarity into a contrastive learning objective to enable learning of embeddings that express human cognitive representations. We demonstrate the utility of our approach by showing that it can be used to capture human-like representations of shape regularity, abstract Euclidean geometric concepts, and semantic hierarchies for natural images.

Over the past several years, the misidentification of SpaceX Starlink satellites as Unidentified Aerial Phenomena (UAP) by pilots and laypersons has generated unnecessary aviation risk and confusion. The many deployment and orbital evolution strategies, coupled with changing sun specular reflection angles, contribute to this gap in space situational awareness. In this paper we present a case analysis of an incident that generated multiple, corroborating reports of a UAP from five pilots on two commercial airline flights over the Pacific Ocean on August 10th, 2022. This incident included two cell phone photos and a video of an unrecognizable and possibly anomalous phenomenon. We then use supplemental two-line elements (TLEs) for the Starlink train of satellites launched that same day and Automatic Dependent Surveillance Broadcast (ADS-B) data from the flight with the photographs to reconstruct a view of these satellites from the cockpit at the time and place of the sighting. The success of this work demonstrates an approach that could, in principle, warn aviators about satellites that could be visible in unusual or novel illumination configurations, thus increasing space situational awareness and supporting aviation safety. We conclude with recommendations for governments and satellite operators to provide better a-priori data that can be used to create advisories to aviators and the public. The automated simulation of known specular reflection off constellations of satellites could also support researchers investigating sightings of unfamiliar aerial or aerospace objects as likely being from normal versus novel space events., Comment: 35 pages, 10 figures, submitted to the 4th IAA Conference on Space Situational Awareness (ICSSA), May 8-10 2024, Daytona Beach, FL, USA; Abstract number AH-92328

As Large Language Models (LLMs) become more pervasive across various users and scenarios, identifying potential issues when using these models becomes essential. Examples of such issues include: bias, inconsistencies, and hallucination. Although auditing the LLM for these problems is often warranted, such a process is neither easy nor accessible for most. An effective method is to probe the LLM using different versions of the same question. This could expose inconsistencies in its knowledge or operation, indicating potential for bias or hallucination. However, to operationalize this auditing method at scale, we need an approach to create those probes reliably and automatically. In this paper we propose the LLMAuditor framework which is an automatic, and scalable solution, where one uses a different LLM along with human-in-the-loop (HIL). This approach offers verifiability and transparency, while avoiding circular reliance on the same LLM, and increasing scientific rigor and generalizability. Specifically, LLMAuditor includes two phases of verification using humans: standardized evaluation criteria to verify responses, and a structured prompt template to generate desired probes. A case study using questions from the TruthfulQA dataset demonstrates that we can generate a reliable set of probes from one LLM that can be used to audit inconsistencies in a different LLM. This process is enhanced by our structured prompt template with HIL, which not only boosts the reliability of our approach in auditing but also yields the delivery of less hallucinated results. The novelty of our research stems from the development of a comprehensive, general-purpose framework that includes a HIL verified prompt template for auditing responses generated by LLMs.