Search

There is widespread optimism that frontier Large Language Models (LLMs) and LLM-augmented systems have the potential to rapidly accelerate scientific discovery across disciplines. Today, many benchmarks exist to measure LLM knowledge and reasoning on textbook-style science questions, but few if any benchmarks are designed to evaluate language model performance on practical tasks required for scientific research, such as literature search, protocol planning, and data analysis. As a step toward building such benchmarks, we introduce the Language Agent Biology Benchmark (LAB-Bench), a broad dataset of over 2,400 multiple choice questions for evaluating AI systems on a range of practical biology research capabilities, including recall and reasoning over literature, interpretation of figures, access and navigation of databases, and comprehension and manipulation of DNA and protein sequences. Importantly, in contrast to previous scientific benchmarks, we expect that an AI system that can achieve consistently high scores on the more difficult LAB-Bench tasks would serve as a useful assistant for researchers in areas such as literature search and molecular cloning. As an initial assessment of the emergent scientific task capabilities of frontier language models, we measure performance of several against our benchmark and report results compared to human expert biology researchers. We will continue to update and expand LAB-Bench over time, and expect it to serve as a useful tool in the development of automated research systems going forward. A public subset of LAB-Bench is available for use at the following URL: https://huggingface.co/datasets/futurehouse/lab-bench, Comment: 40 pages, 5 main figures, 1 main table, 2 supplemental figures, 4 supplemental tables. Submitted to NeurIPS 2024 Datasets and Benchmarks track (in review)

Large language models (LLMs) have emerged as powerful tools in chemistry, significantly impacting molecule design, property prediction, and synthesis optimization. This review highlights LLM capabilities in these domains and their potential to accelerate scientific discovery through automation. We also review LLM-based autonomous agents: LLMs with a broader set of tools to interact with their surrounding environment. These agents perform diverse tasks such as paper scraping, interfacing with automated laboratories, and synthesis planning. As agents are an emerging topic, we extend the scope of our review of agents beyond chemistry and discuss across any scientific domains. This review covers the recent history, current capabilities, and design of LLMs and autonomous agents, addressing specific challenges, opportunities, and future directions in chemistry. Key challenges include data quality and integration, model interpretability, and the need for standard benchmarks, while future directions point towards more sophisticated multi-modal agents and enhanced collaboration between agents and experimental methods. Due to the quick pace of this field, a repository has been built to keep track of the latest studies: https://github.com/ur-whitelab/LLMs-in-science.

Large Language Models (LLMs) generalize well across language tasks, but suffer from hallucinations and uninterpretability, making it difficult to assess their accuracy without ground-truth. Retrieval-Augmented Generation (RAG) models have been proposed to reduce hallucinations and provide provenance for how an answer was generated. Applying such models to the scientific literature may enable large-scale, systematic processing of scientific knowledge. We present PaperQA, a RAG agent for answering questions over the scientific literature. PaperQA is an agent that performs information retrieval across full-text scientific articles, assesses the relevance of sources and passages, and uses RAG to provide answers. Viewing this agent as a question answering model, we find it exceeds performance of existing LLMs and LLM agents on current science QA benchmarks. To push the field closer to how humans perform research on scientific literature, we also introduce LitQA, a more complex benchmark that requires retrieval and synthesis of information from full-text scientific papers across the literature. Finally, we demonstrate PaperQA's matches expert human researchers on LitQA.

Aqueous solubility is a valuable yet challenging property to predict. Computing solubility using first-principles methods requires accounting for the competing effects of entropy and enthalpy, resulting in long computations for relatively poor accuracy. Data-driven approaches, such as deep learning, offer improved accuracy and computational efficiency but typically lack uncertainty quantification. Additionally, ease of use remains a concern for any computational technique, resulting in the sustained popularity of group-based contribution methods. In this work, we addressed these problems with a deep learning model with predictive uncertainty that runs on a static website (without a server). This approach moves computing needs onto the website visitor without requiring installation, removing the need to pay for and maintain servers. Our model achieves satisfactory results in solubility prediction. Furthermore, we demonstrate how to create molecular property prediction models that balance uncertainty and ease of use. The code is available at https://github.com/ur-whitelab/mol.dev, and the model is usable at https://mol.dev.

Evolutionary symbolic regression (SR) fits a symbolic equation to data, which gives a concise interpretable model. We explore using SR as a method to propose which data to gather in an active learning setting with physical constraints. SR with active learning proposes which experiments to do next. Active learning is done with query by committee, where the Pareto frontier of equations is the committee. The physical constraints improve proposed equations in very low data settings. These approaches reduce the data required for SR and achieves state of the art results in data required to rediscover known equations.

The dual use of machine learning applications, where models can be used for both beneficial and malicious purposes, presents a significant challenge. This has recently become a particular concern in chemistry, where chemical datasets containing sensitive labels (e.g. toxicological information) could be used to develop predictive models that identify novel toxins or chemical warfare agents. To mitigate dual use risks, we propose a model-agnostic method of selectively noising datasets while preserving the utility of the data for training deep neural networks in a beneficial region. We evaluate the effectiveness of the proposed method across least squares, a multilayer perceptron, and a graph neural network. Our findings show selectively noised datasets can induce model variance and bias in predictions for sensitive labels with control, suggesting the safe sharing of datasets containing sensitive information is feasible. We also find omitting sensitive data often increases model variance sufficiently to mitigate dual use. This work is proposed as a foundation for future research on enabling more secure and collaborative data sharing practices and safer machine learning applications in chemistry.

Ultra-large chemical libraries are reaching 10s to 100s of billions of molecules. A challenge for these libraries is to efficiently check if a proposed molecule is present. Here we propose and study Bloom filters for testing if a molecule is present in a set using either string or fingerprint representations. Bloom filters are small enough to hold billions of molecules in just a few GB of memory and check membership in sub milliseconds. We found string representations can have a false positive rate below 1% and require significantly less storage than using fingerprints. Canonical SMILES with Bloom filters with the simple FNV hashing function provide fast and accurate membership tests with small memory requirements. We provide a general implementation and specific filters for detecting if a molecule is purchasable, patented, or a natural product according to existing databases at https://github.com/whitead/molbloom

Over the last decades, excellent computational chemistry tools have been developed. Integrating them into a single platform with enhanced accessibility could help reaching their full potential by overcoming steep learning curves. Recently, large-language models (LLMs) have shown strong performance in tasks across domains, but struggle with chemistry-related problems. Moreover, these models lack access to external knowledge sources, limiting their usefulness in scientific applications. In this study, we introduce ChemCrow, an LLM chemistry agent designed to accomplish tasks across organic synthesis, drug discovery, and materials design. By integrating 18 expert-designed tools, ChemCrow augments the LLM performance in chemistry, and new capabilities emerge. Our agent autonomously planned and executed the syntheses of an insect repellent, three organocatalysts, and guided the discovery of a novel chromophore. Our evaluation, including both LLM and expert assessments, demonstrates ChemCrow's effectiveness in automating a diverse set of chemical tasks. Surprisingly, we find that GPT-4 as an evaluator cannot distinguish between clearly wrong GPT-4 completions and Chemcrow's performance. Our work not only aids expert chemists and lowers barriers for non-experts, but also fosters scientific advancement by bridging the gap between experimental and computational chemistry., Comment: Experimental results

Large language models (LLMs) are able to do accurate classification with zero or only a few examples (in-context learning). We show a prompting system that enables regression with uncertainty for in-context learning with frozen LLM (GPT-3, GPT-3.5, and GPT-4) models, allowing predictions without features or architecture tuning. By incorporating uncertainty, our approach enables Bayesian optimization for catalyst or molecule optimization using natural language, eliminating the need for training or simulation. Here, we performed the optimization using the synthesis procedure of catalysts to predict properties. Working with natural language mitigates difficulty synthesizability since the literal synthesis procedure is the model's input. We showed that in-context learning could improve past a model context window (maximum number of tokens the model can process at once) as data is gathered via example selection, allowing the model to scale better. Although our method does not outperform all baselines, it requires zero training, feature selection, and minimal computing while maintaining satisfactory performance. We also find Gaussian Process Regression on text embeddings is strong at Bayesian optimization. The code is available in our GitHub repository: https://github.com/ur-whitelab/BO-LIFT

String-based molecular representations play a crucial role in cheminformatics applications, and with the growing success of deep learning in chemistry, have been readily adopted into machine learning pipelines. However, traditional string-based representations such as SMILES are often prone to syntactic and semantic errors when produced by generative models. To address these problems, a novel representation, SELF-referencIng Embedded Strings (SELFIES), was proposed that is inherently 100% robust, alongside an accompanying open-source implementation. Since then, we have generalized SELFIES to support a wider range of molecules and semantic constraints and streamlined its underlying grammar. We have implemented this updated representation in subsequent versions of \selfieslib, where we have also made major advances with respect to design, efficiency, and supported features. Hence, we present the current status of \selfieslib (version 2.1.1) in this manuscript., Comment: 11 pages, 2 figures

Detector instrumentation is at the heart of scientific discoveries. Cutting edge technologies enable US particle physics to play a leading role worldwide. This report summarizes the current status of instrumentation for High Energy Physics (HEP), the challenges and needs of future experiments and indicates high priority research areas. The Snowmass Instrumentation Frontier studies detector technologies and Research and Development (R&D) needed for future experiments in collider physics, neutrino physics, rare and precision physics and at the cosmic frontier. It is divided into more or less diagonal areas with some overlap among a few of them. We lay out five high-level key messages that are geared towards ensuring the health and competitiveness of the US detector instrumentation community, and thus the entire particle physics landscape., Comment: 53 pages, 0 figures, contribution to Snowmass 2021

The 2022 Snowmass Community Summer Study consisted of two years of engagement with the particle physics community to craft a US particle physics strategy for the next decade, culminating in the 2022 Snowmass Book. This Report of the Topical Group on Calorimetry forms the basis for a small section of the Snowmass Book dealing with the community's vision for calorimetry going forward. It is the distillation of ideas put forth in public meetings and numerous white papers already freely available. The Report describes particle flow and dual readout approaches, integration of precision timing, and trends in calorimetric materials, and highlights key research directions for the next decade., Comment: 13 pages, 1 table, contribution to Snowmass 2021

The rapid development of machine learning (ML) methods has fundamentally affected numerous applications ranging from computer vision, biology, and medicine to accounting and text analytics. Until now, it was the availability of large and often labeled data sets that enabled significant breakthroughs. However, the adoption of these methods in classical physical disciplines has been relatively slow, a tendency that can be traced to the intrinsic differences between correlative approaches of purely data-based ML and the causal hypothesis-driven nature of physical sciences. Furthermore, anomalous behaviors of classical ML necessitate addressing issues such as explainability and fairness of ML. We also note the sequence in which deep learning became mainstream in different scientific disciplines - starting from medicine and biology and then towards theoretical chemistry, and only after that, physics - is rooted in the progressively more complex level of descriptors, constraints, and causal structures available for incorporation in ML architectures. Here we put forth that over the next decade, physics will become a new data, and this will continue the transition from dot-coms and scientific computing concepts of the 90ies to big data of 2000-2010 to deep learning of 2010-2020 to physics-enabled scientific ML.

We derive a formulation of molecular dynamics that generates only symmetric configurations. We implement it for all 2D planar and 3D space groups. An atlas of 2D Lennard-Jones crystals under all planar groups is created with symmetric molecular dynamics.

Efficient quantum memories will be an essential building block of large scale networked quantum systems and provide a link between flying photonic qubits and atomic or quasi-atomic local quantum processors. To provide a path to scalability avoidance of bulky, difficult to maintain systems such as high vacuum and low temperature cryogenics is imperative. Memory efficiencies above 50% are required to be operating above the quantum no-cloning limit. Such high efficiencies have only been achieved in systems with photon sources tailored to the memory bandwidth. In this paper we explore the combination of an ultralow spectral bandwidth source of single photons from cavity-enhanced spontaneous parametric down-conversion with a gas-ensemble atomic memory. Our rubidium vapour gradient echo memory achieves 84$\pm$3% recall efficiency of single photons: a record for an always-ready, hot, and vacuum system free optical memory., Comment: 11 pages, 10 figures

The International Linear Collider (ILC) is on the table now as a new global energy-frontier accelerator laboratory taking data in the 2030s. The ILC addresses key questions for our current understanding of particle physics. It is based on a proven accelerator technology. Its experiments will challenge the Standard Model of particle physics and will provide a new window to look beyond it. This document brings the story of the ILC up to date, emphasizing its strong physics motivation, its readiness for construction, and the opportunity it presents to the US and the global particle physics community., Comment: 356 pages, Large pdf file (40 MB) submitted to Snowmass 2021; v2 references to Snowmass contributions added, additional authors; v3 references added, some updates, additional authors

Exchanging hydrophobic alkyl-based side chains to hydrophilic glycol-based side chains is a widely adopted method for improving mixed-transport device performance, despite the impact on solid state packing and polymer-electrolyte interactions being poorly understood. Presented here is a Molecular Dynamics (MD) force field for modelling alkoxylated and glycolated polythiophenes. The force field is validated against known packing motifs for their monomer crystals. MD simulations, coupled with X-ray Diffraction (XRD), show that alkoxylated polythiophenes will pack with a `tilted stack' and straight interdigitating side chains, whilst their glycolated counterpart will pack with a `deflected stack' and an s-bend side chain configuration. MD simulations reveal water penetration pathways into the alkoxylated and glycolated crystals - through the {\pi}-stack and through the lamellar stack respectively. Finally, the two distinct ways tri-ethylene glycol polymers can bind to cations are revealed, showing the formation of a meta-stable single bound state, or an energetically deep double bound state, both with a strong side chain length dependance. The minimum energy pathways for the formation of the chelates are identified, showing the physical process through which cations can bind to one or two side chains of a glycolated polythiophene, with consequences for ion transport in bithiophene semiconductors., Comment: 10 pages, 4 figures

Mathematical modeling of disease outbreaks can infer the future trajectory of an epidemic, which can inform policy decisions. Another task is inferring the origin of a disease, which is relatively difficult with current mathematical models. Such frameworks -- across varying levels of complexity -- are typically sensitive to input data on epidemic parameters, case-counts and mortality rates, which are generally noisy and incomplete. To alleviate these limitations, we propose a maximum entropy framework that fits epidemiological models, provides a calibrated infection origin probabilities, and is robust to noise due to a prior belief model. Maximum entropy is agnostic to the parameters or model structure used and allows for flexible use when faced with sparse data conditions and incomplete knowledge in the dynamical phase of disease-spread, providing for more reliable modeling at early stages of outbreaks. We evaluate the performance of our model by predicting future disease trajectories in synthetic graph networks and the real mobility network of New York state. In addition, unlike existing approaches, we demonstrate that the method can be used to infer the origin of the outbreak with accurate confidence. Indeed, despite the prevalent belief on the feasibility of contact-tracing being limited to the initial stages of an outbreak, we report the possibility of reconstructing early disease dynamics, including the epidemic seed, at advanced stages.

Chemistry research has both high material and computational costs to conduct experiments. Institutions thus consider chemical data to be valuable and there have been few efforts to construct large public datasets for machine learning. Another challenge is that different intuitions are interested in different classes of molecules, creating heterogeneous data that cannot be easily joined by conventional distributed training. In this work, we introduce federated heterogeneous molecular learning to address these challenges. Federated learning allows end-users to build a global model collaboratively while keeping the training data distributed over isolated clients. Due to the lack of related research, we first simulate a heterogeneous federated learning benchmark (FedChem) by jointly performing scaffold splitting and latent Dirichlet allocation on existing datasets for heterogeneously distributed client data. Our results on FedChem show that significant learning challenges arise when working with heterogeneous molecules across clients. We then propose a method to alleviate the problem, namely Federated Learning by Instance reweighTing (FLIT(+)). FLIT(+) can align the local training across heterogeneous clients by improving the performance for uncertain samples. Comprehensive experiments conducted on our new benchmark FedChem validate the advantages of this method over other federated learning schemes. FedChem should enable a new type of collaboration for improving AI in chemistry that mitigates concerns about valuable chemical data.

Natural language processing models have emerged that can generate usable software and automate a number of programming tasks with high fidelity. These tools have yet to have an impact on the chemistry community. Yet, our initial testing demonstrates that this form of Artificial Intelligence is poised to transform chemistry and chemical engineering research. Here, we review developments that brought us to this point, examine applications in chemistry, and give our perspective on how this may fundamentally alter research and teaching., Comment: 6 pages, 3 figures, plus 8 page SI with additional examples and extended figure details

Vehicle-to-grid (V2G) is a promising approach to solve the problem of grid-level intermittent supply and demand mismatch, caused due to renewable energy resources, because it uses the existing resource of electric vehicle (EV) batteries as the energy storage medium. EV battery design together with an impetus on profitability for participating EV owners is pivotal for V2G success. To better understand what battery device parameters are most important for V2G adoption, we model the economics of V2G process under realistic conditions. Most previous studies that perform V2G economic analysis, assume ideal driving conditions, use linear battery degradation models, or only consider V2G for ancillary services. Our model accounts realistic battery degradation, empirical charging efficiencies, for randomness in commute behavior, and historic hourly electricity prices in six cities in the United States. We model user behavior with Bayesian optimization to provide a best-case scenario for V2G. Across all cities, we find that charging rate and efficiency are the most important factors that determine EV users' profits. Surprisingly, EV battery cost and thus degradation due to cycling has little effect. These findings should help focus research on figures of merit that better reflect real usage of batteries in a V2G economy., Comment: 17 Pages, 10 Figures, 1 Table

Computational fluid dynamics (CFD) analysis is widely used in engineering. Although CFD calculations are accurate, the computational cost associated with complex systems makes it difficult to obtain empirical equations between system variables. Here we combine active learning (AL) and symbolic regression (SR) to get a symbolic equation for system variables from CFD simulations. Gaussian process regression-based AL allows for automated selection of variables by selecting the most instructive points from the available range of possible parameters. The results from these experiments are then passed to SR to find empirical symbolic equations for CFD models. This approach is scalable and applicable for any desired number of CFD design parameters. To demonstrate the effectiveness, we use this method with two model systems. We recover an empirical equation for the pressure drop in a bent pipe and a new equation for predicting backflow in a heart valve under arotic insufficiency., Comment: 14 pages, 6 figures, 3 tables

Inferring the input parameters of simulators from observations is a crucial challenge with applications from epidemiology to molecular dynamics. Here we show a simple approach in the regime of sparse data and approximately correct models, which is common when trying to use an existing model to infer latent variables with observed data. This approach is based on the principle of maximum entropy (MaxEnt) and provably makes the smallest change in the latent joint distribution to fit new data. This method requires no likelihood or model derivatives and its fit is insensitive to prior strength, removing the need to balance observed data fit with prior belief. The method requires the ansatz that data is fit in expectation, which is true in some settings and may be reasonable in all with few data points. The method is based on sample reweighting, so its asymptotic run time is independent of prior distribution dimension. We demonstrate this MaxEnt approach and compare with other likelihood-free inference methods across three systems: a point particle moving in a gravitational field, a compartmental model of epidemic spread and finally molecular dynamics simulation of a protein., Comment: 16 pages, 4 figures