Search

The Automated Audio Captioning (AAC) task asks models to generate natural language descriptions of an audio input. Evaluating these machine-generated audio captions is a complex task that requires considering diverse factors, among them, auditory scene understanding, sound-object inference, temporal coherence, and the environmental context of the scene. While current methods focus on specific aspects, they often fail to provide an overall score that aligns well with human judgment. In this work, we propose CLAIR-A, a simple and flexible method that leverages the zero-shot capabilities of large language models (LLMs) to evaluate candidate audio captions by directly asking LLMs for a semantic distance score. In our evaluations, CLAIR-A better predicts human judgements of quality compared to traditional metrics, with a 5.8% relative accuracy improvement compared to the domain-specific FENSE metric and up to 11% over the best general-purpose measure on the Clotho-Eval dataset. Moreover, CLAIR-A offers more transparency by allowing the language model to explain the reasoning behind its scores, with these explanations rated up to 30% better by human evaluators than those provided by baseline methods. CLAIR-A is made publicly available at https://github.com/DavidMChan/clair-a., Comment: Code is publicly available at https://github.com/DavidMChan/clair-a

AI systems that serve natural language questions over databases promise to unlock tremendous value. Such systems would allow users to leverage the powerful reasoning and knowledge capabilities of language models (LMs) alongside the scalable computational power of data management systems. These combined capabilities would empower users to ask arbitrary natural language questions over custom data sources. However, existing methods and benchmarks insufficiently explore this setting. Text2SQL methods focus solely on natural language questions that can be expressed in relational algebra, representing a small subset of the questions real users wish to ask. Likewise, Retrieval-Augmented Generation (RAG) considers the limited subset of queries that can be answered with point lookups to one or a few data records within the database. We propose Table-Augmented Generation (TAG), a unified and general-purpose paradigm for answering natural language questions over databases. The TAG model represents a wide range of interactions between the LM and database that have been previously unexplored and creates exciting research opportunities for leveraging the world knowledge and reasoning capabilities of LMs over data. We systematically develop benchmarks to study the TAG problem and find that standard methods answer no more than 20% of queries correctly, confirming the need for further research in this area. We release code for the benchmark at https://github.com/TAG-Research/TAG-Bench.

The inference process for large language models is slow and memory-intensive, with one of the most critical bottlenecks being excessive Key-Value (KV) cache accesses. This paper introduces "Double Sparsity," a novel post-training sparse attention technique designed to alleviate this bottleneck by reducing KV cache access. Double Sparsity combines token sparsity, which focuses on utilizing only the important tokens for computing self-attention, with channel sparsity, an approach that uses important feature channels for identifying important tokens. Our key insight is that the pattern of channel sparsity is relatively static, allowing us to use offline calibration to make it efficient at runtime, thereby enabling accurate and efficient identification of important tokens. Moreover, this method can be combined with offloading to achieve significant memory usage reduction. Experimental results demonstrate that Double Sparsity can achieve $\frac{1}{16}$ token and channel sparsity with minimal impact on accuracy across various tasks, including wiki-2 perplexity, key-value retrieval, and long context benchmarks with models including Llama-2-7B, Llama-2-70B, and Mixtral-8x7B. It brings up to a 14.1$\times$ acceleration in attention operations and a 1.9$\times$ improvement in end-to-end inference on GPUs. With offloading, it achieves a decoding speed acceleration of 16.3$\times$ compared to state-of-the-art solutions at a sequence length of 256K. Our code is publicly available at https://github.com/andy-yang-1/DoubleSparse.

Recent advancements in Large Multimodal Models (LMMs) have made significant progress in the field of single-image visual question answering. However, these models face substantial challenges when tasked with queries that span extensive collections of images, similar to real-world scenarios like searching through large photo albums, finding specific information across the internet, or monitoring environmental changes through satellite imagery. This paper explores the task of Multi-Image Visual Question Answering (MIQA): given a large set of images and a natural language query, the task is to generate a relevant and grounded response. We propose a new public benchmark, dubbed "Visual Haystacks (VHs)," specifically designed to evaluate LMMs' capabilities in visual retrieval and reasoning over sets of unrelated images, where we perform comprehensive evaluations demonstrating that even robust closed-source models struggle significantly. Towards addressing these shortcomings, we introduce MIRAGE (Multi-Image Retrieval Augmented Generation), a novel retrieval/QA framework tailored for LMMs that confronts the challenges of MIQA with marked efficiency and accuracy improvements over baseline methods. Our evaluation shows that MIRAGE surpasses closed-source GPT-4o models by up to 11% on the VHs benchmark and offers up to 3.4x improvements in efficiency over text-focused multi-stage approaches., Comment: Project page: https://visual-haystacks.github.io

The $^{246}$Cm(n,$\gamma$) and $^{248}$Cm(n,$\gamma$) cross-sections have been measured at the Experimental Area 2 (EAR2) of the n_TOF facility at CERN with three C$_6$D$_6$ detectors. This measurement is part of a collective effort to improve the capture cross-section data for Minor Actinides (MAs), which are required to estimate the production and transmutation rates of these isotopes in light water reactors and innovative reactor systems. In particular, the neutron capture in $^{246}$Cm and $^{248}$Cm open the path for the formation of other Cm isotopes and heavier elements such as Bk and Cf and the knowledge of (n,$\gamma$) cross-sections of these Cm isotopes plays an important role in the transport, transmutation and storage of the spent nuclear fuel. The reactions $^{246}$Cm(n,$\gamma$) and $^{248}$Cm(n,$\gamma$) have been the two first capture measurements analyzed at n_TOF EAR2. Until this experiment and two recent measurements performed at J-PARC, there was only one set of data of the capture cross-sections of $^{246}$Cm and $^{248}$Cm, that was obtained in 1969 in an underground nuclear explosion experiment. In the measurement at n_TOF a total of 13 resonances of $^{246}$Cm between 4 and 400 eV and 5 of $^{248}$Cm between 7 and 100 eV have been identified and fitted. The radiative kernels obtained for $^{246}$Cm are compatible with JENDL-5, but some of them are not with JENDL-4, which has been adopted by JEFF-3.3 and ENDF/B-VIII.0. The radiative kernels obtained for the first three $^{248}$Cm resonances are compatible with JENDL-5, however, the other two are not compatible with any other evaluation and are 20% and 60% larger than JENDL-5.

Large language models (LLMs) exhibit impressive capabilities across a wide range of tasks, yet the choice of which model to use often involves a trade-off between performance and cost. More powerful models, though effective, come with higher expenses, while less capable models are more cost-effective. To address this dilemma, we propose several efficient router models that dynamically select between a stronger and a weaker LLM during inference, aiming to optimize the balance between cost and response quality. We develop a training framework for these routers leveraging human preference data and data augmentation techniques to enhance performance. Our evaluation on widely-recognized benchmarks shows that our approach significantly reduces costs-by over 2 times in certain cases-without compromising the quality of responses. Interestingly, our router models also demonstrate significant transfer learning capabilities, maintaining their performance even when the strong and weak models are changed at test time. This highlights the potential of these routers to provide a cost-effective yet high-performance solution for deploying LLMs.

The rapid evolution of language models has necessitated the development of more challenging benchmarks. Current static benchmarks often struggle to consistently distinguish between the capabilities of different models and fail to align with real-world user preferences. On the other hand, live crowd-sourced platforms like the Chatbot Arena collect a wide range of natural prompts and user feedback. However, these prompts vary in sophistication and the feedback cannot be applied offline to new models. In order to ensure that benchmarks keep up with the pace of LLM development, we address how one can evaluate benchmarks on their ability to confidently separate models and their alignment with human preference. Under these principles, we developed BenchBuilder, a living benchmark that filters high-quality prompts from live data sources to enable offline evaluation on fresh, challenging prompts. BenchBuilder identifies seven indicators of a high-quality prompt, such as the requirement for domain knowledge, and utilizes an LLM annotator to select a high-quality subset of prompts from various topic clusters. The LLM evaluation process employs an LLM judge to ensure a fully automated, high-quality, and constantly updating benchmark. We apply BenchBuilder on prompts from the Chatbot Arena to create Arena-Hard-Auto v0.1: 500 challenging user prompts from a wide range of tasks. Arena-Hard-Auto v0.1 offers 3x tighter confidence intervals than MT-Bench and achieves a state-of-the-art 89.1% agreement with human preference rankings, all at a cost of only $25 and without human labelers. The BenchBuilder pipeline enhances evaluation benchmarks and provides a valuable tool for developers, enabling them to extract high-quality benchmarks from extensive data with minimal effort.

We introduce Buffer of Thoughts (BoT), a novel and versatile thought-augmented reasoning approach for enhancing accuracy, efficiency and robustness of large language models (LLMs). Specifically, we propose meta-buffer to store a series of informative high-level thoughts, namely thought-template, distilled from the problem-solving processes across various tasks. Then for each problem, we retrieve a relevant thought-template and adaptively instantiate it with specific reasoning structures to conduct efficient reasoning. To guarantee the scalability and stability, we further propose buffer-manager to dynamically update the meta-buffer, thus enhancing the capacity of meta-buffer as more tasks are solved. We conduct extensive experiments on 10 challenging reasoning-intensive tasks, and achieve significant performance improvements over previous SOTA methods: 11% on Game of 24, 20% on Geometric Shapes and 51% on Checkmate-in-One. Further analysis demonstrate the superior generalization ability and model robustness of our BoT, while requiring only 12% of the cost of multi-query prompting methods (e.g., tree/graph of thoughts) on average. Notably, we find that our Llama3-8B+BoT has the potential to surpass Llama3-70B model. Our project is available at: https://github.com/YangLing0818/buffer-of-thought-llm, Comment: Project: https://github.com/YangLing0818/buffer-of-thought-llm

Recent advances in large language models (LLMs) for code applications have demonstrated remarkable zero-shot fluency and instruction following on challenging code related tasks ranging from test case generation to self-repair. Unsurprisingly, however, models struggle to compose syntactically valid programs in programming languages unrepresented in pre-training, referred to as very low-resource Programming Languages (VLPLs). VLPLs appear in crucial settings, including domain-specific languages for internal tools and tool-chains for legacy languages. Inspired by an HCI technique called natural program elicitation, we propose designing an intermediate language that LLMs ``naturally'' know how to use and which can be automatically compiled to a target VLPL. When LLMs generate code that lies outside of this intermediate language, we use compiler techniques to repair the code into programs in the intermediate language. Overall, we introduce \emph{synthetic programming elicitation and compilation} (SPEAC), an approach that enables LLMs to generate syntactically valid code even for VLPLs. We empirically evaluate the performance of SPEAC in a case study and find that, compared to existing retrieval and fine-tuning baselines, SPEAC produces syntactically correct programs significantly more frequently without sacrificing semantic correctness., Comment: 15 pages, 6 figures, 1 table

In this work, we obtain Hubble constant ($H_0$) estimates by using two galaxy cluster gas mass fraction measurement samples, Type Ia supernovae luminosity distances, and the validity of the cosmic distance duality relation. Notably, the angular diameter distance (ADD) to each galaxy cluster in the samples is determined by combining its gas mass fraction measurement with galaxy clustering observations, more precisely, the $\Omega_b/\Omega_m$ ratio. Such a combination results in a $H_0$ estimate that is independent of a specific cosmological framework. In one of the samples, the gas fraction measurements were calculated in spherical shells at radii near $r_{\rm 2500}$ (44 data points), while in the other (103 data points) the measurements were calculated within $ r_{\rm 500}$. We find $H_0=72.7^{+6.3}_{-5.6}$ km/s/Mpc at 68\% CL for the joint analysis of these data sets. We also investigate the impact on the $H_0$ determination by exploring the precision and number of gas mass fraction data by performing a data Monte Carlo simulation. Our simulations show that future measurements could achieve a precision of up to 5\% for $H_0$., Comment: 9 pages, 5 figures. H0 estimate updated with joint analysis of the two gas mass fraction samples

Beyond scaling base models with more data or parameters, fine-tuned adapters provide an alternative way to generate high fidelity, custom images at reduced costs. As such, adapters have been widely adopted by open-source communities, accumulating a database of over 100K adapters-most of which are highly customized with insufficient descriptions. This paper explores the problem of matching the prompt to a set of relevant adapters, built on recent work that highlight the performance gains of composing adapters. We introduce Stylus, which efficiently selects and automatically composes task-specific adapters based on a prompt's keywords. Stylus outlines a three-stage approach that first summarizes adapters with improved descriptions and embeddings, retrieves relevant adapters, and then further assembles adapters based on prompts' keywords by checking how well they fit the prompt. To evaluate Stylus, we developed StylusDocs, a curated dataset featuring 75K adapters with pre-computed adapter embeddings. In our evaluation on popular Stable Diffusion checkpoints, Stylus achieves greater CLIP-FID Pareto efficiency and is twice as preferred, with humans and multimodal models as evaluators, over the base model. See stylus-diffusion.github.io for more., Comment: Project Website: https://stylus-diffusion.github.io

Processing long contexts remains a challenge for large language models (LLMs) due to the quadratic computational and memory overhead of the self-attention mechanism and the substantial KV cache sizes during generation. We propose a novel approach to address this problem by learning contexts offline through context compression and in-domain parameter-efficient finetuning. Our method enables an LLM to create a concise representation of the original context and efficiently retrieve relevant information to answer questions accurately. We introduce LLoCO, a technique that combines context compression, retrieval, and parameter-efficient finetuning using LoRA. Our approach extends the effective context window of a 4k token LLaMA2-7B model to handle up to 128k tokens. We evaluate our approach on several long-context question-answering datasets, demonstrating that LLoCO significantly outperforms in-context learning while using $30\times$ fewer tokens during inference. LLoCO achieves up to $7.62\times$ speed-up and substantially reduces the cost of long document question answering, making it a promising solution for efficient long context processing. Our code is publicly available at https://github.com/jeffreysijuntan/lloco., Comment: The first two authors contributed equally to this work

Large Language Models (LLMs) are evolving beyond their classical role of providing information within dialogue systems to actively engaging with tools and performing actions on real-world applications and services. Today, humans verify the correctness and appropriateness of the LLM-generated outputs (e.g., code, functions, or actions) before putting them into real-world execution. This poses significant challenges as code comprehension is well known to be notoriously difficult. In this paper, we study how humans can efficiently collaborate with, delegate to, and supervise autonomous LLMs in the future. We argue that in many cases, "post-facto validation" - verifying the correctness of a proposed action after seeing the output - is much easier than the aforementioned "pre-facto validation" setting. The core concept behind enabling a post-facto validation system is the integration of an intuitive undo feature, and establishing a damage confinement for the LLM-generated actions as effective strategies to mitigate the associated risks. Using this, a human can now either revert the effect of an LLM-generated output or be confident that the potential risk is bounded. We believe this is critical to unlock the potential for LLM agents to interact with applications and services with limited (post-facto) human involvement. We describe the design and implementation of our open-source runtime for executing LLM actions, Gorilla Execution Engine (GoEX), and present open research questions towards realizing the goal of LLMs and applications interacting with each other with minimal human supervision. We release GoEX at https://github.com/ShishirPatil/gorilla/.

Despite recent advances in multimodal pre-training for visual description, state-of-the-art models still produce captions containing errors, such as hallucinating objects not present in a scene. The existing prominent metric for object hallucination, CHAIR, is limited to a fixed set of MS COCO objects and synonyms. In this work, we propose a modernized open-vocabulary metric, ALOHa, which leverages large language models (LLMs) to measure object hallucinations. Specifically, we use an LLM to extract groundable objects from a candidate caption, measure their semantic similarity to reference objects from captions and object detections, and use Hungarian matching to produce a final hallucination score. We show that ALOHa correctly identifies 13.6% more hallucinated objects than CHAIR on HAT, a new gold-standard subset of MS COCO Captions annotated for hallucinations, and 30.8% more on nocaps, where objects extend beyond MS COCO categories. Our code is available at https://davidmchan.github.io/aloha/., Comment: To appear at NAACL 2024

How galaxies regulate nuclear growth through gas accretion by supermassive black holes (SMBHs) is one of the most fundamental questions in galaxy evolution. One potential way to regulate nuclear growth is through a galactic wind that removes gas from the nucleus. It is unclear whether galactic winds are powered by jets, mechanical winds, radiation, or via magnetohydrodynamic (MHD) processes. Compact obscured nuclei (CONs) represent a significant phase of galactic nuclear growth. These galaxies hide growing SMBHs or unusual starbursts in their very opaque, extremely compact (r $<$ 100 pc) centres. They are found in approximately 30 % of the luminous and ultra-luminous infrared galaxy (LIRG and ULIRG) population. Here, we present high-resolution ALMA observations ($\sim$30 mas, $\sim$5 pc) of ground-state and vibrationally excited HCN towards ESO 320-G030 (IRAS 11506-3851). ESO 320-G030 is an isolated luminous infrared galaxy known to host a compact obscured nucleus and a kiloparsec-scale molecular wind. Our analysis of these high-resolution observations excludes the possibility of a starburst-driven wind, a mechanically or energy driven active galactic nucleus (AGN) wind, and exposes a molecular MDH wind. These results imply that the nuclear evolution of galaxies and the growth of SMBHs are similar to the growth of hot cores or protostars where gravitational collapse of the nuclear torus drives a MHD wind. These results mean galaxies are capable, in part, of regulating the evolution of their nuclei without feedback., Comment: 9 pages, 10 figures

Pretraining Large Language Models (LLMs) on large corpora of textual data is now a standard paradigm. When using these LLMs for many downstream applications, it is common to additionally bake in new knowledge (e.g., time-critical news, or private domain knowledge) into the pretrained model either through RAG-based-prompting, or fine-tuning. However, the optimal methodology for the model to gain such new knowledge remains an open question. In this paper, we present Retrieval Augmented FineTuning (RAFT), a training recipe that improves the model's ability to answer questions in a "open-book" in-domain settings. In RAFT, given a question, and a set of retrieved documents, we train the model to ignore those documents that don't help in answering the question, which we call, distractor documents. RAFT accomplishes this by citing verbatim the right sequence from the relevant document that would help answer the question. This coupled with RAFT's chain-of-thought-style response helps improve the model's ability to reason. In domain-specific RAG, RAFT consistently improves the model's performance across PubMed, HotpotQA, and Gorilla datasets, presenting a post-training recipe to improve pre-trained LLMs to in-domain RAG. RAFT's code and demo are open-sourced at github.com/ShishirPatil/gorilla.

The neutron time-of-flight facility n_TOF at CERN is a spallation source dedicated to measurements of neutron-induced reaction cross-sections of interest in nuclear technologies, astrophysics, and other applications. Since 2014, Experimental ARea 2 (EAR2) is operational and delivers a neutron fluence of $4\times 10^7$ neutrons per nominal proton pulse, which is 50 times higher than the one of Experimental ARea 1 (EAR1) of $8\times10^5$ neutrons per pulse. The high neutron flux at EAR2 results in high counting rates in the detectors that challenged the previously existing capture detection systems. For this reason, a Segmented Total Energy Detector (sTED) has been developed to overcome the limitations in the detectors response, by reducing the active volume per module and by using a photomultiplier (PMT) optimized for high counting rates. This paper presents the main characteristics of the sTED, including energy and time resolution, response to $\gamma$-rays, and provides as well details of the use of the Pulse Height Weighting Technique (PHWT) with this detector. The sTED has been validated to perform neutron-capture cross-section measurements in EAR2 in the neutron energy range from thermal up to at least 400 keV. The detector has already been successfully used in several measurements at n_TOF EAR2.

Analytical database providers (e.g., Redshift, Databricks, BigQuery) have rapidly added support for invoking Large Language Models (LLMs) through native user-defined functions (UDFs) to help users perform natural language tasks, such as classification, entity extraction, and translation, inside analytical workloads. For instance, an analyst might want to extract customer sentiments on millions of product reviews. However, LLM inference is highly expensive in both computational and economic terms: for example, an NVIDIA L4 GPU running Llama2-7B can only process 6 KB of text per second. In this paper, we explore how to optimize LLM inference for analytical workloads that invoke LLMs within relational queries. We show that relational queries present novel opportunities for accelerating LLM inference, including reordering rows to maximize key-value (KV) cache reuse within the LLM inference engine, reordering columns within a row to further increase cache reuse, and deduplicating redundant inference requests. We implement these optimizations in Apache Spark, with vLLM as the model serving backend and achieve up to 4.4x improvement in end-to-end latency on a benchmark of diverse LLM-based queries on real datasets. To the best of our knowledge, this is the first work to explicitly address the problem of optimizing LLM invocations within SQL queries.

Large Language Models (LLMs) have unlocked new capabilities and applications; however, evaluating the alignment with human preferences still poses significant challenges. To address this issue, we introduce Chatbot Arena, an open platform for evaluating LLMs based on human preferences. Our methodology employs a pairwise comparison approach and leverages input from a diverse user base through crowdsourcing. The platform has been operational for several months, amassing over 240K votes. This paper describes the platform, analyzes the data we have collected so far, and explains the tried-and-true statistical methods we are using for efficient and accurate evaluation and ranking of models. We confirm that the crowdsourced questions are sufficiently diverse and discriminating and that the crowdsourced human votes are in good agreement with those of expert raters. These analyses collectively establish a robust foundation for the credibility of Chatbot Arena. Because of its unique value and openness, Chatbot Arena has emerged as one of the most referenced LLM leaderboards, widely cited by leading LLM developers and companies. Our demo is publicly available at \url{https://chat.lmsys.org}.

Nuclear regions of ultraluminous and luminous infrared galaxies (U/LIRGs) are powered by starbursts and/or active galactic nuclei (AGNs). These regions are often obscured by extremely high columns of gas and dust. Molecular lines in the submillimeter windows have the potential to determine the physical conditions of these compact obscured nuclei (CONs). We aim to reveal the distributions of HCN and HCO$^+$ emission in local U/LIRGs and investigate whether and how they are related to galaxy properties. Using ALMA, we have conducted sensitive observations of the HCN J=3--2 and HCO$^+$ J=3--2 lines toward 23 U/LIRGs in the local Universe (z < 0.07) with a spatial resolution of ~0.3" (~50--400 pc). We detected both HCN and HCO$^+$ in 21 galaxies, only HCN in one galaxy, and neither in one galaxy. The global HCN/HCO$^+$ line ratios, averaged over scales of ~0.5--4 kpc, range from 0.4 to 2.3, with an unweighted mean of 1.1. These line ratios appear to have no systematic trend with bolometric AGN luminosity or star formation rate. The line ratio varies with position and velocity within each galaxy, with an average interquartile range of 0.38 on a spaxel-by-spaxel basis. In eight out of ten galaxies known to have outflows and/or inflows, we found spatially and kinematically symmetric structures of high line ratios. These structures appear as a collimated bicone in two galaxies and as a thin spherical shell in six galaxies. Non-LTE analysis suggests that the high HCN/HCO$^+$ line ratio in outflows is predominantly influenced by the abundance ratio. Chemical model calculations indicate that the enhancement of HCN abundance in outflows is likely due to high-temperature chemistry triggered by shock heating. These results imply that the HCN/HCO$^+$ line ratio can aid in identifying the outflow geometry when the shock velocity of the outflows is sufficiently high to heat the gas., Comment: 52 pages, 35 figures, accepted for publication in Astronomy & Astrophysics

We analyze JWST NIRSpec$+$MIRI/MRS observations of the infrared (IR) gas-phase molecular bands of the most enshrouded source (D1) within the interacting system and luminous IR galaxy II Zw 096. We report the detection of rovibrational lines of H$_2$O $\nu_2$=1-0 ($\sim$5.3-7.2 $\mu$m) and $^{12}$CO $\nu$=1-0 ($\sim$4.45-4.95 $\mu$m) in D1. The CO band shows the R- and P-branches in emission and the spectrum of the H$_2$O band shows the P-branch in emission and the R-branch in absorption. The H$_2$O R-branch in absorption unveils an IR-bright embedded compact source in D1 and the CO broad component features a highly turbulent environment. From both bands, we also identified extended intense star-forming (SF) activity associated with circumnuclear photodissociation regions (PDRs), consistent with the strong emission of the ionised 7.7 $\mu$m polycyclic aromatic hydrocarbon band in this source. By including the 4.5-7.0 $\mu$m continuum information derived from the H$_2$O and CO analysis, we modelled the IR emission of D1 with a dusty torus and SF component. The torus is very compact (diameter of $\sim$3 pc at 5 $\mu$m) and characterised by warm dust ($\sim$ 370 K), giving an IR surface brightness of $\sim$3.6$\times$10$^{8}$ L$_{\rm sun}$/pc$^2$. This result suggests the presence of a dust-obscured active galactic nucleus (AGN) in D1, which has an exceptionally high covering factor that prevents the direct detection of AGN emission. Our results open a new way to investigate the physical conditions of inner dusty tori via modelling the observed IR molecular bands., Comment: Accepted for publication in A&A (official date of acceptance: 27/12/2023; submitted 27/11/2023). 11 pages, 10 figures

High-demand LLM inference services (e.g., ChatGPT and BARD) support a wide range of requests from short chat conversations to long document reading. To ensure that all client requests are processed fairly, most major LLM inference services have request rate limits, to ensure that no client can dominate the request queue. However, this rudimentary notion of fairness also results in under-utilization of the resources and poor client experience when there is spare capacity. While there is a rich literature on fair scheduling, serving LLMs presents new challenges due to their unpredictable request lengths and their unique batching characteristics on parallel accelerators. This paper introduces the definition of LLM serving fairness based on a cost function that accounts for the number of input and output tokens processed. To achieve fairness in serving, we propose a novel scheduling algorithm, the Virtual Token Counter (VTC), a fair scheduler based on the continuous batching mechanism. We prove a 2x tight upper bound on the service difference between two backlogged clients, adhering to the requirement of work-conserving. Through extensive experiments, we demonstrate the superior performance of VTC in ensuring fairness, especially in contrast to other baseline methods, which exhibit shortcomings under various conditions. The reproducible code is available at https://github.com/Ying1123/VTC-artifact

Current open-source Large Multimodal Models (LMMs) excel at tasks such as open-vocabulary language grounding and segmentation but can suffer under false premises when queries imply the existence of something that is not actually present in the image. We observe that existing methods that fine-tune an LMM to segment images significantly degrade their ability to reliably determine ("see") if an object is present and to interact naturally with humans ("say"), a form of catastrophic forgetting. In this work, we propose a cascading and joint training approach for LMMs to solve this task, avoiding catastrophic forgetting of previous skills. Our resulting model can "see" by detecting whether objects are present in an image, "say" by telling the user if they are not, proposing alternative queries or correcting semantic errors in the query, and finally "segment" by outputting the mask of the desired objects if they exist. Additionally, we introduce a novel False Premise Correction benchmark dataset, an extension of existing RefCOCO(+/g) referring segmentation datasets (which we call FP-RefCOCO(+/g)). The results show that our method not only detects false premises up to 55% better than existing approaches, but under false premise conditions produces relative cIOU improvements of more than 31% over baselines, and produces natural language feedback judged helpful up to 67% of the time., Comment: Project Page: https://see-say-segment.github.io

Large language models (LLMs) are increasingly used for complex tasks that require multiple generation calls, advanced prompting techniques, control flow, and structured inputs/outputs. However, efficient systems are lacking for programming and executing these applications. We introduce SGLang, a system for efficient execution of complex language model programs. SGLang consists of a frontend language and a runtime. The frontend simplifies programming with primitives for generation and parallelism control. The runtime accelerates execution with novel optimizations like RadixAttention for KV cache reuse and compressed finite state machines for faster structured output decoding. Experiments show that SGLang achieves up to 6.4x higher throughput compared to state-of-the-art inference systems on various large language and multi-modal models on tasks including agent control, logical reasoning, few-shot learning benchmarks, JSON decoding, retrieval-augmented generation pipelines, and multi-turn chat. The code is publicly available at https://github.com/sgl-project/sglang

How do two sets of images differ? Discerning set-level differences is crucial for understanding model behaviors and analyzing datasets, yet manually sifting through thousands of images is impractical. To aid in this discovery process, we explore the task of automatically describing the differences between two $\textbf{sets}$ of images, which we term Set Difference Captioning. This task takes in image sets $D_A$ and $D_B$, and outputs a description that is more often true on $D_A$ than $D_B$. We outline a two-stage approach that first proposes candidate difference descriptions from image sets and then re-ranks the candidates by checking how well they can differentiate the two sets. We introduce VisDiff, which first captions the images and prompts a language model to propose candidate descriptions, then re-ranks these descriptions using CLIP. To evaluate VisDiff, we collect VisDiffBench, a dataset with 187 paired image sets with ground truth difference descriptions. We apply VisDiff to various domains, such as comparing datasets (e.g., ImageNet vs. ImageNetV2), comparing classification models (e.g., zero-shot CLIP vs. supervised ResNet), summarizing model failure modes (supervised ResNet), characterizing differences between generative models (e.g., StableDiffusionV1 and V2), and discovering what makes images memorable. Using VisDiff, we are able to find interesting and previously unknown differences in datasets and models, demonstrating its utility in revealing nuanced insights., Comment: CVPR 2024 Oral

Large language models are increasingly trained on all the data ever produced by humans. Many have raised concerns about the trustworthiness of public benchmarks due to potential contamination in pre-training or fine-tuning datasets. While most data decontamination efforts apply string matching (e.g., n-gram overlap) to remove benchmark data, we show that these methods are insufficient, and simple variations of test data (e.g., paraphrasing, translation) can easily bypass these decontamination measures. Furthermore, we demonstrate that if such variation of test data is not eliminated, a 13B model can easily overfit a test benchmark and achieve drastically high performance, on par with GPT-4. We validate such observations in widely used benchmarks such as MMLU, GSK8k, and HumanEval. To address this growing risk, we propose a stronger LLM-based decontamination method and apply it to widely used pre-training and fine-tuning datasets, revealing significant previously unknown test overlap. For example, in pre-training sets such as RedPajama-Data-1T and StarCoder-Data, we identified that 8-18\% of the HumanEval benchmark overlaps. Interestingly, we also find such contamination in synthetic dataset generated by GPT-3.5/4, suggesting a potential risk of unintentional contamination. We urge the community to adopt stronger decontamination approaches when using public benchmarks. Moreover, we call for the community to actively develop fresh one-time exams to evaluate models accurately. Our decontamination tool is publicly available at https://github.com/lm-sys/llm-decontaminator.

The "pretrain-then-finetune" paradigm is commonly adopted in the deployment of large language models. Low-Rank Adaptation (LoRA), a parameter-efficient fine-tuning method, is often employed to adapt a base model to a multitude of tasks, resulting in a substantial collection of LoRA adapters derived from one base model. We observe that this paradigm presents significant opportunities for batched inference during serving. To capitalize on these opportunities, we present S-LoRA, a system designed for the scalable serving of many LoRA adapters. S-LoRA stores all adapters in the main memory and fetches the adapters used by the currently running queries to the GPU memory. To efficiently use the GPU memory and reduce fragmentation, S-LoRA proposes Unified Paging. Unified Paging uses a unified memory pool to manage dynamic adapter weights with different ranks and KV cache tensors with varying sequence lengths. Additionally, S-LoRA employs a novel tensor parallelism strategy and highly optimized custom CUDA kernels for heterogeneous batching of LoRA computation. Collectively, these features enable S-LoRA to serve thousands of LoRA adapters on a single GPU or across multiple GPUs with a small overhead. Compared to state-of-the-art libraries such as HuggingFace PEFT and vLLM (with naive support of LoRA serving), S-LoRA can improve the throughput by up to 4 times and increase the number of served adapters by several orders of magnitude. As a result, S-LoRA enables scalable serving of many task-specific fine-tuned models and offers the potential for large-scale customized fine-tuning services. The code is available at https://github.com/S-LoRA/S-LoRA

We present an investigation into diffusion models for molecular generation, with the aim of better understanding how their predictions compare to the results of physics-based calculations. The investigation into these models is driven by their potential to significantly accelerate electronic structure calculations using machine learning, without requiring expensive first-principles datasets for training interatomic potentials. We find that the inference process of a popular diffusion model for de novo molecular generation is divided into an exploration phase, where the model chooses the atomic species, and a relaxation phase, where it adjusts the atomic coordinates to find a low-energy geometry. As training proceeds, we show that the model initially learns about the first-order structure of the potential energy surface, and then later learns about higher-order structure. We also find that the relaxation phase of the diffusion model can be re-purposed to sample the Boltzmann distribution over conformations and to carry out structure relaxations. For structure relaxations, the model finds geometries with ~10x lower energy than those produced by a classical force field for small organic molecules. Initializing a density functional theory (DFT) relaxation at the diffusion-produced structures yields a >2x speedup to the DFT relaxation when compared to initializing at structures relaxed with a classical force field.

One of the critical aspects for the accurate determination of neutron capture cross sections when combining time-of-flight and total energy detector techniques is the characterization and control of systematic uncertainties associated to the measuring devices. In this work we explore the most conspicuous effects associated to harsh count rate conditions: dead-time and pile-up effects. Both effects, when not properly treated, can lead to large systematic uncertainties and bias in the determination of neutron cross sections. In the majority of neutron capture measurements carried out at the CERN n\_TOF facility, the detectors of choice are the C$_{6}$D$_{6}$ liquid-based either in form of large-volume cells or recently commissioned sTED detector array, consisting of much smaller-volume modules. To account for the aforementioned effects, we introduce a Monte Carlo model for these detectors mimicking harsh count rate conditions similar to those happening at the CERN n\_TOF 20~m fligth path vertical measuring station. The model parameters are extracted by comparison with the experimental data taken at the same facility during 2022 experimental campaign. We propose a novel methodology to consider both, dead-time and pile-up effects simultaneously for these fast detectors and check the applicability to experimental data from $^{197}$Au($n$,$\gamma$), including the saturated 4.9~eV resonance which is an important component of normalization for neutron cross section measurements.

This paper proposes a simple mathematical model of non-stationary and non-linear stochastic dynamics, which approximates a (globally) non-stationary and non-linear stochastic process by its locally (or \emph{"piecewise"}) stationary version. Profiting from the elegance and simplicity of both, the exact mathematical model referred to as the Ornstein-Uhlenbeck stochastic process (which is \emph{globally} stationary, Markov and Gaussian) and of the Lyapunov criterion associated with the stability of stationarity, we show that the proposed non-linear non-stationary model provides a natural extension of the Onsager-Machlup theory of equilibrium thermal fluctuations, to the realm of non-stationary, non-linear, and non-equilibrium processes. As an illustrative application, we then apply the extended non-equilibrium Onsager-Machlup theory, to the description of thermal fluctuations and irreversible relaxation processes in liquids, leading to the main exact equations employed to construct the \emph{non-equilibrium self-consistent generalized Langevin equation} (NE-SCGLE) theory of irreversible processes in liquids. This generic theory has demonstrated that the most intriguing and long-unsolved questions of the glass and the gel transitions are understood as a natural consequence of the second law of thermodynamics, enunciated in terms of the proposed piecewise stationary stochastic mathematical model.

In this work, we constrain the Hubble constant parameter, $H_0$, using a combination of the Pantheon sample and galaxy clusters (GC) measurements from minimal cosmological assumptions. Assuming the validity of the cosmic distance duality relation, an estimator is created for $H_0$ that only depends on simple geometrical distances, which is evaluated from Pantheon and a GC angular diameter distance sample afterward. The statistical and systematic errors in GC measurements are summed in quadrature in our analysis. We find $H_0 = 67.22 \pm 6.07$ ${\rm km \ s^{-1} Mpc^{-1}}$ in $1\sigma$ confidence level (C.L.). This measurement presents an error of around 9\%, showing that future and better GC measurements can shed light on the current Hubble tension, Comment: 8 pages, two figures

The evaluation of machine-generated image captions poses an interesting yet persistent challenge. Effective evaluation measures must consider numerous dimensions of similarity, including semantic relevance, visual structure, object interactions, caption diversity, and specificity. Existing highly-engineered measures attempt to capture specific aspects, but fall short in providing a holistic score that aligns closely with human judgments. Here, we propose CLAIR, a novel method that leverages the zero-shot language modeling capabilities of large language models (LLMs) to evaluate candidate captions. In our evaluations, CLAIR demonstrates a stronger correlation with human judgments of caption quality compared to existing measures. Notably, on Flickr8K-Expert, CLAIR achieves relative correlation improvements over SPICE of 39.6% and over image-augmented methods such as RefCLIP-S of 18.3%. Moreover, CLAIR provides noisily interpretable results by allowing the language model to identify the underlying reasoning behind its assigned score. Code is available at https://davidmchan.github.io/clair/, Comment: To Appear at EMNLP 2023

We use JWST/MIRI MRS spectroscopy of a sample of six local obscured type 1.9/2 active galactic nuclei (AGN) to compare their nuclear mid-IR absorption bands with the level of nuclear obscuration traced by X-rays. This study is the first to use sub-arcsecond angular resolution data of local obscured AGN to investigate the nuclear mid-IR absorption bands with a wide wavelength coverage (4.9-28.1 $\mu$m). All the nuclei show the 9.7 $\mu$m silicate band in absorption. We compare the strength of the 9.7 and 18 $\mu$m silicate features with torus model predictions. The observed silicate features are generally well explained by clumpy and smooth torus models. We report the detection of the 6 $\mu$m dirty water ice band (i.e., a mix of water and other molecules such as CO and CO$_2$) at sub-arcsecond scales ($\sim$0.26 arcsec at 6 $\mu$m; inner $\sim$50 pc) in a sample of local AGN with different levels of nuclear obscuration in the range log N$_{\rm H}^{\rm X-Ray}$(cm$^{-2}$)$\sim22-25$. We find a good correlation between the 6 $\mu$m water ice optical depths and N$_{\rm H}^{\rm X-Ray}$. This result indicates that the water ice absorption might be a reliable tracer of the nuclear intrinsic obscuration in AGN. The weak water ice absorption in less obscured AGN (log N$_H^{X-ray}$ (cm$^{-2}$)$\lesssim$23.0 cm$^{-2}$) might be related to the hotter dust temperature ($>$T$_{sub}^{H_2O}\sim$110 K) expected to be reached in the outer layers of the torus due to their more inhomogeneous medium. Our results suggest it might be necessary to include the molecular content, such as, H$_2$O, aliphatic hydrocarbons (CH-) and more complex PAH molecules in torus models to better constrain key parameters such as the torus covering factor (i.e. nuclear obscuration)., Comment: Accepted for publication in A&A. 13 pages, 12 Figures

Large language models (LLMs) have revolutionized AI, but are constrained by limited context windows, hindering their utility in tasks like extended conversations and document analysis. To enable using context beyond limited context windows, we propose virtual context management, a technique drawing inspiration from hierarchical memory systems in traditional operating systems that provide the appearance of large memory resources through data movement between fast and slow memory. Using this technique, we introduce MemGPT (Memory-GPT), a system that intelligently manages different memory tiers in order to effectively provide extended context within the LLM's limited context window, and utilizes interrupts to manage control flow between itself and the user. We evaluate our OS-inspired design in two domains where the limited context windows of modern LLMs severely handicaps their performance: document analysis, where MemGPT is able to analyze large documents that far exceed the underlying LLM's context window, and multi-session chat, where MemGPT can create conversational agents that remember, reflect, and evolve dynamically through long-term interactions with their users. We release MemGPT code and data for our experiments at https://memgpt.ai., Comment: Code and data available at https://research.memgpt.ai

FlashAttention (Dao, 2023) effectively reduces the quadratic peak memory usage to linear in training transformer-based large language models (LLMs) on a single GPU. In this paper, we introduce DISTFLASHATTN, a distributed memory-efficient attention mechanism optimized for long-context LLMs training. We propose three key techniques: token-level workload balancing, overlapping key-value communication, and a rematerialization-aware gradient checkpointing algorithm. We evaluate DISTFLASHATTN on Llama-7B and variants with sequence lengths from 32K to 512K. DISTFLASHATTN achieves 8x longer sequences, 4.45 - 5.64x speedup compared to Ring Self-Attention, 2 - 8x longer sequences, 1.24 - 2.01x speedup compared to Megatron-LM with FlashAttention. It achieves 1.67x and 1.26 - 1.88x speedup compared to recent Ring Attention and DeepSpeed-Ulysses. Code is available at https://github.com/RulinShao/LightSeq.

Studying how people interact with large language models (LLMs) in real-world scenarios is increasingly important due to their widespread use in various applications. In this paper, we introduce LMSYS-Chat-1M, a large-scale dataset containing one million real-world conversations with 25 state-of-the-art LLMs. This dataset is collected from 210K unique IP addresses in the wild on our Vicuna demo and Chatbot Arena website. We offer an overview of the dataset's content, including its curation process, basic statistics, and topic distribution, highlighting its diversity, originality, and scale. We demonstrate its versatility through four use cases: developing content moderation models that perform similarly to GPT-4, building a safety benchmark, training instruction-following models that perform similarly to Vicuna, and creating challenging benchmark questions. We believe that this dataset will serve as a valuable resource for understanding and advancing LLM capabilities. The dataset is publicly available at https://huggingface.co/datasets/lmsys/lmsys-chat-1m.

High throughput serving of large language models (LLMs) requires batching sufficiently many requests at a time. However, existing systems struggle because the key-value cache (KV cache) memory for each request is huge and grows and shrinks dynamically. When managed inefficiently, this memory can be significantly wasted by fragmentation and redundant duplication, limiting the batch size. To address this problem, we propose PagedAttention, an attention algorithm inspired by the classical virtual memory and paging techniques in operating systems. On top of it, we build vLLM, an LLM serving system that achieves (1) near-zero waste in KV cache memory and (2) flexible sharing of KV cache within and across requests to further reduce memory usage. Our evaluations show that vLLM improves the throughput of popular LLMs by 2-4$\times$ with the same level of latency compared to the state-of-the-art systems, such as FasterTransformer and Orca. The improvement is more pronounced with longer sequences, larger models, and more complex decoding algorithms. vLLM's source code is publicly available at https://github.com/vllm-project/vllm, Comment: SOSP 2023

We analyze JWST/NIRSpec observations of the CO rovibrational v=1-0 band at 4.67um around the dust-embedded southern active galactic nucleus (AGN) of NGC3256 (d=40Mpc; L(IR)=10^11.6 Lsun). We classify the CO v=1-0 spectra into three categories based on the behavior of P- and R-branches of the band: (a) both branches in absorption toward the nucleus; (b) P-R asymmetry (P-branch in emission and R-branch in absorption) along the disk of the galaxy; and (c) both branches in emission in the outflow region above and below the disk. In this paper, we focus on the outflow. The CO v=1-0 emission can be explained by the vibrational excitation of CO in the molecular outflow by the bright mid-IR ~4.7um continuum from the AGN up to r~250pc. We model the ratios between the P(J+2) and R(J) transitions of the band to derive the physical properties (column density, kinetic temperature, and CO-to-H2 conversion factor, alpha_CO) of the outflowing gas. We find that the 12CO v=1-0 emission is optically thick for J<4, while the 13CO v=1-0 emission remains optically thin. From the P(2)/R(0) ratio, we identify a temperature gradient in the outflow from >40K in the central 100pc to <15K at 250pc sampling the cooling of the molecular gas in the outflow. We used three methods to derive alpha_CO in eight 100pc (0.5") apertures in the outflow by fitting the P(J+2)/R(J) ratios with non-LTE models. We obtain low median alpha_CO factors (0.40-0.61) x 3.2e-4/[CO/H2] Msun (K km/s/pc2)^-1 in the outflow regions. This implies that outflow rates and energetics might be overestimated if a ULIRG-like alpha_CO, which is 1.3-2 times larger, is assumed. The reduced alpha_CO can be explained if the outflowing molecular clouds are not virialized. We also report the first extragalactic detection of a broad (sigma=0.0091um) spectral feature at 4.645um associated with aliphatic deuterium on polycyclic aromatic hydrocarbons (D_n-PAH)., Comment: Accepted for publication in A&A.17 pages, 13 figures

We report a study on the static and dynamic properties of several liquid 5$d$ transition metals at thermodynamic conditions near their respective melting points. This is performed by resorting to ab initio molecular dynamics simulations in the framework of the density functional theory. Results are presented for the static structure factors and pair distribution functions; moreover, the local short range order in the liquid metal is also analized. As for the dynamical properties, both single-particle and collective properties are evaluated. The dynamical structure shows the propagating density fluctuations, and the respective dispersion relation is obtained. Results are also obtained for the longitudinal and transverse current spectral functions along with the associated dispersion of collective excitations. For some metals, we found the existence of two branches of transverse collective excitations in the region around the main peak of the structure factor. Finally, several transport coefficients are also calculated., Comment: 20 pages, 19 figures, 5 tables

The safety of autonomous vehicles (AVs) depends on their ability to perform complex computations on high-volume sensor data in a timely manner. Their ability to run these computations with state-of-the-art models is limited by the processing power and slow update cycles of their onboard hardware. In contrast, cloud computing offers the ability to burst computation to vast amounts of the latest generation of hardware. However, accessing these cloud resources requires traversing wireless networks that are often considered to be too unreliable for real-time AV driving applications. Our work seeks to harness this unreliable cloud to enhance the accuracy of an AV's decisions, while ensuring that it can always fall back to its on-board computational capabilities. We identify three mechanisms that can be used by AVs to safely leverage the cloud for accuracy enhancements, and elaborate why current execution systems fail to enable these mechanisms. To address these limitations, we provide a system design based on the speculative execution of an AV's pipeline in the cloud, and show the efficacy of this approach in simulations of complex real-world scenarios that apply these mechanisms., Comment: IROS 2023 (to appear); 8 pages, 7 figures, 2 tables