Search

We determine the Dirac series of $U(n,2)$ and give related information about Dirac series: the spin lowest $K$-types are multiplicity free, and the Dirac cohomology are consistent with the Dirac index., Comment: 23 pages

We conduct systematic calculations of the S-wave fully strange systems with ``normal" $\left(J^{P C}=0^{++}, 1^{+-}, 2^{++}\right)$ and ``exotic" $\left(J^{P C}=0^{+-}, 1^{++}, 2^{+-}\right)$ C-parities, which are the strange analogue of the fully charmed tetraquark state $X(6900)$. Within a constituent quark potential model, we employ the Gaussian expansion method to solve the four-body Schr\"odinger equation and the complex scaling method to identify resonant states. We obtain a series of resonant states and zero-width states in the mass range of 2.7 to 3.3 GeV, with their widths ranging from less than 1 MeV to about 50 MeV. Their rms radii strongly indicate that they are compact tetraquark states. Among these states, the $T_{4s,2^{++}}(2714)$ may be the most likely one to be observed experimentally. We urge the experimental exploration of the $2^{++}$ $s s \bar{s} \bar{s}$ state around 2.7 GeV in the $\phi\phi$ channel. Since the lowest S-wave $s s \bar{s} \bar{s}$ state is around 2.7 GeV, the compact P-wave $s s \bar{s} \bar{s}$ states are expected to be heavier. Hence, $\phi(2170)$ and $X(2370)$ are unlikely to be compact tetraquark states., Comment: 9 pages, 2 figures. Comments are welcomed

The 1st Workshop on Data Contamination (CONDA 2024) focuses on all relevant aspects of data contamination in natural language processing, where data contamination is understood as situations where evaluation data is included in pre-training corpora used to train large scale models, compromising evaluation results. The workshop fostered a shared task to collect evidence on data contamination in current available datasets and models. The goal of the shared task and associated database is to assist the community in understanding the extent of the problem and to assist researchers in avoiding reporting evaluation results on known contaminated resources. The shared task provides a structured, centralized public database for the collection of contamination evidence, open to contributions from the community via GitHub pool requests. This first compilation paper is based on 566 reported entries over 91 contaminated sources from a total of 23 contributors. The details of the individual contamination events are available in the platform. The platform continues to be online, open to contributions from the community., Comment: https://huggingface.co/spaces/CONDA-Workshop/Data-Contamination-Database

Large language models (LLMs) exhibit impressive capabilities across a wide range of tasks, yet the choice of which model to use often involves a trade-off between performance and cost. More powerful models, though effective, come with higher expenses, while less capable models are more cost-effective. To address this dilemma, we propose several efficient router models that dynamically select between a stronger and a weaker LLM during inference, aiming to optimize the balance between cost and response quality. We develop a training framework for these routers leveraging human preference data and data augmentation techniques to enhance performance. Our evaluation on widely-recognized benchmarks shows that our approach significantly reduces costs-by over 2 times in certain cases-without compromising the quality of responses. Interestingly, our router models also demonstrate significant transfer learning capabilities, maintaining their performance even when the strong and weak models are changed at test time. This highlights the potential of these routers to provide a cost-effective yet high-performance solution for deploying LLMs.

We use the quark potential model to calculate the mass spectrum of the S-wave fully heavy tetraquark systems with different flavors, including the $ bc\bar b\bar c, bb\bar c\bar c, cc\bar c\bar b $ and $ bb\bar b\bar c $ systems. We employ the Gaussian expansion method to solve the four-body Schr\"odinger equation, and the complex scaling method to identify resonant states. The $ bc\bar b\bar c, bb\bar c\bar c, cc\bar c\bar b $ and $ bb\bar b\bar c $ resonant states are obtained in the mass regions of $ (13.2,13.5) $, $ (13.3,13.6) $, $ (10.0,10.3) $, $ (16.5,16.7) $ GeV, respectively. Among these states, the $ bc\bar b\bar c $ tetraquark states are the most promising ones to be discovered in the near future. We recommend the experimental exploration of the $ 1^{++} $ and $ 2^{++} $ $ bc\bar b\bar c $ states with masses near $ 13.3 $ GeV in the $ J/\psi\Upsilon $ channel. From the root-mean-square radii, we find that all the resonant states we have identified are compact tetraquark states., Comment: 10 pages,7 figures,8 tables

Retrieval-augmented generation (RAG) has shown promising potential to enhance the accuracy and factuality of language models (LMs). However, imperfect retrievers or noisy corpora can introduce misleading or even erroneous information to the retrieved contents, posing a significant challenge to the generation quality. Existing RAG methods typically address this challenge by directly predicting final answers despite potentially noisy inputs, resulting in an implicit denoising process that is difficult to interpret and verify. On the other hand, the acquisition of explicit denoising supervision is often costly, involving significant human efforts. In this work, we propose InstructRAG, where LMs explicitly learn the denoising process through self-synthesized rationales -- First, we instruct the LM to explain how the ground-truth answer is derived from retrieved documents. Then, these rationales can be used either as demonstrations for in-context learning of explicit denoising or as supervised fine-tuning data to train the model. Compared to standard RAG approaches, InstructRAG requires no additional supervision, allows for easier verification of the predicted answers, and effectively improves generation accuracy. Experiments show InstructRAG consistently outperforms existing RAG methods in both training-free and trainable scenarios, achieving a relative improvement of 8.3% over the best baseline method on average across five knowledge-intensive benchmarks. Extensive analysis indicates that InstructRAG scales well with increased numbers of retrieved documents and consistently exhibits robust denoising ability even in out-of-domain datasets, demonstrating strong generalizability., Comment: Code: https://github.com/weizhepei/InstructRAG

The rapid evolution of language models has necessitated the development of more challenging benchmarks. Current static benchmarks often struggle to consistently distinguish between the capabilities of different models and fail to align with real-world user preferences. On the other hand, live crowd-sourced platforms like the Chatbot Arena collect a wide range of natural prompts and user feedback. However, these prompts vary in sophistication and the feedback cannot be applied offline to new models. In order to ensure that benchmarks keep up with the pace of LLM development, we address how one can evaluate benchmarks on their ability to confidently separate models and their alignment with human preference. Under these principles, we developed BenchBuilder, a living benchmark that filters high-quality prompts from live data sources to enable offline evaluation on fresh, challenging prompts. BenchBuilder identifies seven indicators of a high-quality prompt, such as the requirement for domain knowledge, and utilizes an LLM annotator to select a high-quality subset of prompts from various topic clusters. The LLM evaluation process employs an LLM judge to ensure a fully automated, high-quality, and constantly updating benchmark. We apply BenchBuilder on prompts from the Chatbot Arena to create Arena-Hard-Auto v0.1: 500 challenging user prompts from a wide range of tasks. Arena-Hard-Auto v0.1 offers 3x tighter confidence intervals than MT-Bench and achieves a state-of-the-art 89.1% agreement with human preference rankings, all at a cost of only $25 and without human labelers. The BenchBuilder pipeline enhances evaluation benchmarks and provides a valuable tool for developers, enabling them to extract high-quality benchmarks from extensive data with minimal effort.

The concept of persona, originally adopted in dialogue literature, has re-surged as a promising framework for tailoring large language models (LLMs) to specific context (e.g., personalized search, LLM-as-a-judge). However, the growing research on leveraging persona in LLMs is relatively disorganized and lacks a systematic taxonomy. To close the gap, we present a comprehensive survey to categorize the current state of the field. We identify two lines of research, namely (1) LLM Role-Playing, where personas are assigned to LLMs, and (2) LLM Personalization, where LLMs take care of user personas. Additionally, we introduce existing methods for LLM personality evaluation. To the best of our knowledge, we present the first survey for role-playing and personalization in LLMs under the unified view of persona. We continuously maintain a paper collection to foster future endeavors: https://github.com/MiuLab/PersonaLLM-Survey, Comment: 8-page version

Large Language Models (LLMs) require careful safety alignment to prevent malicious outputs. While significant research focuses on mitigating harmful content generation, the enhanced safety often come with the side effect of over-refusal, where LLMs may reject innocuous prompts and become less helpful. Although the issue of over-refusal has been empirically observed, a systematic measurement is challenging due to the difficulty of crafting prompts that appear harmful but are benign. This study proposes a novel method for automatically generating large-scale sets of "seemingly toxic prompts" (benign prompts likely rejected by LLMs). Leveraging this technique, we introduce OR-Bench, the first large-scale over-refusal benchmark. OR-Bench comprises 80,000 seemingly toxic prompts across 10 common rejection categories, a subset of around 1,000 hard prompts that are challenging even for state-of-the-art LLMs, and an additional 600 toxic prompts to prevent indiscriminate responses. We then conduct a comprehensive study to measure the over-refusal of 25 popular LLMs across 8 model families. Our datasets are available at https://huggingface.co/datasets/bench-llm/or-bench and the demo can be found at https://huggingface.co/spaces/bench-llm/or-bench. We hope this benchmark can help the community develop better safety aligned models., Comment: version 2, 10 pages main, 22 pages total

Large language models (LLMs) are increasingly integrated into many online services, yet they remain cost-prohibitive to deploy due to the requirement of expensive GPU instances. Prior work has addressed the high cost of LLM serving by improving the inference engine, but less attention has been given to selecting the most cost-efficient GPU type(s) for a specific LLM service. There is a large and growing landscape of GPU types and, within these options, higher cost does not always lead to increased performance. Instead, through a comprehensive investigation, we find that three key LLM service characteristics (request size, request rate, SLO) strongly influence GPU cost efficiency, and differing GPU types are most cost efficient for differing LLM service settings. As a result, the most cost-efficient allocation for a given service is typically a mix of heterogeneous GPU types. Based on this analysis, we introduce M\'elange, a GPU allocation framework that navigates these diverse LLM service characteristics and heterogeneous GPU option space to automatically and efficiently derive the minimal-cost GPU allocation for a given LLM service. We formulate the GPU allocation task as a cost-aware bin packing problem where GPUs are bins and items are slices of the service workload. Our formulation's constraints account for a service's unique characteristics, allowing M\'elange to be flexible to support diverse service settings and heterogeneity-aware to adapt the GPU allocation to a specific service. Compared to using only a single GPU type, M\'elange reduces deployment costs by up to 77% in conversational settings, 33% in document-based settings, and 51% in a mixed setting.

The evaluation of large language models (LLMs) has drawn substantial attention in the field recently. This work focuses on evaluating LLMs in a Chinese context, specifically, for Traditional Chinese which has been largely underrepresented in existing benchmarks. We present TMLU, a holistic evaluation suit tailored for assessing the advanced knowledge and reasoning capability in LLMs, under the context of Taiwanese Mandarin. TMLU consists of an array of 37 subjects across social science, STEM, humanities, Taiwan-specific content, and others, ranging from middle school to professional levels. In addition, we curate chain-of-thought-like few-shot explanations for each subject to facilitate the evaluation of complex reasoning skills. To establish a comprehensive baseline, we conduct extensive experiments and analysis on 24 advanced LLMs. The results suggest that Chinese open-weight models demonstrate inferior performance comparing to multilingual proprietary ones, and open-weight models tailored for Taiwanese Mandarin lag behind the Simplified-Chinese counterparts. The findings indicate great headrooms for improvement, and emphasize the goal of TMLU to foster the development of localized Taiwanese-Mandarin LLMs. We release the benchmark and evaluation scripts for the community to promote future research., Comment: Preprint. Under review

The long-range interaction can fundamentally alter properties in gapped topological phases such as emergent massive edge modes. However, recent research has shifted attention to topological nontrivial critical points or phases, and it is natural to explore how long-range interaction influences them. In this work, we investigate the topological behavior and phase transition of extended Kitaev chains with long-range interactions, which can be derived from the critical Ising model via the Jordan-Wigner transformation in the short-range limit. Specifically, we analytically find the critical edge modes at the critical point remain stable against long-range interaction. More importantly, we observe these critical edge modes remain massless even when long-range interactions become substantially strong. As a byproduct, we numerically find that the critical behavior of the long-range model belongs to the free Majorana fermion universality class, which is entirely different from the long-range universality class in usual long-range spin models. Our work could shed new light on the interplay between long-range interactions (frustrated) and the gapless topological phases of matter., Comment: 13 pages, 11 figures. Any comments or suggestions are welcome !

By constructing an exactly solvable spin model, we investigate the critical behaviors of transverse field Ising chains interpolated with cluster interactions, which exhibit various types of topologically distinct Ising critical points. Using fidelity susceptibility as an indicator, we establish the global phase diagram, including ferromagnetic, trivial paramagnetic, and symmetry-protected topological phases. Different types of critical points exist between these phases, encompassing both topologically trivial and non-trivial Ising critical points, as well as Gaussian critical points. Importantly, we demonstrate the existence of a Lifshitz transition between these topologically distinct Ising critical points, with central charge and critical exponents determined through finite-size scaling. This work serves as a valuable reference for further research on phase transitions within the gapless quantum phase of matter., Comment: 13 pages, 9 figures. Any comments or suggestions are welcome!

Large Language Models (LLMs) have unlocked new capabilities and applications; however, evaluating the alignment with human preferences still poses significant challenges. To address this issue, we introduce Chatbot Arena, an open platform for evaluating LLMs based on human preferences. Our methodology employs a pairwise comparison approach and leverages input from a diverse user base through crowdsourcing. The platform has been operational for several months, amassing over 240K votes. This paper describes the platform, analyzes the data we have collected so far, and explains the tried-and-true statistical methods we are using for efficient and accurate evaluation and ranking of models. We confirm that the crowdsourced questions are sufficiently diverse and discriminating and that the crowdsourced human votes are in good agreement with those of expert raters. These analyses collectively establish a robust foundation for the credibility of Chatbot Arena. Because of its unique value and openness, Chatbot Arena has emerged as one of the most referenced LLM leaderboards, widely cited by leading LLM developers and companies. Our demo is publicly available at \url{https://chat.lmsys.org}.

Affective computing, which aims to recognize, interpret, and understand human emotions, provides benefits in healthcare, such as improving patient care and enhancing doctor-patient communication. However, there is a noticeable absence of a comprehensive summary of recent advancements in affective computing for healthcare, which could pose difficulties for researchers entering this field. To address this, our paper aims to provide an extensive literature review of related studies published in the last five years. We begin by analyzing trends, benefits, and limitations of recent datasets and affective computing methods devised for healthcare. Subsequently, we highlight several healthcare application hotspots of current technologies that could be promising for real-world deployment. Through our analysis, we identify and discuss some ongoing challenges in the field as evidenced by the literature. Concluding with a thorough review, we further offer potential future research directions and hope our findings and insights could guide related researchers to make better contributions to the evolution of affective computing in healthcare.

We calculate the mass spectrum of the S-wave fully heavy tetraquark systems $ QQ\bar Q\bar Q~(Q=c,b) $ with both normal $ (J^{PC}=0^{++},1^{+-},2^{++}) $ and exotic $ (J^{PC}=0^{+-},1^{++},2^{+-}) $ C-parities using three different quark potential models (AL1, AP1, BGS). The exotic C-parity systems refer to the ones that cannot be composed of two S-wave ground heavy quarkonia. We incorporate the molecular dimeson and compact diquark-antidiquark spatial correlations simultaneously, thereby discerning the actual configurations of the states. We employ the Gaussian expansion method to solve the four-body Schr\"odinger equation, and the complex scaling method to identify the resonant states. The mass spectra in three different models qualitatively agree with each other. We obtain several resonant states with $ J^{PC} = 0^{++}, 1^{+-}, 2^{++}, 1^{++} $ in the mass region $(6.92,7.30)\, \mathrm{GeV}$, some of which are good candidates of the experimentally observed $X(6900)$ and $X(7200)$. We also obtain several exotic C-parity zero-width states with $ J^{PC}=0^{+-} $ and $ 2^{+-} $. These zero-width states have no corresponding S-wave diquarkonium threshold and can only decay strongly to final states with P-wave quarkonia. With the notation $T_{4Q,J(C)}(M)$, we deduce from the root mean square radii that the $ X(7200) $ candidates $ T_{4c,0(+)}(7173), T_{4c,2(+)}(7214) $ and the state $ T_{4c,1(-)}(7191) $ look like molecular states although most of the resonant and zero-width states are compact states., Comment: 16 pages, 6 figures, 10 tables. Version accepted by PRD

Large language models are increasingly trained on all the data ever produced by humans. Many have raised concerns about the trustworthiness of public benchmarks due to potential contamination in pre-training or fine-tuning datasets. While most data decontamination efforts apply string matching (e.g., n-gram overlap) to remove benchmark data, we show that these methods are insufficient, and simple variations of test data (e.g., paraphrasing, translation) can easily bypass these decontamination measures. Furthermore, we demonstrate that if such variation of test data is not eliminated, a 13B model can easily overfit a test benchmark and achieve drastically high performance, on par with GPT-4. We validate such observations in widely used benchmarks such as MMLU, GSK8k, and HumanEval. To address this growing risk, we propose a stronger LLM-based decontamination method and apply it to widely used pre-training and fine-tuning datasets, revealing significant previously unknown test overlap. For example, in pre-training sets such as RedPajama-Data-1T and StarCoder-Data, we identified that 8-18\% of the HumanEval benchmark overlaps. Interestingly, we also find such contamination in synthetic dataset generated by GPT-3.5/4, suggesting a potential risk of unintentional contamination. We urge the community to adopt stronger decontamination approaches when using public benchmarks. Moreover, we call for the community to actively develop fresh one-time exams to evaluate models accurately. Our decontamination tool is publicly available at https://github.com/lm-sys/llm-decontaminator.

We calculate the mass spectrum of the $Qs\bar q \bar q$ $(Q=c, b)$ tetraquark states with $J^P=(0,1,2)^+$ using the AL1 quark potential model, which successfully describes the conventional hadron spectrum. We employ the Gaussian expansion method to solve the four-body Schr\"odinger equation, and use the complex scaling method to identify the resonances. With the notation $T_{Q s, I(J)}^{\text {Theo. }}(M)$, we find several near-threshold bound states and resonances, including $T_{cs,0(0)}^{\mathrm{Theo.}}(2350)$, $T_{cs,0(0)}^{\mathrm{Theo.}}(2906)$, $T_{bs,0(0)}^{\mathrm{Theo.}}(5781)$, $T_{bs,0(1)}^{\mathrm{Theo.}}(5840)$, and $T_{bs,0(0)}^{\mathrm{Theo.}}(6240)$ which are close to the $D\bar{K}$, $D^*\bar{K}^*$, $\bar{B}\bar{K}$, $\bar{B}^*\bar{K}$ and $\bar{B}^*\bar{K}^*$ thresholds, respectively. Furthermore, their spatial structures clearly support their molecular natures. The resonance $T_{cs,0(0)}^{\mathrm{Theo.}}(2906)$ has a mass of $2906$ MeV, a width of $20$ MeV, and quantum numbers $I(J^P)=0(0^+)$, which may serve as a good candidate for the experimental $T_{cs0}(2900)$ state. We strongly urge the experimental search of the predicted states., Comment: 9 pages, 4 figures, 3 tables. Version accepted by PRD

In this paper, we address the hallucination problem commonly found in natural language generation tasks. Language models often generate fluent and convincing content but can lack consistency with the provided source, resulting in potential inaccuracies. We propose a new decoding method called Fidelity-Enriched Contrastive Search (FECS), which augments the contrastive search framework with context-aware regularization terms. FECS promotes tokens that are semantically similar to the provided source while penalizing repetitiveness in the generated text. We demonstrate its effectiveness across two tasks prone to hallucination: abstractive summarization and dialogue generation. Results show that FECS consistently enhances faithfulness across various language model sizes while maintaining output diversity comparable to well-performing decoding algorithms., Comment: Accepted as a short paper at EMNLP 2023

In this work, we study two different quantum simulators composed of molecules with dipole-dipole interaction through various theoretical and numerical tools. Our first result provides knowledge upon the quantum order by disorder effect of the $S=1/2$ system, which is programmable in a quantum simulator composed of circular Rydberg atoms in the triangular optical lattice with a controllable diagonal anisotropy. When the numbers of up spins and down spins are equal, a set of sub-extensive degenerate ground states is present in the classical limit, composed of continuous strings whose configuration enjoys a large degree of freedom. Adopting the the real space perturbation theory, our calculation demonstrates a lifting of the degeneracy, favoring the stripe configuration. When $J$ becomes larger, we adopt the infinite projected entangled-pair state~(iPEPS) and numerically check the effect of degeneracy lifting. The iPEPS results show that even when the spin exchange coupling is strong the stripe pattern is still favored. Next, we study the dipolar bosonic model with tilted polar angle which can be realized through a quantum simulator composed of cold atomic gas with dipole-dipole interaction in an optical lattice. By placing the atoms in a triangular lattice and tilting the polar angle, the diagonal anisotropy can also be realized in the bosonic system. With our cluster mean-field theory calculation, we provide various phase diagrams with different tilted angles, showing the abundant underlying phases including the supersolid. Our proposal indicates realizable scenarios through quantum simulators in studying the quantum effect as well as extraordinary phases. We believe that our results indicated here can also become a good benchmark for the two-dimensional quantum simulators., Comment: 15 pages, 6 figures, 2 tables

Studying how people interact with large language models (LLMs) in real-world scenarios is increasingly important due to their widespread use in various applications. In this paper, we introduce LMSYS-Chat-1M, a large-scale dataset containing one million real-world conversations with 25 state-of-the-art LLMs. This dataset is collected from 210K unique IP addresses in the wild on our Vicuna demo and Chatbot Arena website. We offer an overview of the dataset's content, including its curation process, basic statistics, and topic distribution, highlighting its diversity, originality, and scale. We demonstrate its versatility through four use cases: developing content moderation models that perform similarly to GPT-4, building a safety benchmark, training instruction-following models that perform similarly to Vicuna, and creating challenging benchmark questions. We believe that this dataset will serve as a valuable resource for understanding and advancing LLM capabilities. The dataset is publicly available at https://huggingface.co/datasets/lmsys/lmsys-chat-1m.

We explore the extension of chain-of-thought (CoT) prompting to medical reasoning for the task of automatic diagnosis. Motivated by doctors' underlying reasoning process, we present Diagnostic-Reasoning CoT (DR-CoT). Empirical results demonstrate that by simply prompting large language models trained only on general text corpus with two DR-CoT exemplars, the diagnostic accuracy improves by 15% comparing to standard prompting. Moreover, the gap reaches a pronounced 18% in out-domain settings. Our findings suggest expert-knowledge reasoning in large language models can be elicited through proper promptings., Comment: Accepted as a Tiny Paper at ICLR 2023 (10 pages, 5 figures)

Diagrammatic summation is a common bottleneck in modern applications of projected entangled-pair states, especially in computing low-energy excitations of a two-dimensional quantum many-body system. To solve this problem, here we extend the generating function approach for tensor network diagrammatic summation, a scheme previously proposed in the context of matrix product states. Taking the form of a one-particle excitation, we show that the excited state can be computed efficiently in the generating function formalism, which can further be used in evaluating the dynamical structure factor of the system. Our benchmark results for the spin-$1/2$ transverse-field Ising model and Heisenberg model on the square lattice provide a desirable accuracy, showing good agreement with known results. We then study the spin-$1/2$ $J_1$-$J_2$ model on the same lattice and investigate the dynamical properties of the putative gapless spin liquid phase. We conclude with a discussion on generalizations to multi-particle excitations., Comment: 13 pages, 6 figures

Evaluating large language model (LLM) based chat assistants is challenging due to their broad capabilities and the inadequacy of existing benchmarks in measuring human preferences. To address this, we explore using strong LLMs as judges to evaluate these models on more open-ended questions. We examine the usage and limitations of LLM-as-a-judge, including position, verbosity, and self-enhancement biases, as well as limited reasoning ability, and propose solutions to mitigate some of them. We then verify the agreement between LLM judges and human preferences by introducing two benchmarks: MT-bench, a multi-turn question set; and Chatbot Arena, a crowdsourced battle platform. Our results reveal that strong LLM judges like GPT-4 can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans. Hence, LLM-as-a-judge is a scalable and explainable way to approximate human preferences, which are otherwise very expensive to obtain. Additionally, we show our benchmark and traditional benchmarks complement each other by evaluating several variants of LLaMA and Vicuna. The MT-bench questions, 3K expert votes, and 30K conversations with human preferences are publicly available at https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judge., Comment: NeurIPS 2023 Datasets and Benchmarks Track