Search

Lossless speculative decoding accelerates target large language model (LLM) inference by employing a lightweight draft model for generating tree-structured candidates, which are subsequently verified in parallel by the target LLM. Currently, effective approaches leverage feature-level rather than token-level autoregression within the draft model to facilitate more straightforward predictions and enhanced knowledge distillation. In this paper, we reassess these approaches and propose FSPAD (Feature Sampling and Partial Alignment Distillation for Lossless Speculative Decoding), which introduces two straightforward and effective components within the existing framework to boost lossless speculative decoding. Firstly, FSPAD utilizes token embeddings to sample features of the target LLM in high-dimensional space before feeding them into the draft model, due to the inherent uncertainty of the features preventing the draft model from obtaining the specific token output by the target LLM. Secondly, FSPAD introduces partial alignment distillation to weaken the draft model's connection between features and logits, aiming to reduce the conflict between feature alignment and logit confidence during training. Our experiments include both greedy and non-greedy decoding on the largest and smallest models from the Vicuna and LLaMA3-Instruct series, as well as tasks in multi-turn conversation, translation, summarization, question answering, mathematical reasoning, and retrieval-augmented generation. The results show that FSPAD outperforms the state-of-the-art method across all the aforementioned tasks and target LLMs., Comment: The work was not submitted to AAAI 2025

The general capabilities of Large Language Models (LLM) highly rely on the composition and selection on extensive pretraining datasets, treated as commercial secrets by several institutions. To mitigate this issue, we open-source the details of a universally applicable data processing pipeline and validate its effectiveness and potential by introducing a competitive LLM baseline. Specifically, the data processing pipeline consists of broad collection to scale up and reweighting to improve quality. We then pretrain a 7B model BaichuanSEED with 3T tokens processed by our pipeline without any deliberate downstream task-related optimization, followed by an easy but effective supervised fine-tuning stage. BaichuanSEED demonstrates consistency and predictability throughout training and achieves comparable performance on comprehensive benchmarks with several commercial advanced large language models, such as Qwen1.5 and Llama3. We also conduct several heuristic experiments to discuss the potential for further optimization of downstream tasks, such as mathematics and coding., Comment: 19 pages, 6 figures

Large Language Models (LLMs) have become instrumental across various applications, with the customization of these models to specific scenarios becoming increasingly critical. System message, a fundamental component of LLMs, is consist of carefully crafted instructions that guide the behavior of model to meet intended goals. Despite the recognized potential of system messages to optimize AI-driven solutions, there is a notable absence of a comprehensive benchmark for evaluating how well different LLMs follow these system messages. To fill this gap, we introduce SysBench, a benchmark that systematically analyzes system message following ability in terms of three challenging aspects: constraint complexity, instruction misalignment and multi-turn stability. In order to enable effective evaluation, SysBench constructs multi-turn user conversations covering various interaction relationships, based on six common types of constraints from system messages in real-world scenarios. Our dataset contains 500 system messages from various domains, each paired with 5 turns of user conversations, which have been manually formulated and checked to guarantee high quality. SysBench provides extensive evaluation across various LLMs, measuring their ability to follow specified constraints given in system messages. The results highlight both the strengths and weaknesses of existing models, offering key insights and directions for future research. The open source library SysBench is available at https://github.com/PKU-Baichuan-MLSystemLab/SysBench.

With the development of Multimodal Large Language Models (MLLMs), the evaluation of multimodal models in the context of mathematical problems has become a valuable research field. Multimodal visual-textual mathematical reasoning serves as a critical indicator for evaluating the comprehension and complex multi-step quantitative reasoning abilities of MLLMs. However, previous multimodal math benchmarks have not sufficiently integrated visual and textual information. To address this gap, we proposed MathScape, a new benchmark that emphasizes the understanding and application of combined visual and textual information. MathScape is designed to evaluate photo-based math problem scenarios, assessing the theoretical understanding and application ability of MLLMs through a categorical hierarchical approach. We conduct a multi-dimensional evaluation on 11 advanced MLLMs, revealing that our benchmark is challenging even for the most sophisticated models. By analyzing the evaluation results, we identify the limitations of MLLMs, offering valuable insights for enhancing model performance.

The adeptness of Large Language Models (LLMs) in comprehending and following natural language instructions is critical for their deployment in sophisticated real-world applications. Existing evaluations mainly focus on fragmented constraints or narrow scenarios, but they overlook the comprehensiveness and authenticity of constraints from the user's perspective. To bridge this gap, we propose CFBench, a large-scale Comprehensive Constraints Following Benchmark for LLMs, featuring 1,000 curated samples that cover more than 200 real-life scenarios and over 50 NLP tasks. CFBench meticulously compiles constraints from real-world instructions and constructs an innovative systematic framework for constraint types, which includes 10 primary categories and over 25 subcategories, and ensures each constraint is seamlessly integrated within the instructions. To make certain that the evaluation of LLM outputs aligns with user perceptions, we propose an advanced methodology that integrates multi-dimensional assessment criteria with requirement prioritization, covering various perspectives of constraints, instructions, and requirement fulfillment. Evaluating current leading LLMs on CFBench reveals substantial room for improvement in constraints following, and we further investigate influencing factors and enhancement strategies. The data and code are publicly available at https://github.com/PKU-Baichuan-MLSystemLab/CFBench, Comment: 15 pages, 10 figures

Large Language Models (LLMs) frequently suffer from inefficiencies, largely attributable to the discord between the requirements of auto-regressive decoding and the architecture of contemporary GPUs. Recently, regressive lightweight speculative decoding has garnered attention for its notable efficiency improvements in text generation tasks. This approach utilizes a lightweight regressive draft model, like a Recurrent Neural Network (RNN) or a single transformer decoder layer, leveraging sequential information to iteratively predict potential tokens. Specifically, RNN draft models are computationally economical but tend to deliver lower accuracy, while attention decoder layer models exhibit the opposite traits. This paper presents Clover-2, an advanced iteration of Clover, an RNN-based draft model designed to achieve comparable accuracy to that of attention decoder layer models while maintaining minimal computational overhead. Clover-2 enhances the model architecture and incorporates knowledge distillation to increase Clover's accuracy and improve overall efficiency. We conducted experiments using the open-source Vicuna 7B and LLaMA3-Instruct 8B models. The results demonstrate that Clover-2 surpasses existing methods across various model architectures, showcasing its efficacy and robustness.

In recent years, the rise of Large Language Models (LLMs) has spurred a growing demand for plug-and-play AI systems. Among the various AI techniques, prompt engineering stands out as particularly significant. However, users often face challenges in writing prompts due to the steep learning curve and significant time investment, and existing automatic prompt engineering (APE) models can be difficult to use. To address this issue, we propose PAS, an LLM-based plug-and-play APE system. PAS utilizes LLMs trained on high-quality, automatically generated prompt complementary datasets, resulting in exceptional performance. In comprehensive benchmarks, PAS achieves state-of-the-art (SoTA) results compared to previous APE models, with an average improvement of 6.09 points. Moreover, PAS is highly efficient, achieving SoTA performance with only 9000 data points. Additionally, PAS can autonomously generate prompt augmentation data without requiring additional human labor. Its flexibility also allows it to be compatible with all existing LLMs and applicable to a wide range of tasks. PAS excels in human evaluations, underscoring its suitability as a plug-in for users. This combination of high performance, efficiency, and flexibility makes PAS a valuable system for enhancing the usability and effectiveness of LLMs through improved prompt engineering.

As the field of Large Language Models (LLMs) continues to evolve, the context length in inference is steadily growing. Key-Value Cache (KVCache), a crucial component in LLM inference, has now become the primary memory bottleneck due to limited GPU memory. Current methods selectively determine suitable keys and values for self-attention computation in LLMs to address the issue. However, they either fall short in maintaining model quality or result in high serving latency. Drawing inspiration from advanced embedding retrieval techniques used in the database community, we consider the storage and searching of KVCache as a typical embedding retrieval problem. We propose PQCache, which employs Product Quantization (PQ) to manage KVCache, maintaining model quality while ensuring low serving latency. During the prefilling phase, we apply PQ to tokens' keys for each LLM layer and head. During the autoregressive decoding phase, for each newly generated token, we first identify important tokens through Maximum Inner-Product Search (MIPS) using PQ codes and centroids, then fetch the corresponding key-value pairs for self-attention computation. Through meticulous design of overlapping and caching, we minimize any additional computation and communication overhead during both phases. Extensive experiments show that PQCache achieves both effectiveness and efficiency. It maintains model quality even when only 1/5 of the tokens are involved in attention, while attaining acceptable system latency.

With the rapid development of video Multimodal Large Language Models (MLLMs), numerous benchmarks have been proposed to assess their video understanding capability. However, due to the lack of rich events in the videos, these datasets may suffer from the short-cut bias that the answers can be deduced from a few frames, without the need to watch the entire video. To address this issue, we introduce Event-Bench, an event-oriented long video understanding benchmark built on existing datasets and human annotations. Event-Bench includes six event-related tasks and 2,190 test instances to comprehensively evaluate video event understanding ability. Additionally, we propose Video Instruction Merging~(VIM), a cost-effective method that enhances video MLLMs using merged, event-intensive video instructions, addressing the scarcity of human-annotated, event-intensive data. Extensive experiments show that the best-performing model, GPT-4o, achieves an overall accuracy of 53.33, significantly outperforming the best open-source model by 41.42%. Leveraging an effective instruction synthesis method and an adaptive model architecture, VIM surpasses both state-of-the-art open-source models and GPT-4V on the Event-Bench. All code, data, and models are publicly available at https://github.com/RUCAIBox/Event-Bench., Comment: Work on progress

Synthetic data has been proposed as a solution to address the issue of high-quality data scarcity in the training of large language models (LLMs). Studies have shown that synthetic data can effectively improve the performance of LLMs on downstream benchmarks. However, despite its potential benefits, our analysis suggests that there may be inherent flaws in synthetic data. The uniform format of synthetic data can lead to pattern overfitting and cause significant shifts in the output distribution, thereby reducing the model's instruction-following capabilities. Our work delves into these specific flaws associated with question-answer (Q-A) pairs, a prevalent type of synthetic data, and presents a method based on unlearning techniques to mitigate these flaws. The empirical results demonstrate the effectiveness of our approach, which can reverse the instruction-following issues caused by pattern overfitting without compromising performance on benchmarks at relatively low cost. Our work has yielded key insights into the effective use of synthetic data, aiming to promote more robust and efficient LLM training., Comment: 15 pages

Retrieval-augmented generation (RAG) effectively addresses issues of static knowledge and hallucination in large language models. Existing studies mostly focus on question scenarios with clear user intents and concise answers. However, it is prevalent that users issue broad, open-ended queries with diverse sub-intents, for which they desire rich and long-form answers covering multiple relevant aspects. To tackle this important yet underexplored problem, we propose a novel RAG framework, namely RichRAG. It includes a sub-aspect explorer to identify potential sub-aspects of input questions, a multi-faceted retriever to build a candidate pool of diverse external documents related to these sub-aspects, and a generative list-wise ranker, which is a key module to provide the top-k most valuable documents for the final generator. These ranked documents sufficiently cover various query aspects and are aware of the generator's preferences, hence incentivizing it to produce rich and comprehensive responses for users. The training of our ranker involves a supervised fine-tuning stage to ensure the basic coverage of documents, and a reinforcement learning stage to align downstream LLM's preferences to the ranking of documents. Experimental results on two publicly available datasets prove that our framework effectively and efficiently provides comprehensive and satisfying responses to users.

The advent of large language models (LLMs) like GPT-4 has catalyzed the exploration of multi-task learning (MTL), in which a single model demonstrates proficiency across diverse tasks. Task arithmetic has emerged as a cost-effective approach for MTL. It enables performance enhancement across multiple tasks by adding their corresponding task vectors to a pre-trained model. However, the current lack of a method that can simultaneously achieve optimal performance, computational efficiency, and data privacy limits their application to LLMs. In this paper, we propose \textbf{M}odel \textbf{E}xclusive \textbf{T}ask \textbf{A}rithmetic for merging \textbf{GPT}-scale models, which formalizes the objective of model merging into a multi-task learning framework, aiming to minimize the average loss difference between the merged model and each individual task model. Since data privacy limits the use of multi-task training data, we leverage LLMs' local linearity and task vectors' orthogonality to separate the data term and scaling coefficients term and derive a model-exclusive task arithmetic method. Our proposed MetaGPT is data-agnostic and bypasses the heavy search process, making it cost-effective and easy to implement for LLMs.Extensive experiments demonstrate that MetaGPT leads to improvements in task arithmetic and achieves state-of-the-art performance on multiple tasks., Comment: 19 pages

Deep Neural Networks (DNNs) excel in various domains but face challenges in providing accurate uncertainty estimates, which are crucial for high-stakes applications. Large Language Models (LLMs) have recently emerged as powerful tools, demonstrating exceptional performance in language tasks. However, traditional calibration metrics such as Expected Calibration Error (ECE) and classwise-ECE (cw-ECE) are inadequate for LLMs due to their vast vocabularies, data complexity, and distributional focus. To address this, we propose a novel calibration concept called full calibration and introduce its corresponding metric, Full-ECE. Full-ECE evaluates the entire predicted probability distribution, offering a more accurate and robust measure of calibration for LLMs.

Video understanding is a crucial next step for multimodal large language models (MLLMs). To probe specific aspects of video understanding ability, existing video benchmarks typically require careful video selection based on the target capability, along with laborious annotation of query-response pairs to match the specific video content. This process is both challenging and resource-intensive. In this paper, we propose VideoNIAH (Video Needle In A Haystack), a benchmark construction framework through synthetic video generation. VideoNIAH decouples test video content from their query-responses by inserting unrelated image/text 'needles' into original videos. It generates annotations solely from these needles, ensuring diversity in video sources and a variety of query-responses. Additionally, by inserting multiple needles, VideoNIAH rigorously evaluates the temporal understanding capabilities of models. We utilized VideoNIAH to compile a video benchmark VNBench, including tasks such as retrieval, ordering, and counting. VNBench can efficiently evaluate the fine-grained understanding ability and spatio-temporal modeling ability of a video model, while also supporting the long-context evaluation. Additionally, we evaluated recent video-centric multimodal large language models (MLLMs), both open-source and proprietary, providing a comprehensive analysis. We found that although proprietary models have significant advantages over open-source models, all existing video models still perform poorly on long-distance dependency tasks. VideoNIAH is a simple yet highly scalable benchmark construction framework, and we believe it will inspire future video benchmark works. The code and data are available at https://github.com/joez17/VideoNIAH.

Transformer-based large language models (LLMs) typically have a limited context window, resulting in significant performance degradation when processing text beyond the length of the context window. Extensive studies have been proposed to extend the context window and achieve length extrapolation of LLMs, but there is still a lack of in-depth interpretation of these approaches. In this study, we explore the positional information within and beyond the context window for deciphering the underlying mechanism of LLMs. By using a mean-based decomposition method, we disentangle positional vectors from hidden states of LLMs and analyze their formation and effect on attention. Furthermore, when texts exceed the context window, we analyze the change of positional vectors in two settings, i.e., direct extrapolation and context window extension. Based on our findings, we design two training-free context window extension methods, positional vector replacement and attention window extension. Experimental results show that our methods can effectively extend the context window length.

Position embedding is a core component of current Large Language Models (LLMs). Rotary position embedding (RoPE), a technique that encodes the position information with a rotation matrix, has been the de facto choice for position embedding in many LLMs, such as the Llama series. RoPE has been further utilized to extend long context capability, which is roughly based on adjusting the \textit{base} parameter of RoPE to mitigate out-of-distribution (OOD) problems in position embedding. However, in this paper, we find that LLMs may obtain a superficial long-context ability based on the OOD theory. We revisit the role of RoPE in LLMs and propose a novel property of long-term decay, we derive that the \textit{base of RoPE bounds context length}: there is an absolute lower bound for the base value to obtain certain context length capability. Our work reveals the relationship between context length and RoPE base both theoretically and empirically, which may shed light on future long context training., Comment: 17 pages

Large language models (LLMs) suffer from low efficiency as the mismatch between the requirement of auto-regressive decoding and the design of most contemporary GPUs. Specifically, billions to trillions of parameters must be loaded to the GPU cache through its limited memory bandwidth for computation, but only a small batch of tokens is actually computed. Consequently, the GPU spends most of its time on memory transfer instead of computation. Recently, parallel decoding, a type of speculative decoding algorithms, is becoming more popular and has demonstrated impressive efficiency improvement in generation. It introduces extra decoding heads to large models, enabling them to predict multiple subsequent tokens simultaneously and verify these candidate continuations in a single decoding step. However, this approach deviates from the training objective of next token prediction used during pre-training, resulting in a low hit rate for candidate tokens. In this paper, we propose a new speculative decoding algorithm, Clover, which integrates sequential knowledge into the parallel decoding process. This enhancement improves the hit rate of speculators and thus boosts the overall efficiency. Clover transmits the sequential knowledge from pre-speculated tokens via the Regressive Connection, then employs an Attention Decoder to integrate these speculated tokens. Additionally, Clover incorporates an Augmenting Block that modifies the hidden states to better align with the purpose of speculative generation rather than next token prediction. The experiment results demonstrate that Clover outperforms the baseline by up to 91% on Baichuan-Small and 146% on Baichuan-Large, respectively, and exceeds the performance of the previously top-performing method, Medusa, by up to 37% on Baichuan-Small and 57% on Baichuan-Large, respectively.

The rapid proliferation of large language models (LLMs) such as GPT-4 and Gemini underscores the intense demand for resources during their training processes, posing significant challenges due to substantial computational and environmental costs. To alleviate this issue, we propose checkpoint merging in pretraining LLM. This method utilizes LLM checkpoints with shared training trajectories, and is rooted in an extensive search space exploration for the best merging weight via Bayesian optimization. Through various experiments, we demonstrate that: (1) Our proposed methodology exhibits the capacity to augment pretraining, presenting an opportunity akin to obtaining substantial benefits at minimal cost; (2) Our proposed methodology, despite requiring a given held-out dataset, still demonstrates robust generalization capabilities across diverse domains, a pivotal aspect in pretraining.

As Large Language Models (LLMs) continue to advance in performance, their size has escalated significantly, with current LLMs containing billions or even trillions of parameters. However, in this study, we discovered that many layers of LLMs exhibit high similarity, and some layers play a negligible role in network functionality. Based on this observation, we define a metric called Block Influence (BI) to gauge the significance of each layer in LLMs. We then propose a straightforward pruning approach: layer removal, in which we directly delete the redundant layers in LLMs based on their BI scores. Experiments demonstrate that our method, which we call ShortGPT, significantly outperforms previous state-of-the-art (SOTA) methods in model pruning. Moreover, ShortGPT is orthogonal to quantization-like methods, enabling further reduction in parameters and computation. The ability to achieve better results through simple layer removal, as opposed to more complex pruning techniques, suggests a high degree of redundancy in the model architecture.

The polar Kerr effect and the closely related anomalous charge Hall effect are among the most distinguishing signatures of the superconducting state in Sr$_2$RuO$_4$, as well as in several other compounds. These effects are often thought to be derived from chiral superconducting pairing, and different mechanisms have been invoked for the explanation. However, the intrinsic mechanisms proposed previously often involve unrealistically strong interband Cooper pairing. In this study we show that, even without interband pairing, nonunitary superconducting states can support intrinsic anomalous charge Hall effect, thanks to the quantum geometric properties of the Bloch electrons. The key here is to have a normal-state spin Hall effect, for which a nonzero spin-orbit coupling is essential. A finite charge Hall effect then naturally arises at the onset of a spin-polarized nonunitary superconducting pairing. It depends on both the spin polarization and the normal-state electron Berry curvature, the latter of which is the imaginary part of the quantum geometric tensor of the Bloch states. Applying our results to the weakly-paired Sr$_2$RuO$_4$ we conclude that, if the reported Kerr effect is of intrinsic origin, the superconducting state is most likely nonunitary and has odd-parity. Our theory may be generalized to other superconductors that exhibit polar Kerr effect., Comment: 7+7 pages

Large language models (LLMs) have demonstrated remarkable performance on a variety of natural language tasks based on just a few examples of natural language instructions, reducing the need for extensive feature engineering. However, most powerful LLMs are closed-source or limited in their capability for languages other than English. In this technical report, we present Baichuan 2, a series of large-scale multilingual language models containing 7 billion and 13 billion parameters, trained from scratch, on 2.6 trillion tokens. Baichuan 2 matches or outperforms other open-source models of similar size on public benchmarks like MMLU, CMMLU, GSM8K, and HumanEval. Furthermore, Baichuan 2 excels in vertical domains such as medicine and law. We will release all pre-training model checkpoints to benefit the research community in better understanding the training dynamics of Baichuan 2., Comment: Baichuan 2 technical report. Github: https://github.com/baichuan-inc/Baichuan2

One approach to probe the still controversial superconductivity in Sr$_2$RuO$_4$ is to apply external perturbations that break the underlying tetragonal crystalline symmetry. Chiral $p_x+ip_y$ and $d_{xz}+id_{yz}$ states respond to such perturbations in ways that may help to distinguish them from other superconducting pairings. However, past experimental efforts along this line, using uniaxial strains and magnetic fields parallel to the RuO$_2$ plane, have not been able to reach an unambiguous conclusion. In this study, we propose to further examine the possibility of chiral superconducting order in Sr$_2$RuO$_4$ using an alternative tetragonal-symmetry-breaking perturbation -- in-plane supercurrent. We study the superconducting phase diagram as a function of both temperature and the applied supercurrent. Supercurrent generically splits the transition of the two chiral order parameter components, and we show that the splitting can give rise to visible specific heat anomalies. Furthermore, supercurrent parallel and anti-parallel to the unidirectional propagation of the chiral edge modes impact the edge states in different manner. This difference manifests in the tunneling spectrum, thereby providing an additional means to probe the chirality even when the related spontaneous edge current is vanishingly small. Finally, we discuss the distinction of supercurrent responses in non-chiral time-reversal-symmetry-breaking superconducting states. Our proposal can be applied to other candidate chiral superconductors., Comment: 7 pages, 7 figures

In the absence of time-reversal invariance, metals without inversion symmetry may exhibit nonreciprocal charge transport -- a magnetochiral anisotropy that manifests as unequal electrical resistance for opposite current flow directions. If superconductivity also sets in, the charge transmission may become dissipationless in one direction while remaining dissipative in the opposite, thereby realizing a superconducting diode. Through both direct-current and alternating-current measurements, we study the nonreciprocal effects in thin films of the noncentrosymmetric superconductor T$_d$-MoTe\textsubscript{2} with disorders. We observe nonreciprocal superconducting critical currents with a diode efficiency close to 20\%~, and a large magnetochiral anisotropy coefficient up to $\SI{5.9e8}{\per\tesla\per\ampere}$, under weak out-of-plane magnetic field in the millitesla range. The great enhancement of rectification efficiency under out-of-plane magnetic field is likely abscribed to the vortex ratchet effect, which naturally appears in the noncentrosymmetric superconductor with disorders. Intriguingly, unlike the finding in Rashba systems, the strongest in-plane nonreciprocal effect does not occur when the field is perpendicular to the current flow direction. We develop a phenomenological theory to demonstrate that this peculiar behavior can be attributed to the asymmetric structure of spin-orbit coupling in T$_d$-MoTe\textsubscript{2}. Our study highlights how the crystallographic symmetry critically impacts the nonreciprocal transport, and would further advance the research for designing the superconducting diode with the best performance., Comment: 7 pages, 5figures

The anomalous Hall effect and the closely related polar Kerr effect are among the most direct evidence of chiral Cooper pairing in some superconductors. While it has been known that disorder or multiband pairing is typically needed for these effects to manifest, there is a lack of direct real-space investigation with regard to how disorder impacts the Hall response in both single-band and multiband chiral superconductors. On the basis of chiral superconducting models often adopted for \SRO, we study in this work the anomalous Hall effect in the presence of random non-magnetic impurities on real-space lattices. The single-band chiral p-wave ($p_x+ip_y$) calculation qualitatively reproduces the Hall conductivity obtained in previous skew-scattering-type diagrammatic analyses, along with some quantitative difference originating primarily from contributions involving impurity-induced in-gap states. The non-p-wave chiral states, such as $d_{x^2-y^2}+id_{xy}$, generically exhibit finite Hall response in the presence of random impurities, in contrast to a conclusion drawn from the aforementioned diagrammatic study. In particular, while pointlike impurities appears to induce minuscule Hall conductivity in non-self-consistent calculations, self-consistency and finite-range impurity potentials can both lead to substantial Hall conductivity. On the other hand, the intrinsic Hall conductivity in multiband chiral superconductors, which is related to interband transitions, decreases parametrically as disorder suppresses the superconducting order parameter. In addition, we check that random impurities do not induce anomalous Hall effect in non-chiral but time-reversal symmetry breaking superconducting states the likes of $s+i d_{x^2-y^2}$ and $d_{x^2-y^2}+i g_{xy(x^2-y^2)}$. We briefly remark on the implications of our results for Kerr effect measurements., Comment: 9 pages, 6 figures. Minor revision. Version accepted by PRB

Bloch electrons in multiorbital systems carry quantum geometric information characteristic of their wavevector-dependent interorbital mixing. The geometric nature impacts electromagnetic responses, and this effect carries over to the superconducting state, which receives a geometric contribution to the superfluid weight. In this paper, we show that this contribution could become negative under certain appropriate circumstances. This may facilitate the stabilization of Cooper pairings with real space phase modulation, i.e. the pair density wave order, as we demonstrate through two-orbital model Bogoliubov de-Gennes mean-field calculations. The quantum geometric effect therefore constitutes an intrinsic mechanism for the formation of such a novel phase of matter in the absence of external magnetic field., Comment: 5+2 pages, 2 figures. Version published by Science China: Physics, Mechanics and Astronomy

Bloch electrons in multiorbital systems carry nontrivial quantum geometric information characteristic of their orbital composition as a function of their wave vector. When such electrons form Cooper pairs, the resultant superconducting state naturally inherits aspects of the quantum geometry. In this paper, we study how this geometric character is revealed in the intrinsic optical response of the superconducting state. In particular, due to the superconducting gap opening, interband optical transitions involving states around the Fermi level are forbidden. This generally leads to an anomalous suppression of the optical conductivity at frequencies matching the band separation -- which could be significantly higher than the superconducting gap energy. We discuss how the predicted anomaly may have already emerged in two earlier measurements on an iron-based superconductor. When interband Cooper pairing is present, intraband optical transitions may be allowed and finite conductivity emerges at low frequencies right above the gap edge. These conductivities depend crucially on the off-diagonal elements of the velocity matrix in the band representation, i.e. the interband velocity -- which is related to the non-Abelian Berry connection of the Bloch states., Comment: 5 pages plus 1 supplemental section, 4 figures, 1 table

With the development of deep learning techniques and large scale datasets, the question answering (QA) systems have been quickly improved, providing more accurate and satisfying answers. However, current QA systems either focus on the sentence-level answer, i.e., answer selection, or phrase-level answer, i.e., machine reading comprehension. How to produce compositional answers has not been throughout investigated. In compositional question answering, the systems should assemble several supporting evidence from the document to generate the final answer, which is more difficult than sentence-level or phrase-level QA. In this paper, we present a large-scale compositional question answering dataset containing more than 120k human-labeled questions. The answer in this dataset is composed of discontiguous sentences in the corresponding document. To tackle the ComQA problem, we proposed a hierarchical graph neural networks, which represents the document from the low-level word to the high-level sentence. We also devise a question selection and node selection task for pre-training. Our proposed model achieves a significant improvement over previous machine reading comprehension methods and pre-training methods. Codes and dataset can be found at \url{https://github.com/benywon/ComQA}., Comment: Accepted by WWW2021

We study the pairing symmetry of Sr$_2$RuO$_4$ through the group-theoretical approach. We emphasize the role of pairing interaction between the quasi-one-dimensional(Q1D) $d_{xz/yz}$ and quasi-two-dimensional(Q2D) $d_{xy}$ orbitals. It is found that two degenerate inter-orbital time-reversal-invariant(TRI) p-wave pairings, one is spin-singlet and the other spin-triplet with out-of-plane $\bm{d}$-vector, could be the most promising candidates. Several important physical quantities are presented, including the near-nodal gap structure, the unchanged out-of-plane Knight shift, and no split transition under strain, which are consistent with the experiments. In addition, these p-wave pairings shed light on resolving the contradiction between the time-reversal breaking and reduced in-plane Knight shift measurements. As the system reaches the Van Hove singularity under applied strain, the pairing symmetry would become a d-wave pairing mainly consisting of inter-orbital components, which could be responsible for the strained 3$K$ phase., Comment: 11 pages, 5 figures

We study transport properties of topological Weyl nodal-line semimetals(NLSs). Starting from a minimal lattice model with a single nodal loop, and by focusing on a normal-metal-NLS-normal-metal junction, we investigate the dependence of the novel transport behavior on the orientation of the nodal loop. When the loop is parallel to the junction interfaces, the transmitted current is found to be nearly fully spin-polarized. Correspondingly, there exists a spin orientation, along which the incident electrons would be totally reflected. An unusual resonance of half transmission with the participation of surface states also occurs for a pair of incident electrons with opposite spin orientations. All these phenomena have been shown to originate from the existence of a single forward-propagating mode in the NLS of the junction, and argued to survive in more generic multi-band Weyl NLSs., Comment: 12 pages, 11 figures

We study the possibility of topological superconductivity in the noncentrosymmetric monolayer and trilayer TaS$_2$ with out-of-plane mirror symmetry. A gapless time-reversal invariant f+s-wave pairing state with even mirror parity is found to be a promising candidate. This mixing state holds 12(36) nodes at the Fermi pockets around $\Gamma$ for monolayer(trilayer) case and its unconventional superconductivity is consistent with the STM experiments observed in 2$H$-TaS$_2$ thin flakes. Furthermore, with doping or under uniaxial pressure for trilayer 2$H$-TaS$_2$, large-Chern-number time-reversal symmetry breaking mixing states between d+id- and p-ip-wave pairings can be realized in the phase diagram., Comment: 11 pages, 5 figures