Search

Finetuning language models (LMs) is crucial for adapting the models to downstream data and tasks. However, full finetuning is usually costly. Existing work, such as parameter-efficient finetuning (PEFT), often focuses on \textit{how to finetune} but neglects the issue of \textit{where to finetune}. As a pioneering work on answering where to finetune (at the layer level), we conduct a semantic analysis of the LM inference process. We first propose a virtual transition of the latent representation and then trace its factual transition. Based on the deviation in transitions, we estimate the gain of finetuning each model layer, and further, narrow down the scope for finetuning. We perform extensive experiments across well-known LMs and datasets. The results show that our approach is effective and efficient, and outperforms the existing baselines. Our approach is orthogonal to existing efficient techniques, such as PEFT methods, offering practical values on LM finetuning., Comment: 13 pages, 5 figures, under peer-review

In the framework of the relativized quark model, the calculation of spin-orbit interactions is improved by considering the contribution from the light quark cluster in a singly heavy baryon. It enhances the energy level splitting of the orbital excitation and causes a partial removal of the energy level degeneracy and the emergence of fine structures for $\Sigma_{Q}$, $\Xi '_{Q}$ and $\Omega_{Q}$ baryons. Based on this improvement, we systematically analyze the fine structures and retest the heavy quark dominance mechanism. This mechanism is found to be violated in the $1P$-wave states of the $\Sigma_{c}$, $\Xi '_{c}$ and $\Omega_{c}$ baryons although it remains effective overall, which may help to understand the nature of the heavy quarks and strong interactions. With the predicted fine structures, we make the precise assignments of those observed heavy baryons which once could not be accurately explained due to their close mass values. The method used in this work is instructive and applicable for the study of more complex exotic hadron states, such as the heavy tetraquarks and pentaquarks., Comment: 7 pages, 2 figures, 2 tables

Understanding the latent space of language models (LMs) is important for improving the performance and interpretability of LMs. Existing analyses often fail to provide insights that take advantage of the semantic properties of language models and often overlook crucial aspects of language model adaptation. In response, we introduce a pioneering approach called vocabulary-defined semantics, which establishes a reference frame grounded in LM vocabulary within the LM latent space. We propose a novel technique to compute disentangled logits and gradients in latent space, not entangled ones on vocabulary. Further, we perform semantical clustering on data representations as a novel way of LM adaptation. Through extensive experiments across diverse text understanding datasets, our approach outperforms state-of-the-art methods of retrieval-augmented generation and parameter-efficient finetuning, showcasing its effectiveness and efficiency., Comment: under peer-review

Large Language Models (LLMs) have found widespread adoption in software engineering, particularly in code generation tasks. However, updating these models with new knowledge can be prohibitively expensive, yet it is essential for maximizing their utility. In this paper, we propose a novel and effective model editing approach, \textsc{MENT}, to patch LLMs in coding tasks. \textsc{MENT} is effective, efficient, and reliable. It can correct a neural model by patching 1 or 2 neurons. As the pioneer work on neuron-level model editing of generative models, we formalize the editing process and introduce the involved concepts. Besides, we also introduce new measures to evaluate its generalization ability, and build a benchmark for further study. Our approach is evaluated on three coding tasks, including API-seq recommendation, line-level code generation, and pseudocode-to-code transaction. The experimental results show that the proposed approach outperforms the state of the arts by a significant margin in both effectiveness and efficiency measures. In addition, we demonstrate the usages of \textsc{MENT} for LLM reasoning in software engineering. By editing LLM knowledge, the directly or indirectly dependent behaviors of API invocation in the chain-of-thought will change accordingly. It explained the significance of repairing LLMs., Comment: 12 pages, 6 figures, 6 tables, under peer-review

A mechanism of the heavy quark dominance in the orbital excitation is proposed in this paper which is testified to be reasonable for singly and doubly heavy baryons. In the relativistic quark model, an analysis of the Hamiltonian figures out the mechanism that the excitation mode with lower energy levels is always associated with the heavy quark(s), and the splitting of the energy levels is suppressed by the heavy quark(s). So, the heavy quarks dominate the orbital excitation of singly and doubly heavy baryons. Furthermore, a physical understanding of this mechanism is given in a semi-classical way. Accordingly, the predicted mass spectra of singly and doubly heavy baryons confirm the rationality of this mechanism. In addition, an interesting consequence of this mechanism is that a heavy-light meson is more likely to be produced in the strong decay of the high-orbital excited states, which is supported by experiments. This mechanism is rooted in the breakdown of the mass symmetry. Therefore, it may be also valid for other multi-quark systems, such as the tetraquarks Qqqq and QQqq, or the pentaquarks Qqqqq and QQqqq., Comment: 12 pages, 7 figures, 5 tables

In this paper, we investigate the mass spectra of bottom-charm baryons systematically, where the relativistic quark model and the infinitesimally shifted Gaussian basis function method are employed. Our calculation shows that the $\rho$-mode appears lower in energy than the other excited modes. According to this feature, the allowed quantum states are selected and a systematic study of the mass spectra for $\Xi_{bc}^{'}$ ($\Xi_{bc}$) and $\Omega_{bc}^{'}$ ($\Omega_{bc}$) families is performed. The root mean square radii and quark radial probability density distributions of these baryons are analyzed as well. Next, the Regge trajectories in the $(J,M^{2})$ plane are successfully constructed based on the mass spectra. At last, we present the structures of the mass spectra, and analyze the difficulty and opportunity in searching for the ground states of bottom-charm baryons in experiment., Comment: 19 pages, 9 figures, 6 tables. arXiv admin note: text overlap with arXiv:2210.13085

Based on the relativistic quark model and the infinitesimally shifted Gaussian basis function method, we investigate the mass spectra of double bottom baryons systematically. In the $\rho$-mode which appears lower in energy than the other excited modes, we obtain the allowed quantum states and perform a systematic study of the mass spectra of the $\Xi_{bb}$ and $\Omega_{bb}$ families. We analyze the root mean square radii and quark radial probability density distributions to deeply understand the structure of the heavy baryons. Meanwhile, the mass spectra allow us to successfully construct the Regge trajectories in the $(J,M^{2})$ plane. We also predict the masses of the ground states of double bottom baryons and discuss the differences between the structures of our spectra and those from other theoretical methods. At last, the shell structure of the double bottom baryon spectra is shown, from which one could get a bird's-eye view of the mass spectra., Comment: 16 pages, 9 figures, 5 tables. arXiv admin note: text overlap with arXiv:2207.04167

Deep code generation is a topic of deep learning for software engineering (DL4SE), which adopts neural models to generate code for the intended functions. Since end-to-end neural methods lack domain knowledge and software hierarchy awareness, they tend to perform poorly w.r.t project-level tasks. To systematically explore the potential improvements of code generation, we let it participate in the whole top-down development from \emph{expressibles} to \emph{executables}, which is possible in limited scopes. In the process, it benefits from massive samples, features, and knowledge. As the foundation, we suggest building a taxonomy on code data, namely code taxonomy, leveraging the categorization of code information. Moreover, we introduce a three-layer semantic pyramid (SP) to associate text data and code data. It identifies the information of different abstraction levels, and thus introduces the domain knowledge on development and reveals the hierarchy of software. Furthermore, we propose a semantic pyramid framework (SPF) as the approach, focusing on software of high modularity and low complexity. SPF divides the code generation process into stages and reserves spots for potential interactions. In addition, we conceived preliminary applications in software development to confirm the neuro-symbolic framework., Comment: 5 pages, 3 figures, 2 tables, accepted by ESEC/FSE 2023, the camera-ready version

Motivated by the experimental progress in the study of heavy baryons, we investigate the mass spectra of strange single heavy baryons in the $\lambda$-mode, where the relativistic quark model and the infinitesimally shifted Gaussian basis function method are employed. It is shown that the experimental data can be well reproduced by the predicted masses. The root mean square radii and radial probability density distributions of the wave functions are analyzed in detail. Meanwhile, the mass spectra allow us to successfully construct the Regge trajectories in the $(J,M^{2})$ plane. We also preliminarily assign quantum numbers to the recently observed baryons, including $\Xi_{c}(3055)$, $\Xi_{c}(3080)$, $\Xi_{c}(2930)$, $\Xi_{c}(2923)$, $\Xi_{c}(2939)$, $\Xi_{c}(2965)$, $\Xi_{c}(2970)$, $\Xi_{c}(3123)$, $\Xi_{b}(6100)$, $\Xi_{b}(6227)$, $\Xi_{b}(6327)$ and $\Xi_{b}(6333)$. At last, the spectral structure of the strange single heavy baryons is shown. Accordingly, we predict several new baryons that might be observed in forthcoming experiments., Comment: 27 pages, 11 figures, 8 tables

3D hand pose estimation from single depth is a fundamental problem in computer vision, and has wide applications.However, the existing methods still can not achieve satisfactory hand pose estimation results due to view variation and occlusion of human hand. In this paper, we propose a new virtual view selection and fusion module for 3D hand pose estimation from single depth.We propose to automatically select multiple virtual viewpoints for pose estimation and fuse the results of all and find this empirically delivers accurate and robust pose estimation. In order to select most effective virtual views for pose fusion, we evaluate the virtual views based on the confidence of virtual views using a light-weight network via network distillation. Experiments on three main benchmark datasets including NYU, ICVL and Hands2019 demonstrate that our method outperforms the state-of-the-arts on NYU and ICVL, and achieves very competitive performance on Hands2019-Task1, and our proposed virtual view selection and fusion module is both effective for 3D hand pose estimation., Comment: Accepted by AAAI2022

Automatic code summarization is beneficial to daily software development since it could help reduce the requirement of manual writing. Currently, artificial intelligence is undergoing a paradigm shift. The foundation models pretrained on massive data and finetuned to downstream tasks surpass specially customized models. This trend inspired us to consider reusing foundation models instead of learning from scratch. Thereby, we propose a flexible and robust approach for automatic code summarization, based on neural models. We assemble available foundation models, such as CodeBERT and GPT-2, into a single neural model named AdaMo. Moreover, we utilize Gaussian noise as the simulation of contextual information to optimize the latent representation. Furthermore, we introduce two adaptive schemes from the perspective of knowledge transfer, namely continuous pretraining and intermediate finetuning, and design intermediate stage tasks for general sequence-to-sequence learning. Finally, we evaluate AdaMo against a benchmark dataset for code summarization, by comparing it with state-of-the-art models., Comment: 12 pages, 2 figures, 8 tables, accepted by SANER 2022, the camera-ready version

The Transformer architecture and transfer learning have marked a quantum leap in natural language processing, improving the state of the art across a range of text-based tasks. This paper examines how these advancements can be applied to and improve code search. To this end, we pre-train a BERT-based model on combinations of natural language and source code data and fine-tune it on pairs of StackOverflow question titles and code answers. Our results show that the pre-trained models consistently outperform the models that were not pre-trained. In cases where the model was pre-trained on natural language "and" source code data, it also outperforms an information retrieval baseline based on Lucene. Also, we demonstrated that the combined use of an information retrieval-based approach followed by a Transformer leads to the best results overall, especially when searching into a large search pool. Transfer learning is particularly effective when much pre-training data is available and fine-tuning data is limited. We demonstrate that natural language processing models based on the Transformer architecture can be directly applied to source code analysis tasks, such as code search. With the development of Transformer models designed more specifically for dealing with source code data, we believe the results of source code analysis tasks can be further improved., Comment: Accepted for publication in the IEEE Transactions on Software Engineering (TSE)

Semantic code search is about finding semantically relevant code snippets for a given natural language query. In the state-of-the-art approaches, the semantic similarity between code and query is quantified as the distance of their representation in the shared vector space. In this paper, to improve the vector space, we introduce tree-serialization methods on a simplified form of AST and build the multimodal representation for the code data. We conduct extensive experiments using a single corpus that is large-scale and multi-language: CodeSearchNet. Our results show that both our tree-serialized representations and multimodal learning model improve the performance of code search. Last, we define intuitive quantification metrics oriented to the completeness of semantic and syntactic information of the code data, to help understand the experimental findings., Comment: 12 pages, 9 figures, 7 tables, accepted by ICSME 2021, the camera-ready version