Search

Finetuning language models (LMs) is crucial for adapting the models to downstream data and tasks. However, full finetuning is usually costly. Existing work, such as parameter-efficient finetuning (PEFT), often focuses on \textit{how to finetune} but neglects the issue of \textit{where to finetune}. As a pioneering work on answering where to finetune (at the layer level), we conduct a semantic analysis of the LM inference process. We first propose a virtual transition of the latent representation and then trace its factual transition. Based on the deviation in transitions, we estimate the gain of finetuning each model layer, and further, narrow down the scope for finetuning. We perform extensive experiments across well-known LMs and datasets. The results show that our approach is effective and efficient, and outperforms the existing baselines. Our approach is orthogonal to existing efficient techniques, such as PEFT methods, offering practical values on LM finetuning., Comment: 13 pages, 5 figures, under peer-review

In the framework of the relativized quark model, the calculation of spin-orbit interactions is improved by considering the contribution from the light quark cluster in a singly heavy baryon. It modifies the energy level splitting of the orbital excitation significantly and causes the emergence of fine structures for $\Sigma_{Q}$, $\Xi '_{Q}$ and $\Omega_{Q}$ baryons. Based on this improvement, we systematically analyze the fine structures and retest the heavy quark dominance mechanism. This mechanism is found to be violated in the $1P$-wave states of the $\Sigma_{c}$, $\Xi '_{c}$ and $\Omega_{c}$ baryons although it remains effective overall, which may help to understand the nature of the heavy quarks and strong interactions. With the predicted fine structures, we make the precise assignments of those observed heavy baryons which once could not be accurately explained due to their close mass values. The method used in this work is instructive and applicable for the study of more complex exotic hadrons, such as the heavy tetraquarks and pentaquarks., Comment: 10 pages, 3 figures, 2 tables

Understanding the latent space of language models (LMs) is important for improving the performance and interpretability of LMs. Existing analyses often fail to provide insights that take advantage of the semantic properties of language models and often overlook crucial aspects of language model adaptation. In response, we introduce a pioneering approach called vocabulary-defined semantics, which establishes a reference frame grounded in LM vocabulary within the LM latent space. We propose a novel technique to compute disentangled logits and gradients in latent space, not entangled ones on vocabulary. Further, we perform semantical clustering on data representations as a novel way of LM adaptation. Through extensive experiments across diverse text understanding datasets, our approach outperforms state-of-the-art methods of retrieval-augmented generation and parameter-efficient finetuning, showcasing its effectiveness and efficiency., Comment: under peer-review

Large Language Models (LLMs) have already gained widespread adoption in software engineering, particularly in code generation tasks. However, updating these models with new knowledge can be prohibitively expensive, yet it is essential to maximize their utility, such as implementing a hotfix technique to address urgent or critical LLM errors. In this paper, we propose \textsc{MENT}, a novel and effective model editing approach to repair LLMs in coding tasks. \textsc{MENT} is effective, efficient, and reliable, capable of correcting a neural model by patching just one or two neurons. As pioneering work on neuron-level model editing of generative models, we formalize the editing process and introduce the involved concepts. We also introduce new measures to evaluate its generalization ability and establish a benchmark for further study. Our approach is evaluated on three coding tasks: line-level code generation, shellcode generation, and intent-to-bash translation. The experimental results demonstrate that the proposed approach significantly outperforms the state-of-the-art in both effectiveness and efficiency measures. Furthermore, we showcase the applications of \textsc{MENT} for LLM reasoning in software engineering. By editing LLM knowledge, the directly or indirectly dependent behaviors of API invocation in the chain-of-thought change accordingly. This illustrates the significance of repairing LLMs in the context of software engineering., Comment: 12 pages, 7 figures, 7 tables, under peer-review

A mechanism of the heavy quark dominance in the orbital excitation is proposed in this paper which is testified to be reasonable for singly and doubly heavy baryons. In the relativistic quark model, an analysis of the Hamiltonian figures out the mechanism that the excitation mode with lower energy levels is always associated with the heavy quark(s), and the splitting of the energy levels is suppressed by the heavy quark(s). So, the heavy quarks dominate the orbital excitation of singly and doubly heavy baryons. Furthermore, a physical understanding of this mechanism is given in a semi-classical way. Accordingly, the predicted mass spectra of singly and doubly heavy baryons confirm the rationality of this mechanism. In addition, an interesting consequence of this mechanism is that a heavy-light meson is more likely to be produced in the strong decay of the high-orbital excited states, which is supported by experiments. This mechanism is rooted in the breakdown of the mass symmetry. Therefore, it may be also valid for other multi-quark systems, such as the tetraquarks Qqqq and QQqq, or the pentaquarks Qqqqq and QQqqq., Comment: 12 pages, 7 figures, 5 tables

In this paper, we investigate the mass spectra of bottom-charm baryons systematically, where the relativistic quark model and the infinitesimally shifted Gaussian basis function method are employed. Our calculation shows that the $\rho$-mode appears lower in energy than the other excited modes. According to this feature, the allowed quantum states are selected and a systematic study of the mass spectra for $\Xi_{bc}^{'}$ ($\Xi_{bc}$) and $\Omega_{bc}^{'}$ ($\Omega_{bc}$) families is performed. The root mean square radii and quark radial probability density distributions of these baryons are analyzed as well. Next, the Regge trajectories in the $(J,M^{2})$ plane are successfully constructed based on the mass spectra. At last, we present the structures of the mass spectra, and analyze the difficulty and opportunity in searching for the ground states of bottom-charm baryons in experiment., Comment: 19 pages, 9 figures, 6 tables. arXiv admin note: text overlap with arXiv:2210.13085

Based on the relativistic quark model and the infinitesimally shifted Gaussian basis function method, we investigate the mass spectra of double bottom baryons systematically. In the $\rho$-mode which appears lower in energy than the other excited modes, we obtain the allowed quantum states and perform a systematic study of the mass spectra of the $\Xi_{bb}$ and $\Omega_{bb}$ families. We analyze the root mean square radii and quark radial probability density distributions to deeply understand the structure of the heavy baryons. Meanwhile, the mass spectra allow us to successfully construct the Regge trajectories in the $(J,M^{2})$ plane. We also predict the masses of the ground states of double bottom baryons and discuss the differences between the structures of our spectra and those from other theoretical methods. At last, the shell structure of the double bottom baryon spectra is shown, from which one could get a bird's-eye view of the mass spectra., Comment: 16 pages, 9 figures, 5 tables. arXiv admin note: text overlap with arXiv:2207.04167

Deep code generation is a topic of deep learning for software engineering (DL4SE), which adopts neural models to generate code for the intended functions. Since end-to-end neural methods lack domain knowledge and software hierarchy awareness, they tend to perform poorly w.r.t project-level tasks. To systematically explore the potential improvements of code generation, we let it participate in the whole top-down development from \emph{expressibles} to \emph{executables}, which is possible in limited scopes. In the process, it benefits from massive samples, features, and knowledge. As the foundation, we suggest building a taxonomy on code data, namely code taxonomy, leveraging the categorization of code information. Moreover, we introduce a three-layer semantic pyramid (SP) to associate text data and code data. It identifies the information of different abstraction levels, and thus introduces the domain knowledge on development and reveals the hierarchy of software. Furthermore, we propose a semantic pyramid framework (SPF) as the approach, focusing on software of high modularity and low complexity. SPF divides the code generation process into stages and reserves spots for potential interactions. In addition, we conceived preliminary applications in software development to confirm the neuro-symbolic framework., Comment: 5 pages, 3 figures, 2 tables, accepted by ESEC/FSE 2023, the camera-ready version

Motivated by the experimental progress in the study of heavy baryons, we investigate the mass spectra of strange single heavy baryons in the $\lambda$-mode, where the relativistic quark model and the infinitesimally shifted Gaussian basis function method are employed. It is shown that the experimental data can be well reproduced by the predicted masses. The root mean square radii and radial probability density distributions of the wave functions are analyzed in detail. Meanwhile, the mass spectra allow us to successfully construct the Regge trajectories in the $(J,M^{2})$ plane. We also preliminarily assign quantum numbers to the recently observed baryons, including $\Xi_{c}(3055)$, $\Xi_{c}(3080)$, $\Xi_{c}(2930)$, $\Xi_{c}(2923)$, $\Xi_{c}(2939)$, $\Xi_{c}(2965)$, $\Xi_{c}(2970)$, $\Xi_{c}(3123)$, $\Xi_{b}(6100)$, $\Xi_{b}(6227)$, $\Xi_{b}(6327)$ and $\Xi_{b}(6333)$. At last, the spectral structure of the strange single heavy baryons is shown. Accordingly, we predict several new baryons that might be observed in forthcoming experiments., Comment: 27 pages, 11 figures, 8 tables

3D hand pose estimation from single depth is a fundamental problem in computer vision, and has wide applications.However, the existing methods still can not achieve satisfactory hand pose estimation results due to view variation and occlusion of human hand. In this paper, we propose a new virtual view selection and fusion module for 3D hand pose estimation from single depth.We propose to automatically select multiple virtual viewpoints for pose estimation and fuse the results of all and find this empirically delivers accurate and robust pose estimation. In order to select most effective virtual views for pose fusion, we evaluate the virtual views based on the confidence of virtual views using a light-weight network via network distillation. Experiments on three main benchmark datasets including NYU, ICVL and Hands2019 demonstrate that our method outperforms the state-of-the-arts on NYU and ICVL, and achieves very competitive performance on Hands2019-Task1, and our proposed virtual view selection and fusion module is both effective for 3D hand pose estimation., Comment: Accepted by AAAI2022

Automatic code summarization is beneficial to daily software development since it could help reduce the requirement of manual writing. Currently, artificial intelligence is undergoing a paradigm shift. The foundation models pretrained on massive data and finetuned to downstream tasks surpass specially customized models. This trend inspired us to consider reusing foundation models instead of learning from scratch. Thereby, we propose a flexible and robust approach for automatic code summarization, based on neural models. We assemble available foundation models, such as CodeBERT and GPT-2, into a single neural model named AdaMo. Moreover, we utilize Gaussian noise as the simulation of contextual information to optimize the latent representation. Furthermore, we introduce two adaptive schemes from the perspective of knowledge transfer, namely continuous pretraining and intermediate finetuning, and design intermediate stage tasks for general sequence-to-sequence learning. Finally, we evaluate AdaMo against a benchmark dataset for code summarization, by comparing it with state-of-the-art models., Comment: 12 pages, 2 figures, 8 tables, accepted by SANER 2022, the camera-ready version

The Transformer architecture and transfer learning have marked a quantum leap in natural language processing, improving the state of the art across a range of text-based tasks. This paper examines how these advancements can be applied to and improve code search. To this end, we pre-train a BERT-based model on combinations of natural language and source code data and fine-tune it on pairs of StackOverflow question titles and code answers. Our results show that the pre-trained models consistently outperform the models that were not pre-trained. In cases where the model was pre-trained on natural language "and" source code data, it also outperforms an information retrieval baseline based on Lucene. Also, we demonstrated that the combined use of an information retrieval-based approach followed by a Transformer leads to the best results overall, especially when searching into a large search pool. Transfer learning is particularly effective when much pre-training data is available and fine-tuning data is limited. We demonstrate that natural language processing models based on the Transformer architecture can be directly applied to source code analysis tasks, such as code search. With the development of Transformer models designed more specifically for dealing with source code data, we believe the results of source code analysis tasks can be further improved., Comment: Accepted for publication in the IEEE Transactions on Software Engineering (TSE)

Semantic code search is about finding semantically relevant code snippets for a given natural language query. In the state-of-the-art approaches, the semantic similarity between code and query is quantified as the distance of their representation in the shared vector space. In this paper, to improve the vector space, we introduce tree-serialization methods on a simplified form of AST and build the multimodal representation for the code data. We conduct extensive experiments using a single corpus that is large-scale and multi-language: CodeSearchNet. Our results show that both our tree-serialized representations and multimodal learning model improve the performance of code search. Last, we define intuitive quantification metrics oriented to the completeness of semantic and syntactic information of the code data, to help understand the experimental findings., Comment: 12 pages, 9 figures, 7 tables, accepted by ICSME 2021, the camera-ready version

This paper proposes a knowledge distillation method for foreground object search (FoS). Given a background and a rectangle specifying the foreground location and scale, FoS retrieves compatible foregrounds in a certain category for later image composition. Foregrounds within the same category can be grouped into a small number of patterns. Instances within each pattern are compatible with any query input interchangeably. These instances are referred to as interchangeable foregrounds. We first present a pipeline to build pattern-level FoS dataset containing labels of interchangeable foregrounds. We then establish a benchmark dataset for further training and testing following the pipeline. As for the proposed method, we first train a foreground encoder to learn representations of interchangeable foregrounds. We then train a query encoder to learn query-foreground compatibility following a knowledge distillation framework. It aims to transfer knowledge from interchangeable foregrounds to supervise representation learning of compatibility. The query feature representation is projected to the same latent space as interchangeable foregrounds, enabling very efficient and interpretable instance-level search. Furthermore, pattern-level search is feasible to retrieve more controllable, reasonable and diverse foregrounds. The proposed method outperforms the previous state-of-the-art by 10.42% in absolute difference and 24.06% in relative improvement evaluated by mean average precision (mAP). Extensive experimental results also demonstrate its efficacy from various aspects. The benchmark dataset and code will be release shortly., Comment: This paper will appear at ECCV 2020

Motivated by the fast adoption of WebAssembly, we propose the first functional pipeline to support the superoptimization of WebAssembly bytecode. Our pipeline works over LLVM and Souper. We evaluate our superoptimization pipeline with 12 programs from the Rosetta code project. Our pipeline improves the code section size of 8 out of 12 programs. We discuss the challenges faced in superoptimization of WebAssembly with two case studies., Comment: 4 pages, 3 figures. Proceedings of MoreVMs: Workshop on Modern Language Runtimes, Ecosystems, and VMs (2020)

Online hand gesture recognition (HGR) techniques are essential in augmented reality (AR) applications for enabling natural human-to-computer interaction and communication. In recent years, the consumer market for low-cost AR devices has been rapidly growing, while the technology maturity in this domain is still limited. Those devices are typical of low prices, limited memory, and resource-constrained computational units, which makes online HGR a challenging problem. To tackle this problem, we propose a lightweight and computationally efficient HGR framework, namely LE-HGR, to enable real-time gesture recognition on embedded devices with low computing power. We also show that the proposed method is of high accuracy and robustness, which is able to reach high-end performance in a variety of complicated interaction environments. To achieve our goal, we first propose a cascaded multi-task convolutional neural network (CNN) to simultaneously predict probabilities of hand detection and regress hand keypoint locations online. We show that, with the proposed cascaded architecture design, false-positive estimates can be largely eliminated. Additionally, an associated mapping approach is introduced to track the hand trace via the predicted locations, which addresses the interference of multi-handedness. Subsequently, we propose a trace sequence neural network (TraceSeqNN) to recognize the hand gesture by exploiting the motion features of the tracked trace. Finally, we provide a variety of experimental results to show that the proposed framework is able to achieve state-of-the-art accuracy with significantly reduced computational cost, which are the key properties for enabling real-time applications in low-cost commercial devices such as mobile devices and AR/VR headsets., Comment: Published in: 2019 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)