Search

Code generation, the task of producing source code from prompts, has seen significant advancements with the advent of pre-trained large language models (PLMs). Despite these achievements, there lacks a comprehensive taxonomy of weaknesses about the benchmark and the generated code, which risks the community's focus on known issues at the cost of under-explored areas. Our systematic study aims to fill this gap by evaluating five state-of-the-art PLMs: three larger models, CodeGen2.5 with 7 billion parameters, CodeGeeX2 with 6 billion parameters, GPT-4 Turbo, and two smaller ones, UnixCoder with 110 million parameters and CodeT5 base with 220 million parameters, across three popular datasets, CoNaLa, HumanEval Plus, and DS-1000. We assess the quality of generated code using match-based and execution-based metrics, then conduct thematic analysis to develop a taxonomy of nine types of weaknesses. We dissected weakness distributions in both larger and smaller models, applying an extensive methodology that encompasses model-specific as well as collective analysis (union and intersection) across models. Our research uncovers three salient findings: 1. In the CoNaLa dataset, inaccurate prompts are a notable problem, causing all large models to fail in 26.84% of cases, with even higher failure rates of 40% for smaller models; 2. Missing pivotal semantics is a pervasive issue across benchmarks, with one or more large models omitting key semantics in 65.78% of CoNaLa tasks, and similarly high occurrences in HumanEval Plus (66.09%) and DS-1000 (80.51%); 3. All models struggle with proper API usage, a challenge amplified by vague or complex prompts. Our findings aim to steer researchers towards addressing specific weaknesses and challenges in code generation. Furthermore, our annotations can offer a targeted benchmark subset for detailed analysis.

Software architectures are usually meticulously designed to address multiple quality concerns and support long-term maintenance. However, due to the imbalance between the cost and value for developers to document design rationales (i.e., the design alternatives and the underlying arguments for making or rejecting decisions), these rationales are often obsolete or even missing. The lack of design knowledge has motivated a number of studies to extract design information from various platforms in recent years. Unfortunately, despite the wealth of discussion records related to design information provided by platforms like open-source communities, existing research often overlooks the underlying arguments behind alternatives due to challenges such as the intricate semantics of discussions and the lack of benchmarks for design rationale extraction. In this paper, we propose a novel method, named by DRMiner, to automatically mine latent design rationales from developers' live discussion in open-source community (i.e., issue logs in Jira). To better identify solutions and the arguments supporting them, DRMiner skillfully decomposes the problem into multiple text classification tasks and tackles them using prompt tuning of language models and customized text-related features. To evaluate DRMiner, we acquire issue logs from Cassandra, Flink, and Solr repositories in Jira, and then annotate and process them under a rigorous scheme, ultimately forming a dataset for design rationale mining. Experimental results show that DRMiner achieves an F1 score of 65% for mining design rationales, outperforming all baselines with a 7% improvement over GPT-4.0. Furthermore, we investigate the usefulness of the design rationales mined by DRMiner for automated program repair (APR) and find that the design rationales significantly enhance APR, achieving 14 times higher full-match repairs on average.

Software requirements specification is undoubtedly critical for the whole software life-cycle. Nowadays, writing software requirements specifications primarily depends on human work. Although massive studies have been proposed to fasten the process via proposing advanced elicitation and analysis techniques, it is still a time-consuming and error-prone task that needs to take domain knowledge and business information into consideration. In this paper, we propose an approach, named ReqGen, which can provide recommendations by automatically generating natural language requirements specifications based on certain given keywords. Specifically, ReqGen consists of three critical steps. First, keywords-oriented knowledge is selected from domain ontology and is injected to the basic Unified pre-trained Language Model (UniLM) for domain fine-tuning. Second, a copy mechanism is integrated to ensure the occurrence of keywords in the generated statements. Finally, a requirement syntax constrained decoding is designed to close the semantic and syntax distance between the candidate and reference specifications. Experiments on two public datasets from different groups and domains show that ReqGen outperforms six popular natural language generation approaches with respect to the hard constraint of keywords(phrases) inclusion, BLEU, ROUGE and syntax compliance. We believe that ReqGen can promote the efficiency and intelligence of specifying software requirements.

Completeness is one of the most important attributes of software requirement specifications. Unfortunately, incompleteness is meanwhile one of the most difficult problems to detect. Some approaches have been proposed to detect missing requirements based on the requirement-oriented domain model. However, this kind of models are lacking for lots of domains. Fortunately, the domain models constructed for different purposes can usually be found online. This raises a question: whether or not these domain models are helpful in finding the missing functional information in requirement specification? To explore this question, we design and conduct a preliminary study by computing the overlapping rate between the entities in domain models and the concepts of natural language software requirements and then digging into four regularities of the occurrence of these entities(concepts) based on two example domains. The usefulness of these regularities, especially the one based on our proposed metric AHME (with F2 gains of 146% and 223% on the two domains than without any regularity), has been shown in experiments.

Context: Conflicts between software requirements bring uncertainties to product development. Some great approaches have been proposed to identify these conflicts. However, they usually require the software requirements represented with specific templates and/or depend on other external source which is often uneasy to build for lots of projects in practice. Objective: We aim to propose an approach Finer Semantic Analysis-based Requirements Conflict Detector (FSARC) to automatically detecting the conflicts between the given natural language functional requirements by analyzing their finer semantic compositions. Method: We build a harmonized semantic meta-model of functional requirements with the form of eight-tuple. Then we propose algorithms to automatically analyze the linguistic features of requirements and to annotate the semantic elements for their semantic model construction. And we define seven types of conflicts as long as their heuristic detecting rules on the ground of their text pattern and semantical dependency. Finally, we design and implement the algorithm for conflicts detection. Results: The experiment with four requirement datasets illustrates that the recall of FSARC is nearly 100% and the average precision is 83.88% on conflicts detection. Conclusion: We provide a useful tool for detecting the conflicts between natural language functional requirements to improve the quality of the final requirements set. Besides, our approach is capable of transforming the natural language functional requirements into eight semantic tuples, which is useful not only the detection of the conflicts between requirements but also some other tasks such as constructing the association between requirements and so on., Comment: 17 pages, 2 figures

The overwhelming foreground causes severe contamination on the detection of 21-cm signal during the Epoch of Reionization (EoR). Among various foreground components, the Galactic free-free emission is less studied, so that its impact on the EoR observation remains unclear. To better constrain this emission, we perform the Monte Carlo simulation of H$\alpha$ emission, which comprises direct and scattered H$\alpha$ radiation from HII regions and warm ionized medium (WIM). The positions and radii of HII regions are quoted from the WISE HII catalog, and the WIM is described by an axisymmetric model. The scattering is off dust and free electrons that are realized by applying an exponential fitting to the HI4PI HI map and an exponential disk model, respectively. The simulated H$\alpha$ intensity, the Simfast21 software, and the latest SKA1-Low layout configuration are employed to simulate the SKA "observed" images of Galactic free-free emission and the EoR signal. By analyzing the one-dimensional power spectra, we find that the Galactic free-free emission can be about $10^{5.4}$-$10^{2.1}$, $10^{5.0}$-$10^{1.7}$, and $10^{4.3}$-$10^{1.1}$ times more luminous than the EoR signal on scales of $0.1~{\rm Mpc^{-1}} < k < 2~{\rm Mpc^{-1}}$ in the 116-124, 146-154, and 186-194 MHz frequency bands, respectively. We further calculate the two-dimensional power spectra inside the EoR window and show that the power leaked by Galactic free-free emission can still be significant, as the power ratios can reach about $110\%$-$8000\%$, $30\%$-$2400\%$, and $10\%$-$250\%$ on scales of $0.5~{\rm Mpc^{-1}} \lesssim k \lesssim 1~{\rm Mpc^{-1}}$ in three frequency bands. Therefore, we indicate that the Galactic free-free emission should be carefully treated in future EoR detections., Comment: 19 pages,17figures, the ApJS published version. arXiv admin note: text overlap with arXiv:2005.04196

The overwhelming foreground contamination hinders the accurate detection of the 21-cm signal of neutral hydrogen during the Epoch of Reionization (EoR). Among various foreground components, the Galactic free-free emission is less studied, so that its impact on the EoR observations remains unclear. In this work, we employ the observed $\rm H\alpha$ intensity map with the correction of dust absorption and scattering, the Simfast21 software, and the latest SKA1-Low layout configuration to simulate the SKA "observed" images of Galactic free-free emission and the EoR signal. By calculating the one-dimensional power spectra from the simulated image cubes, we find that the Galactic free-free emission is about $10^{3.5}$-$10^{2.0}$, $10^{3.0}$-$10^{1.3}$, and $10^{2.5}$-$10^{1.0}$ times more luminous than the EoR signal on scales of $0.1~\rm Mpc^{-1} < k < 2~\rm Mpc^{-1}$ in the $116$-$124$, $146$-$154$, and $186$-$194$ ${\rm MHz}$ frequency bands. We further analyse the two-dimensional power spectra inside the properly defined EoR window and find that the leaked Galactic free-free emission can still cause non-negligible contamination, as the ratios of its power (amplitude squared) to the EoR signal power can reach about $200\%$, $60\%$, and $15\%$ on scales of $1.2~\rm Mpc^{-1}$ in three frequency bands, respectively. Therefore, we conclude that the Galactic free-free emission, as a severe contaminating foreground component, needs to be carefully treated in the forthcoming deep EoR observations., Comment: 15 pages, 10 figures, the MNRAS published version

Cosmic ray electron (CRE) acceleration and cooling are important physical processes in astrophysics. We develop an approximative framework to treat CRE physics in the parallel smoothed particle hydrodynamics code Gadget-3. In our methodology, the CRE spectrum of each fluid element is approximated by a single power-law distribution with spatially varying amplitude, upper cut-off, lower cut-off, and spectral index. We consider diffusive shock acceleration to be the source of injection, and oppositely the sinking processes is attributed to synchrotron radiation, inverse Compton scatters, and Coulomb scatters. The adiabatic gains and losses are also included. We show that our formalism produces the energy and pressure with an accuracy of $ > 90\%$ for a free cooling CRE spectrum. Both slope and intensity of the radio emission computed from the CRE population given by our method in cosmological hydro-simulation coincide well with observations, and our results also show that relaxed clusters have lower fluxes. Finally, we investigate several impacts of the CRE processes on the cosmological hydro-simulation, we find that: (1) the pressure of the CRE spectrum is very small and can be ignored in hydro-simulation, (2) the impacts of the CRE processes on the gas phase-space state of hydro-simulation is up to $3\%$, (3) the CRE processes induce a $5\%$ influence on the mass function in the mass range $10^{12} -10^{13} h^{-1} M_{\odot}$, (4) The gas temperature of massive galaxy cluster is influenced by the CRE processes up to $\sim 10\%$., Comment: 13 pages, 11 figures

The overwhelming foreground contamination is one of the primary impediments to probing the Epoch of Reionization (EoR) through measuring the redshifted 21 cm signal. Among various foreground components, radio halos are less studied and their impacts on the EoR observations are still poorly understood. In this work, we employ the Press-Schechter formalism, merger-induced turbulent re-acceleration model, and the latest SKA1-Low layout configuration to simulate the SKA "observed" images of radio halos. We calculate the one-dimensional power spectra from simulated images and find that radio halos can be about $10^4$, $10^3$ and $10^{2.5}$ times more luminous than the EoR signal on scales of $0.1\,\text{Mpc}^{-1} < k < 2\,\text{Mpc}^{-1}$ in the 120-128, 154-162, and 192-200 MHz bands, respectively. By examining the two-dimensional power spectra inside properly defined EoR windows, we find that the power leaked by radio halos can still be significant, as the power ratios of radio halos to the EoR signal on scales of $0.5\,\text{Mpc}^{-1} \lesssim k \lesssim 1\,\text{Mpc}^{-1}$ can be up to about 230-800%, 18-95%, and 7-40% in the three bands, when the 68% uncertainties caused by the variation of the number density of bright radio halos are considered. Furthermore, we find that radio halos located inside the far side-lobes of the station beam can also impose strong contamination within the EoR window. In conclusion, we argue that radio halos are severe foreground sources and need serious treatments in future EoR experiments., Comment: main text 18 pages, 16 figures, 2 appendixes; updated to match the ApJ edited version