Search

One of the main objectives of testing is to achieve adequate code coverage. Modern code coverage standards suggest MC/DC (Modified Condition/Decision Coverage) instead of MCC (Multiple Condition Coverage) due to its ability to generate a feasible number of test cases. In contrast to the MC/DC, which only takes independent pairs into consideration, the MCC often considers each and every test case. In our work, we suggest SC-MCC, i.e., MCC with Short-Circuit. The key aspect of this paper is to demonstrate the effectiveness of SC-MCC-based test cases compared to MC/DC using Coverage-Guided Fuzzing (CGF) technique. In this work, we have considered American Fuzzy Lop (AFL) tool to generate both the SC-MCC and MC/DC test cases for 54 RERS benchmark programs. As part of this paper, we propose unique goal constraint generation and fuzz-instrumentation techniques that help in mitigating the masking problem of AFL. Subsequently, we performed mutation testing by employing the GCOV tool and computed the mutation score in order to evaluate the quality of the generated test cases. Finally, based on our observations, SC-MCC has performed better for over 85% of the programs taken into consideration. [ABSTRACT FROM AUTHOR]

Software defect prediction (SDP) aims to distinguish between defective and nondefective instances, but the imbalance between these two classes often leads to reduced prediction performance. Conventional SDP approaches use oversampling techniques, such as synthetic oversampling, to tackle the problem of imbalanced data. However, these methods merely synthesize new instances based on traditional code features without considering actual defects at the code level. To address the issue of data imbalance while preserving semantic features of code samples, a mutation‐based data augmentation approach in SDP is proposed. The method utilizes the mutation operator to generate mutants that mutate nondefective instances and create new defective instances. Six projects from the PROMISE dataset are used to evaluate the approach, employing four traditional and two deep classifiers. The experimental results demonstrate the effectiveness of this method in improving defect prediction performance for both traditional and deep classifiers compared with other data augmentation methods. A novel mutation‐based data augmentation method is proposed, in which data are increased at the code level while preserving its semantic features. The method utilizes the mutation operator for generating mutants to mutate against nondefective instances and then generates new defective instances. Experimental results demonstrate that this approach achieves the best average overall performance. [ABSTRACT FROM AUTHOR]

This is the RCR report of the artifact for the article "Fine-grained Coverage-based Fuzzing." This report contains scripts and pre-build binary programs to reproduce the results presented in the main article. The artifact is released on Zenodo with DOI: 10.5281/zenodo.7275184. We claim the artifact to be available, functional, and reusable. The technology skills needed to review the artifact are knowing how to use Linux/Unix terminal and a basic understanding of Docker. [ABSTRACT FROM AUTHOR]

Fault localization is a complex, costly and time‐consuming task in software debugging. Numerous automated techniques have been developed to expedite this process. Mutation‐based fault localization (MBFL) is one of the most widely studied techniques which uses mutation analysis to generate mutants for revealing potential faults in the program. However, our theoretical analysis exposes an inherent conflict between the fundamental assumption and the essential meaning of existing MBFL suspiciousness. This conflict is caused by mutant bias. Intuitively, the suspiciousness can be corrected by eliminating the mutant bias for more accurately measuring the faulty probability of the corresponding mutant statement. In this paper, we introduce Delta4Ms, a fault localization approach designed to eliminate mutant bias. Delta4Ms integrates the principles of signal theory, modelling the actual suspiciousness and mutant bias as the desired and false signal components, respectively. Based on theoretical derivation, the average suspiciousness of mutants serves as an estimate of mutant bias. Delta4Ms effectively mitigates mutant bias, extracting the desired signal and yielding corrected suspiciousness for fault localization. To precisely estimate mutant bias, higher order mutants (HOMs) are incorporated. We conduct an extensive experimental evaluation of Delta4Ms on 320 real‐fault programs from Codeflaws. The results indicate that our model significantly outperforms existing SBFL and MBFL techniques, showing a considerable improvement in fault localization effectiveness. We further assessed the robustness of Delta4Ms by examining different HOM ratios and HOM generation strategies. Moreover, Delta4Ms achieves a substantial reduction in mutation execution cost and minimal accuracy loss through the implementation of test case reduction. Finally, we perform preliminary experiments on 15 real‐fault programs from the Defects4J benchmark to assess the generalization of the model's fault localization effectiveness. [ABSTRACT FROM AUTHOR]

Background. Software testing is a critical activity for ensuring the quality and reliability of software systems. To evaluate the effectiveness of different test suites, researchers have developed a variety of metrics. Problem. However, comparing these metrics is challenging due to the lack of a standardized evaluation framework including comprehensive factors. As a result, researchers often focus on single factors (e.g., size), which finally leads to different or even contradictory conclusions. After comparing dozens of pieces of work in detail, we have found two main problems most troubling to our community: (1) researchers tend to oversimplify the description of the ground truth they use, and (2) data involving real defects is not suitable for analysis using traditional statistical indicators. Objective. We aim at scrutinizing the whole process of comparing test suites for our community. Method. To hit this aim, we propose a framework ASSENT (evAluating teSt Suite EffectiveNess meTrics) to guide the follow-up research for evaluating a test suite effectiveness metric. ASSENT consists of three fundamental components: ground truth, benchmark test suites, and agreement indicator. Its functioning is as follows: first, users clarify the ground truth for determining the real order in effectiveness among test suites. Second, users generate a set of benchmark test suites and derive their ground truth order in effectiveness. Third, users use the metric to derive the order in effectiveness for the same test suites. Finally, users calculate the agreement indicator between the two orders derived by two metrics. Result. With ASSENT, we are able to compare the accuracy of different test suite effectiveness metrics. We apply ASSENT to evaluate representative test suite effectiveness metrics, including mutation score and code coverage metrics. Our results show that, based on the real faults, mutation score, and subsuming mutation score are the best metrics to quantify test suite effectiveness. Meanwhile, by using mutants instead of real faults, test effectiveness will be overestimated by more than 20% in values. Conclusion. We recommend that the standardized evaluation framework ASSENT should be used for evaluating and comparing test effectiveness metrics in the future work. [ABSTRACT FROM AUTHOR]

The competent programmer hypothesis is one of the fundamental assumptions of mutation testing, which claims that most programmers are competent enough to create correct or almost correct source code. This implies that faults should usually manifest through small variations of the correct code. Consequently, researchers assumed that the synthetic faults injected in source code through the mutation operators closely resemble the real faults. Unfortunately, it is still unclear whether the competent programmer hypothesis holds, as past research presents contradictory claims. Within this article, we provide a new perspective on the competent programmer hypothesis and its relation to mutation testing. We try to re‐create real‐world faults through chains of mutations to understand if there is a direct link between mutation testing and faults. The lengths of these paths help us to understand if the source code is really almost correct, or if large variations are required. Our experiments used a state‐of‐the‐art benchmark database of real faults named Defects4J 2.0.0. It contains 835 reproducible real‐world faults in 17 open‐source projects that comprise a total of 1044 bug‐fix pairs of files. Our results indicate that while the competent programmer hypothesis seems to be true, mutation testing is missing important operators to generate representative real‐world faults. [ABSTRACT FROM AUTHOR]

Smart contracts (SC) are programs embodying certain business logic stored on a blockchain network like Ethereum. The execution of transactions on SC has a cost, measured in gas units, that depends on the low‐level operations performed. Therefore, a poor choice of high‐level language constructs could lead to overcharging users for their transactions. Thus, a testing process focused on possible deviations of the gas used in diverse scenarios could provide substantial global savings. This paper presents a gas‐centered mutation testing approach for taking care of the gas consumed by Solidity SCs. This approach can be useful to improve the test quality to detect gas‐related problems, reason about performance issues that only manifest in certain situations, and identify alternative more optimal implementations. We define and implement several mutation operators specifically designed to perturb gas consumption while preserving contract semantics in general. Our experiments using several real‐world SCs show the feasibility of the technique, with some mutants reproducing meaningful differences in the consumption and exposing some gas limits not tight enough in historic transactions. Therefore, our approach is shown to be a good ally to prevent the appearance of gas‐related issues and lays the groundwork for researchers seeking to improve performance testing practices. [ABSTRACT FROM AUTHOR]

Mutation testing is a valuable technique for measuring the quality of test suites in terms of detecting faults. However, one of its main drawbacks is its high computational cost. For this purpose, several approaches have been recently proposed to speed-up the mutation testing process by exploiting computational resources in distributed systems. However, bottlenecks have been detected when those techniques are applied in large-scale systems. This work improves the performance of mutation testing using large-scale systems by proposing a new load distribution algorithm, and parallelising different steps of the process. To demonstrate the benefits of our approach, we report on a thorough empirical evaluation, which analyses and compares our proposal with existing solutions executed in large-scale systems. The results show that our proposal outperforms the state-of-the-art distribution algorithms up to 35% in three different scenarios, reaching a reduction of the execution time of—at best—up to 99.66%. [ABSTRACT FROM AUTHOR]

Testing an intricate plexus of advanced software system architecture is quite challenging due to the absence of test oracle. Metamorphic testing is a popular technique to alleviate the test oracle problem. The effectiveness of metamorphic testing is dependent on metamorphic relations (MRs). MRs represent the essential properties of the system under test and are evaluated by their fault detection rates. The existing techniques for the evaluation of MRs are not comprehensive, as very few mutation operators are used to generate very few mutants. In this research, we have proposed six new MRs for dilation and erosion operations. The fault detection rate of six newly proposed MRs is determined using mutation testing. We have used eight applicable mutation operators and determined their effectiveness. By using these applicable operators, we have ensured that all the possible numbers of mutants are generated, which shows that all the faults in the system under test are fully identified. Results of the evaluation of four MRs for edge detection show an improvement in all the respective MRs, especially in MR1 and MR4, with a fault detection rate of 76.54% and 69.13%, respectively, which is 32% and 24% higher than the existing technique. The fault detection rate of MR2 and MR3 is also improved by 1%. Similarly, results of dilation and erosion show that out of 8 MRs, the fault detection rates of four MRs are higher than the existing technique. In the proposed technique, MR1 is improved by 39%, MR4 is improved by 0.5%, MR6 is improved by 17%, and MR8 is improved by 29%. We have also compared the results of our proposed MRs with the existing MRs of dilation and erosion operations. Results show that the proposed MRs complement the existing MRs effectively as the new MRs can find those faults that are not identified by the existing MRs. [ABSTRACT FROM AUTHOR]

The precise generation of train control curves for the on-board Automatic Train Protection (ATP) of the Chinese Train Control System Level 2 (CTCS-2) relies significantly on accurate train control engineering data, which serves as a critical element in ensuring the safe operation of trains. However, the traditional verification methods of train control engineering data depend on manual validation, which lacks timeliness and completeness and makes it easy to overlook potential errors. On the other hand, traditional verification rules are derived from railway technical specifications and standards, expressed in textual language that is often ambiguous, which leads to insufficient completeness in data verification. To address these challenges, this paper proposes a formal verification method for train control engineering data based on a Reduced Ordered Binary Decision Diagram (ROBDD). First, the attribute constraints of the train control data and the implicit constraint relations between different data objects are mined using existing railway technical specifications and expert knowledge of railway signals. These constraints are then converted into Boolean function models. Second, we formulate the ROBDD generation algorithm and the evaluation decision algorithm of the Boolean function using the equivalent canonical data structure ROBDD of the Boolean function. Finally, based on the train control engineering data from actual passenger-dedicated lines, the automatic verification method is developed by constructing four types of “mutation” data, including mutate, swap, add, and remove. The test cases indicate that our proposed formal verification method is feasible and capable of achieving high efficiency and completeness in the verification of CTCS-2 level train control engineering data.

ABSTRACTIn software testing, the quality of the test suite plays a very important role for not only the effectiveness of the testing but also the quality assurance of software. Mutation testing is considered as the usable, automatic and very effective technique in detecting mistakes of the set of test cases such as missing test cases, redundant test cases··· However, when using the mutation testing technique in practice, the generation of a large number of mutants has led to very high computational costs. This raises the question of whether we can reliably and accurately predict this mutation score without running mutants or not. If we can do this, it will save a lot of time and effort but still ensure the effectiveness of mutation testing. In this paper, we propose the approach using machine learning to perform mutation score cross-prediction for software which are new and completely different from the software used to generate test data (mutants) in model training and testing. The experimental results have shown that our proposed approach has achieved the positive results and is highly feasible. Thus, we believe that the approach can be applied to significantly reduce the cost of mutation testing.

Mutation testing is a powerful test adequacy assessment technique that can guarantee the deployment of deeply scrutinized and more reliable Smart Contract code. However, regularly evaluating the test suite during the evolution of a project can be unfeasible due to the high costs of the assessment. This paper presents ReSuMo, the first regression mutation testing approach and tool for Solidity Smart Contracts. ReSuMo uses a static and file-level technique to select a subset of Smart Contracts to mutate, and a subset of test files to re-run during a regression mutation testing campaign. After each mutation testing run, ReSuMo incrementally updates the results using the test outcomes registered on the previous program revision. In this way, it can speed up the mutation testing process for evolving projects, while providing a comprehensive adequacy assessment of the whole test suite. [ABSTRACT FROM AUTHOR]

Mutation testing is an important technique able to evaluate the bug-detection effectiveness of existing software test suites. Mutation testing tools exist for several languages, e.g., Java and JavaScript, but no solutions are available for managing the mutation testing process for entire web applications, in the context of end-to-end (E2E) web testing. In this paper, we propose Mutta, a novel tool able to automate the entire mutation testing process. Mutta mutates the various server source files of the target web application, runs the E2E test suite against the mutated web applications, and finally collects the test outcomes. To evaluate Mutta, we designed a case study using the mutated versions of the target web application with the aim of comparing the effectiveness of two different approaches to E2E web testing: (1) test cases based on classical assertions and (2) test cases relying on differential testing. In detail, Mutta has been executed on two web applications, each equipped with different test suites to compare assertions with differential testing. In this scenario, Mutta generated a large number of mutants (more than 15k overall), took into account the coverage information to consider only the mutants actually executed, deployed the mutated web app, ran the entire E2E test suites (about 87k tests runs overall), and finally, it correctly saved the test suite results. Thus, results of the case study show that Mutta can be successfully employed to automate the entire mutation testing process of E2E web test suites and, therefore, can be used in practice to evaluate the effectiveness of different test suites (e.g., based on different techniques, E2E frameworks, or composed by a different number of test scripts). [ABSTRACT FROM AUTHOR]

In software testing, the quality of the test suite plays a very important role for not only the effectiveness of the testing but also the quality assurance of software. Mutation testing is considered as the usable, automatic and very effective technique in detecting mistakes of the set of test cases such as missing test cases, redundant test cases··· However, when using the mutation testing technique in practice, the generation of a large number of mutants has led to very high computational costs. This raises the question of whether we can reliably and accurately predict this mutation score without running mutants or not. If we can do this, it will save a lot of time and effort but still ensure the effectiveness of mutation testing. In this paper, we propose the approach using machine learning to perform mutation score cross-prediction for software which are new and completely different from the software used to generate test data (mutants) in model training and testing. The experimental results have shown that our proposed approach has achieved the positive results and is highly feasible. Thus, we believe that the approach can be applied to significantly reduce the cost of mutation testing. [ABSTRACT FROM AUTHOR]

Mutation testing is the state‐of‐the‐art technique for assessing the fault detection capacity of a test suite. Unfortunately, a full mutation analysis is often prohibitively expensive. The CppCheck project for instance, demands a build time of 5.8 min and a test execution time of 17 s on our desktop computer. An unoptimised mutation analysis, for 55,000 generated mutants took 11.8 days in total, of which 4.3 days is spent on (re)compiling the project. In this paper, we present a feasibility study, investigating how a number of optimisation strategies can be implemented based on the Clang front‐end. These optimisation strategies allow to eliminate the compilation and execution overhead in order to support efficient mutation testing for the C language family. We provide a proof‐of‐concept tool that achieves a speedup of between 2 × and 30 ×. We make a detailed analysis of the speedup induced by the optimisations, elaborate on the lessons learned and point out avenues for further improvements. [ABSTRACT FROM AUTHOR]

Evaluating mutation testing behavior can help decide whether refactoring successfully maintains the expected initial test results. Moreover, manually performing this analytical work is both time-consuming and prone to errors. This paper extends an approach to assess test code behavior and proposes a tool called MeteoR. This tool comprises an IDE plugin to detect issues that may arise during test code refactoring, reducing the effort required to perform evaluations. A preliminary assessment was conducted to validate the tool and ensure the proposed test code refactoring approach is adequate. By analyzing not only the mutation score but also the generated mutants in the pre- and post-refactoring process, results show that the approach is capable of checking whether the behavior of the mutants remains unchanged throughout the refactoring process. This proposal represents one more step toward the practice of test code refactoring. It can improve overall software quality, allowing developers and testers to safely refactor the test code in a scalable and automated way. [ABSTRACT FROM AUTHOR]

Ensuring the safety, security, robustness, and fault tolerance of advanced robotic systems is essential for various fields, including healthcare, industry, and space. To address these issues, it is necessary to use software testing techniques and standards, similar to those applied in other safety-critical applications. The Robot Operating System (ROS) is a popular choice for developing robotic systems, so it is important to have specialized testing libraries and methods for it. In this study, a novel mutation testing library for ROS was developed and integrated it into the automated/tailored mutation-based software fault injection tool (IM-FIT). IM-FIT is an open-source automated software testing tool that is used to evaluate the software robustness of safety-critical systems using mutation-based tests. The proposed ROSMutation library mutates ROS-specific code snippets (publisher, subscriber, params, services, etc.) in the Python code within ROS-compatible software packages using IM-FIT. We evaluated the effectiveness of the ROSMutation library in two scenarios (basic and advanced), applying it to ROS-compatible code through IM-FIT and measuring its ability to assess the software robustness of ROS-compatible and Python-based software packages. The results showed that the ROSMutation library is effective in evaluating software robustness criteria for ROS-compatible and Python based software.

Activating mutations in Kirsten rat sarcoma viral oncogene homologue (KRAS), in particular, a point mutation leading to a glycine-to-cysteine substitution at codon 12 (G12C), are among the most frequent genomic alterations in non-small cell lung cancer (NSCLC). Several agents targeting KRAS G12C have recently entered clinical development. Sotorasib, a first-in-class specific small molecule that irreversibly inhibits KRAS G12C, has since obtained Health Canada approval. The emergence of novel KRAS-targeted therapies warrants the development of evidence-based consensus recommendations to help clinicians better understand and contextualize the available data. A Canadian expert panel was convened to define the key clinical questions, review recent evidence, and discuss and agree on recommendations for the treatment of advanced KRAS G12C-mutated NSCLC. The panel agreed that testing for KRAS G12C should be performed as part of a comprehensive panel that includes current standard-of-care biomarkers. Sotorasib, the only approved KRAS G12C inhibitor in Canada, is recommended for patients with advanced KRAS G12C-mutated NSCLC who progressed on guideline-recommended first-line standard of care for advanced NSCLC without driver alterations (immune-checkpoint inhibitor(s) [ICIs] +/− chemotherapy). Sotorasib could also be offered as second-line therapy to patients who progressed on ICI monotherapy that are not candidates for a platinum doublet and those that received first-line chemotherapy with a contraindication to ICIs. Preliminary data indicate the activity of KRAS G12C inhibitors in brain metastases; however, the evidence is insufficient to make specific recommendations. Regular liver function monitoring is recommended when patients are prescribed KRAS G12C inhibitors due to risk of hepatotoxicity.

Testing an intricate plexus of advanced software system architecture is quite challenging due to the absence of test oracle. Metamorphic testing is a popular technique to alleviate the test oracle problem. The effectiveness of metamorphic testing is dependent on metamorphic relations (MRs). MRs represent the essential properties of the system under test and are evaluated by their fault detection rates. The existing techniques for the evaluation of MRs are not comprehensive, as very few mutation operators are used to generate very few mutants. In this research, we have proposed six new MRs for dilation and erosion operations. The fault detection rate of six newly proposed MRs is determined using mutation testing. We have used eight applicable mutation operators and determined their effectiveness. By using these applicable operators, we have ensured that all the possible numbers of mutants are generated, which shows that all the faults in the system under test are fully identified. Results of the evaluation of four MRs for edge detection show an improvement in all the respective MRs, especially in MR1 and MR4, with a fault detection rate of 76.54% and 69.13%, respectively, which is 32% and 24% higher than the existing technique. The fault detection rate of MR2 and MR3 is also improved by 1%. Similarly, results of dilation and erosion show that out of 8 MRs, the fault detection rates of four MRs are higher than the existing technique. In the proposed technique, MR1 is improved by 39%, MR4 is improved by 0.5%, MR6 is improved by 17%, and MR8 is improved by 29%. We have also compared the results of our proposed MRs with the existing MRs of dilation and erosion operations. Results show that the proposed MRs complement the existing MRs effectively as the new MRs can find those faults that are not identified by the existing MRs.

The software development life cycle includes software testing as a critical step. Oracle testing is necessary to determine if a particular test case detects an error because the purpose of testing is to ensure that the application adheres to its specifications. This test leads to the consumption of time and cost and may lead to errors in which the tester falls. This research aims to present a proposed system based on the automation of the Oracle test using intelligent techniques. The model was trained using the random forest algorithm and using the convolutional neural network. This proposed system is considered one of the box techniques. Test blacks that depend on the output of the system being tested.

Activating mutations in Kirsten rat sarcoma viral oncogene homologue (KRAS), in particular, a point mutation leading to a glycine-to-cysteine substitution at codon 12 (G12C), are among the most frequent genomic alterations in non-small cell lung cancer (NSCLC). Several agents targeting KRAS G12C have recently entered clinical development. Sotorasib, a first-in-class specific small molecule that irreversibly inhibits KRAS G12C, has since obtained Health Canada approval. The emergence of novel KRAS-targeted therapies warrants the development of evidence-based consensus recommendations to help clinicians better understand and contextualize the available data. A Canadian expert panel was convened to define the key clinical questions, review recent evidence, and discuss and agree on recommendations for the treatment of advanced KRAS G12C-mutated NSCLC. The panel agreed that testing for KRAS G12C should be performed as part of a comprehensive panel that includes current standard-of-care biomarkers. Sotorasib, the only approved KRAS G12C inhibitor in Canada, is recommended for patients with advanced KRAS G12C-mutated NSCLC who progressed on guideline-recommended first-line standard of care for advanced NSCLC without driver alterations (immune-checkpoint inhibitor(s) [ICIs] +/− chemotherapy). Sotorasib could also be offered as second-line therapy to patients who progressed on ICI monotherapy that are not candidates for a platinum doublet and those that received first-line chemotherapy with a contraindication to ICIs. Preliminary data indicate the activity of KRAS G12C inhibitors in brain metastases; however, the evidence is insufficient to make specific recommendations. Regular liver function monitoring is recommended when patients are prescribed KRAS G12C inhibitors due to risk of hepatotoxicity. [ABSTRACT FROM AUTHOR]

Due to the fast-paced development of technology, the software has become a crucial aspect of modern life, facilitating the operation and management of hardware devices. Nevertheless, using substandard software can result in severe complications for users, putting human lives at risk. This underscores the significance of error-free and premium-quality software. Verification and validation are essential in ensuring high-quality software development; software testing is integral to this process. Although code coverage is a prevalent method for assessing the efficacy of test suites, it has some limitations. Therefore, mutation testing is proposed as a remedy to tackle these limitations. Furthermore, mutation testing is recognized as a method for directing test case creation and evaluating the effectiveness of test suites. Our proposed method involves autonomously learning mutations from faults in real-world software applications. Firstly, our approach involves extracting bug fixes at the method-level, classifying them according to mutation types, and performing code abstraction. Subsequently, the approach utilizes a deep learning technique based on neural machine translation to develop mutation models. Our method has been trained and assessed using approximately $\sim $ 588k bug fix commits extracted from GitHub. The results of our experimental assessment indicate that our models can forecast mutations resembling resolved bugs in 6% to 35% of instances. The models effectively revert fixed code to its original buggy version, reproducing the original bug and generating various other buggy codes with up to 94% accuracy. More than 96% of the generated mutants also demonstrate lexical and syntactic accuracy.

Test suite minimization is the task of finding a smaller test suite that still fulfills the properties of the original test suite but which comprises fewer test cases. It is important in practice, especially in the context of regression testing, where test suites are re-executed. However, test suite minimization as a set covering problem is known as an NP-complete problem, which requires applications of heuristics. Although many test suite minimization techniques have been applied previously but obtained conflicting results primarily due to inherent differences in underlying programming languages and experimental setup. In this respect, we study traditional greedy-based algorithms for test suite minimization that allow to remove test cases in a way such that the reduced test suite satisfies all requirements. Specifically, we evaluated commonly discussed approaches on publicly available JavaScript applications using mutation coverage. We show that the discussed algorithms reduce the test suite size of the studied example programs on average to 70% without compromising the fault-detection capability of the original test suite. The suggested approach not only minimizes the test suite’s size, thereby reducing the regression testing cost, but also ensures that the reduced test suite catches the same number of faults as that of the original test suite. Further, we also examine their performance in scenarios when meeting all testing requirements is not feasible due to time and budget constraints.

With advanced neural network techniques, language models can generate content that looks genuinely created by humans. Such advanced progress benefits society in numerous ways. However, it may also bring us threats that we have not seen before. A neural text detector is a classification model that separates machine-generated text from human-written ones. Unfortunately, a pretrained neural text detector may be vulnerable to adversarial attack, aiming to fool the detector into making wrong classification decisions. Through this work, we propose μ Attacking, a mutation-based general framework that can be used to evaluate the robustness of neural text detectors systematically. Our experiments demonstrate that μ Attacking identifies the detector's flaws effectively. Inspired by the insightful information revealed by μ Attacking, we also propose an RR-training strategy, a straightforward but effective method to improve the robustness of neural text detectors through finetuning. Compared with the normal finetuning method, our experiments demonstrated that RR-training effectively increased the model robustness by up to 11.33 % without increasing much effort when finetuning a neural text detector. We believe the μ Attacking and RR-training are useful tools for developing and evaluating neural language models. [ABSTRACT FROM AUTHOR]

Objectives: Lung cancer is an important cause of mortality in the United States. Targeted mutation analysis has the potential to alter mortality in those with non-small-cell lung cancer. As such, the importance of timely tissue turnaround time (TAT) is substantial. We evaluated TAT at Mayo Clinic Arizona and found it to be delayed relative to national standards. Material and Methods: We conducted a series of plan, do, study, and act (PDSA) cycles at a single institution to identify areas for improvement with our lung cancer genomic testing. We assembled a multidisciplinary team and held serial meetings to discuss data from each PDSA cycle. Results: Using PDSA cycles and multidisciplinary discussions, we were able to identify a number of process limitations slowing TAT. We were then able to generate enhanced and timely communication between providers and pathology, educate and enforce the order/requisition workflow, and establish pathology accessioning with lung cancer specimens top priority. Conclusion: We were able to generate and implement a standard operating procedure for genomic testing of lung cancer specimens at our institution, thereby reducing tissue TAT. [ABSTRACT FROM AUTHOR]

Fluoropyrimidines remain some of the most used chemotherapeutics, despite the appearance in the therapeutic arsenal of targeted therapy and immunotherapy. Fluropyrimidines related cardiotoxicity is an undesirable adverse event and affects almost 20% of patients. The mechanisms of fluoropyrimidine toxicity are closely related to deficient allelic variants of DPYD, but considering the low penetrance and interindividual variability, not all adverse reactions are explained by their presence. In this case, we report a patient with recurrent fluoropyrimidine toxicity without a deficient allelic variant and how this case was managed by the oncologist and cardiologist, considering the need to use fluoropyrimidine in the treatment. [ABSTRACT FROM AUTHOR]

Currently, there are many research studies that apply and improve mutation testing techniques including traditional mutation testing or first-order mutation testing, and higher-order mutation testing (HOMT) for evaluating the quality of the set of test data in particular, and the quality of test suites in general. The results of those studies have proven the effectiveness of mutation testing in the field of software testing. Mutation testing allows the quality of test cases to be automatically evaluated, thereby helping the testers to improve the quality in the design and execution of the software testing. Besides, these studies have also pointed out the main barriers in applying mutation testing techniques in practice. However, we are the first to introduce a method that can reduce the cost, but keep the quality of testing activity based on evaluating the quality of the mutation operator as well as the quality of the test cases. In this paper, we concentrate on two problems regarding higher-order mutation testing: Evaluating the quality of mutation operators as well as generated mutants and prioritizing test cases based upon its capability of killing mutants. This may help developers allocate suitably their resources during testing phase. The study of this paper is an extended version of our previous study titled “Evaluating Mutation Operator and Test Case Effectiveness by Means of Mutation Testing”, which is published in the proceedings of the 13th Asian Conference on Intelligent Information and Database Systems, ACIIDS 2021 (V. N. Do, Q. V. Nguyen and T. B. Nguyen, Evaluating Mutation Operator and Test Case Effectiveness by Means of Mutation Testing, in Intelligent Information and Database Systems. ACIIDS 2021, eds. N. T. Nguyen, S. Chittayasothorn, D. Niyato and B. Trawiński. Lecture Notes in Computer Science, Vol. 12672 (Springer, Cham), https://doi.org/10.1007/978-3-030-73280-6_66) to confirm the usefulness of our proposed method.

Software Testing is one of the most significant phases within the software development life cycle since software bugs can be costly and traumatic. However, the traditional software testing process is not enough on its own as some undiscovered faults might still exist due to the test cases’ inability to detect all underlying faults. Amidst the various proposed techniques of test suites’ efficiency detection comes mutation testing, one of the most effective approaches as declared by many researchers. Nevertheless, there is not enough research on how well the mutation testing tools adhere to the theory of mutation or how well their mutation operators are performing the tasks they were developed for. This research paper presents an investigative study on two different mutation testing tools for Java programming language, namely PIT and µJava. The study aims to point out the weaknesses and strengths of each tool involved through performing mutation testing on four different open-source Java programs to identify the best mutation tool among them. The study aims to further identify and compare the mutation operators of each tool by calculating the mutation score. That is, the operators’ performance is evaluated with the mutation score, with the presumption that the more prominent the number of killed mutants is, the higher the mutation score, thus the more effective the mutation operator and the affiliated tool.

Abstract Background Circulating cell free DNA (cfDNA) testing of plasma for EGFR somatic variants in lung cancer patients is being widely implemented and with any new service, external quality assessment (EQA) is required to ensure patient safety. An international consortium, International Quality Network for Pathology (IQNPath), has delivered a second round of assessment to measure the accuracy of cfDNA testing for lung cancer and the interpretation of the results. Methods A collaboration of five EQA provider organisations, all members of IQNPath, have delivered the assessment during 2018–19 to a total of 264 laboratories from 45 countries. Bespoke plasma reference material containing a range of EGFR mutations at varying allelic frequencies were supplied to laboratories for testing and reporting according to routine procedures. The genotyping accuracy and clinical reporting was reviewed against standardised criteria and feedback was provided to participants. Results The overall genotyping error rate in the EQA was found to be 11.1%. Low allelic frequency samples were the most challenging and were not detected by some testing methods, resulting in critical genotyping errors. This was reflected in higher false negative rates for samples with variant allele frequencies (VAF) rates less than 1.5% compared to higher frequencies. A sample with two different EGFR mutations gave inconsistent detection of both mutations. However, for one sample, where two variants were present at a VAF of less than 1% then both mutations were correctly detected in 145/263 laboratories. Reports often did not address the risk that tumour DNA may have not been tested and limitations of the methodologies provided by participants were insufficient. This was reflected in the average interpretation score for the EQA being 1.49 out of a maximum of 2. Conclusions The variability in the standard of genotyping and reporting highlighted the need for EQA and educational guidance in this field to ensure the delivery of high-quality clinical services where testing of cfDNA is the only option for clinical management.

Context: When software evolves, opportunities for introducing faults appear. Therefore, it is important to test the evolved program behaviors during each evolution cycle. However, while software evolves, its complexity is also evolving, introducing challenges to the testing process. To deal with this issue, testing techniques should be adapted to target the effect of the program changes instead of the entire program functionality. To this end, commit-aware mutation testing, a powerful testing technique, has been proposed. Unfortunately, commit-aware mutation testing is challenging due to the complex program semantics involved. Hence, it is pertinent to understand the characteristics, predictability, and potential of the technique. Objective: We conduct an exploratory study to investigate the properties of commit-relevant mutants, i.e., the test elements of commit-aware mutation testing, by proposing a general definition and an experimental approach to identify them. We thus aim at investigating the prevalence, location, and comparative advantages of commit-aware mutation testing over time (i.e., the program evolution). We also investigate the predictive power of several commit-related features in identifying and selecting commit-relevant mutants to understand the essential properties for its best-effort application case. Method: Our commit-relevant definition relies on the notion of observational slicing, approximated by higher-order mutation. Specifically, our approach utilizes the impact of mutants, effects of one mutant on another in capturing and analyzing the implicit interactions between the changed and unchanged code parts. The study analyses millions of mutants (over 10 million), 288 commits, five (5) different open-source software projects involving over 68,213 CPU days of computation and sets a ground truth where we perform our analysis. Results: Our analysis shows that commit-relevant mutants are located mainly outside of program commit change (81%), suggesting a limitation in previous work. We also note that effective selection of commitrelevant mutants has the potential of reducing the number of mutants by up to 93%. In addition, we demonstrate that commit relevant mutation testing is significantly more effective and efficient than state-of-the-art baselines, i.e., random mutant selection and analysis of only mutants within the program change. In our analysis of the predictive power of mutants and commit-related features (e.g., number of mutants within a change, mutant type, and commit size) in predicting commit-relevant mutants, we found that most proxy features do not reliably predict commit-relevant mutants. Conclusion: This empirical study highlights the properties of commit-relevant mutants and demonstrates the importance of identifying and selecting commit-relevant mutants when testing evolving software systems. [ABSTRACT FROM AUTHOR]

Multi-agent systems are proposed as a solution to mitigate nowadays software requirements: open and distributed architectures with dynamic and adaptive behaviour. Like any other software, multi-agent systems development process is error-prone; thus testing is a key activity to ensure the quality of the developed product. This paper sheds light on agent testing as it is the primary artefact for any multi-agent system's testing process. A framework called JADE Testing Framework (JTF) for JADE platform's agent testing is proposed. JTF allows testing agents at two levels: unit (inner-components) and agent (agent interactions) levels. JTF is the result of the integration of two testing solutions: JAT a well-known framework for JADE's agent's interaction testing and UJade, a new solution that was developed for agent's unit testing. UJade provides also a toolbox that allows for enhancing JAT capabilities. The evidence of JTF usability and effectiveness in JADE agent testing was supported by an empirical study conducted on seven multi-agent systems. The results of the study show that: when an agent's code can be tested either at agent or unit levels UJade is less test's effort consuming than JAT; JTF provides better testing capabilities and the developed tests are more effective than those developed using UJade or JAT alone. [ABSTRACT FROM AUTHOR]

Evaluating mutation testing behavior can help decide whether refactoring successfully maintains the expected initial test results. Moreover, manually performing this analytical work is both time-consuming and prone to errors. This paper extends an approach to assess test code behavior and proposes a tool called MeteoR. This tool comprises an IDE plugin to detect issues that may arise during test code refactoring, reducing the effort required to perform evaluations. A preliminary assessment was conducted to validate the tool and ensure the proposed test code refactoring approach is adequate. By analyzing not only the mutation score but also the generated mutants in the pre- and post-refactoring process, results show that the approach is capable of checking whether the behavior of the mutants remains unchanged throughout the refactoring process. This proposal represents one more step toward the practice of test code refactoring. It can improve overall software quality, allowing developers and testers to safely refactor the test code in a scalable and automated way.

In software engineering, testing has long been a research area of software maintenance. Testing is extremely expensive, and there is no guarantee that all defects will be found within a single round of testing. Therefore, fixing defects that are not discovered by a single round of testing is important for reducing the test costs. During the software maintenance process, testing is conducted within the scope of a set of test cases called a test suite. Mutation testing is a method that uses mutants to evaluate whether the test cases of the test suite are appropriate. In this paper, an approach is proposed that uses the mutants of a mutation test to identify defects that are not discovered through a single round of testing. The proposed method simultaneously applies two or more mutants to a single program to define and record the relationships between different lines of code. In turn, these relationships are examined using the defects that were discovered by a single round of testing, and possible defects are recommended from among the recorded candidates. To evaluate the proposed method, a comparative study was conducted using the fault localization method, which is commonly employed in defect prediction, as well as the Defects4J defect prediction dataset, which is widely used in software defect prediction. The results of the evaluation showed that the proposed method achieves a better performance than seven other fault localization methods (Tarantula, Ochiai, Opt2, Barinel, Dstar2, Muse, and Jaccard).

Summary: Function block diagram (FBD) is a standard programming language for programmable logic controllers (PLCs). PLCs have been widely used to develop safety‐critical systems such as nuclear reactor protection systems. It is crucial to test FBD programs for such systems effectively. This paper presents an automated test sequence generation approach using mutation testing techniques for FBD programs and the developed tool, MuFBDTester. Given an FBD program, MuFBDTester analyses the program and generates mutated programs based on mutation operators. MuFBDTester translates the given program and mutants into the input language of a satisfiability modulo theories (SMT) solver to derive a set of test sequences. The primary objective is to find the test data that can distinguish between the results of the given program and mutants. We conducted experiments with several examples including real industrial cases to evaluate the effectiveness and efficiency of our approach. With the control of test size, the results indicated that the mutation‐based test suites were statistically more effective at revealing artificial faults than structural coverage‐based test suites. Furthermore, the mutation‐based test suites detected more reproduced faults, found in industrial programs, than structural coverage‐based test suites. Compared to structural coverage‐based test generation time, the time required by MuFBDTester to generate one test sequence from industrial programs is approximately 1.3 times longer; however, it is considered to be worth paying the price for high effectiveness. Using MuFBDTester, the manual effort of creating test suites was significantly reduced from days to minutes due to automated test generation. MuFBDTester can provide highly effective test suites for FBD engineers. [ABSTRACT FROM AUTHOR]

Summary: This paper proposes TRANSMUT‐Spark for automating mutation testing of big data processing code within Spark programs. Apache Spark is an engine for big data analytics/processing that hides the inherent complexity of parallel big data programming. Nonetheless, programmers must cleverly combine Spark built‐in functions within programs and guide the engine to use the right data management strategies to exploit the computational resources required by big data processing and avoid substantial production losses. Many programming details in Spark data processing code are prone to false statements that must be correctly and automatically tested. This paper explores the application of mutation testing in Spark programs, a fault‐based testing technique that relies on fault simulation to evaluate and design test sets. The paper introduces TRANSMUT‐Spark for testing Spark programs by automating the most laborious steps of the process and fully executing the mutation testing process. The paper describes how the TRANSMUT‐Spark automates the mutant generation, test execution and adequacy analysis phases of mutation testing. It also discusses the results of experiments to validate the tool and argues its scope and limitations. [ABSTRACT FROM AUTHOR]

Test amplification aims to automatically improve a test suite. One technique generates new test methods through transformations of the original tests. These test amplification tools heavily rely on analysis techniques that benefit a lot from type declarations present in the source code of projects written in statically typed languages. In dynamically typed languages, such type declarations are not available, and therefore, research regarding test amplification for those languages is sparse. Recent work has brought test amplification to the dynamically typed language Pharo Smalltalk by introducing the concept of dynamic type profiling. The technique is dependent on Pharo‐specific frameworks and has not yet been generalized to other languages. Another significant downside in test amplification tools based on the mutation score of a test suite is their high time cost. In this paper, we present AmPyfier, a tool that brings test amplification and type profiling to the dynamically typed language Python. AmPyfier introduces multi‐metric selection in order to increase the time efficiency of test amplification. We evaluated AmPyfier on 11 open‐source projects and found that AmPyfier could strengthen 37 out of 54 test classes. Multi‐metric selection decreased the time cost ranging from 17% to 98% as opposed to selection based on the full mutation score. [ABSTRACT FROM AUTHOR]

Mutation testing is a technique that changes code instructions to assess the quality of automated software tests. Industry has not broadly adopted the technique because execution and analysis times are too long and not considered worth the effort. To change this, a variation called "extreme mutation testing" emerged, which mutates whole methods instead of instructions. The extreme variant trades accuracy for speed gains and also provides pre‐analyzed results. In this study, we aim to analyze both techniques on their granularity levels, look for benefits when combining them, and find motivations when a developer considers killing mutants. For that, we conducted a case study in a company from the semiconductor industry. We mutated a large Java software project which is tested by more than 11,000 unit tests, analyzed the results, manually inspected more than 1000 mutants, and conducted a focus group with five developers of the software. Among other results, we provide the distribution of traditional across extreme mutants as well as qualitative coding results of our mutant inspection and focus group transcript. We conclude that the traditional approach can be similarly strategically applied as the extreme one and that motivations of developers to target mutants are mostly not code related. [ABSTRACT FROM AUTHOR]