Search

Regression testing is carried out to ensure that changes or enhancements are not impacting previous working software. Deciding how much retesting is required after modifications, bug fixes or before product deployments are difficult. Therefore, Test Case Selection (TCS) select the satisfactory subset of modified test cases from already executed test suites. The testing primary concerns in TCS for regression testing are efficiency (i.e., coverage, fault detection ability, redundancy) and time. The first challenge in TCS concerns the efficiency of multi-objective test case selection. The second challenge is to improve the execution time to detect the changes in a test suite, which makes it impractical to use these efficiency measures as a single goal for TCS. To overcome these challenges, there is a need to introduce an efficient detection-based multi-objective framework to improve the Time and efficiency of TCS. A multi-objective advanced and efficient regression test case selection (ARTeCS) framework is devised to improve the time performance and efficiency of a given TCS objective relative to the other TCS approaches. An algorithm to detect the changes in test cases using multiple TCS objectives. This comparison found that the enhanced ARTeCS algorithm improves redundancy efficiency by 44.02%. The selection technique showed ARTeCS improved the modified change detection by 43.00%, whereas the Hybrid Whale Optimization Algorithm (HWOA) stated 23% and ACO showed 33% only for selected test cases. Regarding average for fault detection, ACO scores 21%, HWOA scores 11%, and ARTeCS scores 31.08% with total execution times of 12, 21 and 09 seconds, respectively. In conclusion, the multiple-objective ARTeCS framework with four test suite selection parameters is more efficient than the existing multi-objective selection framework.

New approaches to software testing are required due to the rising complexity of today’s software applications and the rapid growth of software engineering practices. Among these methods, one that has shown promise is the introduction of Natural Language Processing (NLP) tools to software testing practices. NLP has witnessed a rise in popularity within all IT fields, especially in software engineering, where its use has improved the way we extract information from textual data. The goal of this systematic literature review (SLR) is to provide an in-depth analysis of the present body of the literature on the expanding subject of NLP-based software testing. Through a repeatable process, that takes into account the quality of the research, we examined 24 papers extracted from Web of Science and Scopus databases to extract insights about the usage of NLP techniques in the field of software testing. Requirements analysis and test case generation popped up as the most hot topics in the field. We also explored NLP techniques, software testing types, machine/deep learning algorithms, and NLP tools and frameworks used in the studied body of literature. This study also stressed some recurrent open challenges that need further work in future research such as the generalization of the NLP algorithm across domains and languages and the ambiguity in the natural language requirements. Software testing professionals and researchers can get important insights from the findings of this SLR, which will help them comprehend the advantages and challenges of using NLP in software testing.

Estimating software testability can crucially assist software managers to optimize test budgets and software quality. In this paper, we propose a new approach that radically differs from the traditional approach of pursuing testability measurements based on software metrics, e.g., the size of the code or the complexity of the designs. Our approach exploits automatic test generation and mutation analysis to quantify the evidence about the relative hardness of developing effective test cases. In the paper, we elaborate on the intuitions and the methodological choices that underlie our proposal for estimating testability, introduce a technique and a prototype that allows for concretely estimating testability accordingly, and discuss our findings out of a set of experiments in which we compare the performance of our estimations both against and in combination with traditional software metrics. The results show that our testability estimates capture a complementary dimension of testability that can be synergistically combined with approaches based on software metrics to improve the accuracy of predictions.

Behavior-driven development (BDD) is an Agile testing methodology fostering collaboration among developers, QA analysts, and stakeholders. In this manuscript, we propose a novel approach to enhance BDD practices using large language models (LLMs) to automate acceptance test generation. Our study uses zero and few-shot prompts to evaluate LLMs such as GPT-3.5, GPT-4, Llama-2-13B, and PaLM-2. The paper presents a detailed methodology that includes the dataset, prompt techniques, LLMs, and the evaluation process. The results demonstrate that GPT-3.5 and GPT-4 generate error-free BDD acceptance tests with better performance. The few-shot prompt technique highlights its ability to provide higher accuracy by incorporating examples for in-context learning. Furthermore, the study examines syntax errors, validation accuracy, and comparative analysis of LLMs, revealing their effectiveness in enhancing BDD practices. However, our study acknowledges that there are limitations to the proposed approach. We emphasize that this approach can support collaborative BDD processes and create opportunities for future research into automated BDD acceptance test generation using LLMs.

Dynamic analysis is a technique that is used to fully understand the internals of a system at runtime. On Android, dynamic security analysis involves real-time assessment and active adaptation of an app’s behaviour, and is used for various tasks, including network monitoring, system-call tracing, and taint analysis. The research on dynamic analysis has made significant progress in the past years. However, to the best of our knowledge, there is a lack in secondary studies that analyse the novel ideas and common limitations of current security research. The main aim of this work is to understand dynamic security analysis research on Android to present the current state of knowledge, highlight research gaps, and provide insights into the existing body of work in a structured and systematic manner. We conduct a systematic literature review (SLR) on dynamic security analysis for Android. The systematic review establishes a taxonomy, defines a classification scheme, and explores the impact of advanced Android app testing tools on security solutions in software engineering and security research. The study’s key findings centre on tool usage, research objectives, constraints, and trends. Instrumentation and network monitoring tools play a crucial role, with research goals focused on app security, privacy, malware detection, and software testing automation. Identified limitations include code coverage constraints, security-related analysis obstacles, app selection adequacy, and non-deterministic behaviour. Our study results deepen the understanding of dynamic analysis in Android security research by an in-depth review of 43 publications. The study highlights recurring limitations with automated testing tools and concerns about detecting or obstructing dynamic analysis.

Increasing the effectiveness of programming education has emerged as an important goal in teaching programming languages in the last decade. Automatic evaluation of the correctness of the student’s source code saves teachers time and effort and allows a more comprehensive focus on the preparation of assignments with integrated feedback. The study aims to present an approach that will enable effective testing of students’ source codes within object-oriented programming courses while minimising the demands on teachers when preparing the assignment. This approach also supports variability in testing and preventing student cheating. Based on the principles of different types of testing (black-box, white-box, grey-box), an integrated solution for source code verification was designed and verified. The basic idea is to use a reference class, which is assumed to be part of every assignment, as the correct solution. This reference class is compared to the student solution using the grey-box method. Due to their identical interface (defined by assignment), comparing instance states and method outputs is a matter of basic programming language mechanisms. A significant advantage is that a random generation of test cases can be used in such a case, while the rules for their generation can be determined using simple formulas. The proposed procedure was implemented and gradually improved over 4 years on groups of bachelor students of applied informatics with a high level of acceptance.

Behavior Driven Development (BDD) is a widely adopted agile methodology for software development that emphasizes the behavior of an application as a series of test cases, using the keywords, which include “Given,” “When,” and “Then.” It involves writing requirements in a structured and testable format that can be evaluated to ensure compliance with the expected behavior. Although a significant amount of research has been conducted to examine the impact of using BDD on software development process yet rare work is observed to synthesize these studies and identify areas for future exploration. This study presents a review of the state-of-the-art BDD by synthesizing the recent advancements in its uses and applications. It aims to systematically investigate the impact of BDD on software development process as well as on product quality by aiding to bridge the communication gap between the stakeholders. The results reveal that BDD is an effective technique to clarify requirements during the software development process as it helps minimizing the intrinsic ambiguities. This work proposes a taxonomy based on the role and applications of BDD in various contexts. It suggests a framework for applying BDD in software development and defines a workflow for its application in software development. Finally, this work highlights some pertinent future directions for the use of BDD in software development.

Due to the fast-paced development of technology, the software has become a crucial aspect of modern life, facilitating the operation and management of hardware devices. Nevertheless, using substandard software can result in severe complications for users, putting human lives at risk. This underscores the significance of error-free and premium-quality software. Verification and validation are essential in ensuring high-quality software development; software testing is integral to this process. Although code coverage is a prevalent method for assessing the efficacy of test suites, it has some limitations. Therefore, mutation testing is proposed as a remedy to tackle these limitations. Furthermore, mutation testing is recognized as a method for directing test case creation and evaluating the effectiveness of test suites. Our proposed method involves autonomously learning mutations from faults in real-world software applications. Firstly, our approach involves extracting bug fixes at the method-level, classifying them according to mutation types, and performing code abstraction. Subsequently, the approach utilizes a deep learning technique based on neural machine translation to develop mutation models. Our method has been trained and assessed using approximately $\sim $ 588k bug fix commits extracted from GitHub. The results of our experimental assessment indicate that our models can forecast mutations resembling resolved bugs in 6% to 35% of instances. The models effectively revert fixed code to its original buggy version, reproducing the original bug and generating various other buggy codes with up to 94% accuracy. More than 96% of the generated mutants also demonstrate lexical and syntactic accuracy.

Regression testing is one of the most expensive processes in testing. Prioritizing test cases in regression testing is critical for the goal of detecting the faults sooner within a large set of test cases. We propose a test case prioritization (TCP) technique for regression testing called LoM-Score inspired by the Law of Minimum (LoM) from biology. This technique calculates the impact probabilities of methods calculated by change impact analysis with forward slicing and orders test cases according to LoM. However, this ordering doesn’t consider the possibility that consecutive test cases may be covering the same methods repeatedly. Thereby, such ordering can delay the time of revealing faults that exist in other methods. To solve this problem, we enhance the LoM-Score TCP technique with an adaptive approach, namely with a dissimilarity-based coordinate analysis approach. The dissimilarity-based coordinate analysis uses Jaccard Similarity for calculating the similarity coefficients between test cases in terms of covered methods and the enhanced technique called Dissimilarity-LoM-Score (Dis-LoM-Score) applies a penalty with respective on the ordered test cases. We performed our case study on 10 open-source Java projects from Defects4J, which is a dataset of real bugs and an infrastructure for controlled experiments provided for software engineering researchers. Then, we hand-seeded multiple mutants generated by Major, which is a mutation testing tool. Then we compared our TCP techniques LoM-Score and Dis-LoM-Score with the four traditional TCP techniques based on their Average Percentage of Faults Detected (APFD) results.

The importance of software security has increased along with the number and severity of incidents in recent years. Security is a multidisciplinary aspect of the software development lifecycle, operation, and user utilization. Being a complex and specialized area of software engineering, it is often sidestepped in software development methodologies and processes. We address software security at the design level by adopting design patterns that encapsulate reusable solutions for recurring security problems. Design patterns can help development teams implement the best-proven solutions for a specialized problem domain. However, from the analysis of three secure pattern implementations by 70 junior programmers, we detected several structural errors resulting from their interpretation. We propose reusable unit testing test cases based on annotations to avoid secure pattern interpretation errors and provide an example for one popular secure pattern. Providing these test cases to the same group of programmers, they implemented the pattern without errors. The reason is annotations build a framework that disciplines programmers to incorporate secure patterns in their applications and ensure automatic testing.

Effort estimation of software testing plays a vital role in the effective completion of testing. In particular, software test effort estimation in defense projects is not an easy and simple phenomenon, owing to internal and external factors. This study aims to investigate the relationships between software test metrics used in the industry and software testing efforts. A method for estimating the testing effort is proposed using a novel set of software testing metrics that have not been used in any previously proposed software test effort estimation methods. In this study, 15 completed software projects of a CMMI Level-3 certified defense industry company were analyzed. The results of the empirical study show that the proposed method with the given metrics provides acceptable prediction quality, with Pred (0.25) and Pred (0.30) values equal to 0.867. We obtained plausible results and demonstrated that our newly proposed metrics can be safely used for software test effort estimation.

From the early days of computer systems to the present, software testing has been considered as a crucial process that directly affects the quality and reliability of software-oriented products and services. Accordingly, there is a huge amount of literature regarding the improvement of software testing approaches. However, there are limited reviews that show the whole picture of the software testing studies covering the topics and trends of the field. This study aims to provide a general figure reflecting topics and trends of software testing by analyzing the majority of software testing articles published in the last 40 years. A semi-automated methodology is developed for the analysis of software testing corpus created from core publication sources. The methodology of the study is based on the implementation of probabilistic topic modeling approach to discover hidden semantic patterns in the 14,684 published articles addressing software testing issues between 1980 and 2019. The results revealed 42 topics of the field, highlighting five software development ages, namely specification, detection, generation, evaluation, and prediction. The recent accelerations of the topics also showed a trend toward prediction-based software testing actions. Additionally, a higher trend on the topics concerning “Security Vulnerability”, “Open Source” and “Mobile Application” was identified. This study showed that the current trend of software testing is towards prediction-based testing strategies. Therefore, the findings of this study may provide valuable insights for the industry and software communities to be prepared for the possible changes in the software testing procedures using prediction-based approaches.

In software debugging, fault localization is the most difficult, expensive, tedious, and time-consuming task, particularly for large-scale software systems. This is due to the fact that it requires significant human participation and it is difficult to automate its sub-tasks. Therefore, there is a high demand for automatic fault localization techniques that can help software engineers effectively find the locations of faults with minimal human intervention. This has led to the proposal of implementing different types of such techniques. However, Spectrum Based Fault Localization (SBFL) is considered amongst the most prominent techniques in this respect due to its efficiency and effectiveness. In SBFL, the probability of each program element (e.g., statement, block, or function) being faulty is calculated based on the results of executing test cases and their corresponding code coverage information. However, SBFL techniques are not yet widely adopted in the industry. The rationale behind this is that they pose a number of issues and their performance is affected by several influential factors. For example, the characteristics of bugs, target programs, test suites, and supporting tools make their effectiveness differ dramatically from one case to another. There are massive studies on SBFL that cover its usage, formulas, performance, etc. So far, no dedicated survey points out comprehensively the issues of SBFL. In this paper, various SBFL challenges and issues have been identified, categorized, and discussed alongside many directions. Also, the paper raises awareness of the works being achieved to address the identified issues and suggests some potential solutions too.

With the rapid increase of software scale and complexity, the cost of traditional software testing methods will increase faster than the scale of software. In order to improve test efficiency, it is particularly important to automatically generate high-quality test cases. This paper introduces a framework for automatic test data generation based on the generative adversarial network (GAN). GAN is employed to train a generative model over execution path information to learn the behavior of the software. Then we can use the trained generative model to produce new test data, and select the test data that can improve the branch coverage according to our proposed selection strategy. Compared to prior work, our proposed method is able to handle programs under test with large-scale branches without analyzing branch expressions. In the experiment, we exhibit the performance of our method by using two modules in GNU Scientific Library. In particular, we consider the application of our method in two testing scenarios; unit testing and integration testing, and conduct a series of experiments to compare the performance of three types of GAN models. Results indicate that the WGAN-GP shows the best performance in our framework. Compared with the random testing method, the WGAN-GP based framework improves the test coverage of five functions out of the seven in the unit testing.

Due to changing software requirements and increasing system complexity, which put forward new challenges to the related testing processes to ensure software trustworthiness. Software testing is an important means to improve software quality, and standardized software testing process has become an effective measure to ensure software testing quality. In order to solve the problem of software quality decline caused by non-standard behaviors in software testing processes, this paper proposes a standardization workflow technology to construct the management system of software testing processes, and applies the workflow management system to the software testing of RSA timing attack tasks. For the numerical experimental setup, two software fault detection time-domain datasets are considered, i.e., datasets generated from the RSA timing attack program in functional testing processes and workflow verification system used to manage the RSA timing attack processes in non-functional testing processes. Then, software reliability is evaluated by employing multiple heterogeneous software reliability growth models (SRGMs) with corresponding assumed conditions, parameter information and reliability prediction methods. Furthermore, we compare the fitting power and predictive performance of multiple SRGMs in terms of the model evaluation criteria, which helps testers select a better SRGM from the model evaluation results for reliability analysis. The experimental results show that the proposed standardization workflow technology can reasonably regulate the software testing processes, making the fault data input into SRGMs more reliable and reliability analysis results more credible.

Erasable-itemset mining is often used in production planning to identify itemsets that, if removed, would make little effect on the production profits. Consequently, much recent attention has been focused on increasing the mining efficiency. Yet, in addition to mining efficiency, guaranteeing mining correctness is also an important issue. In real applications, wrong mining results might lead a company to make inappropriate decisions. Therefore, in this paper, we use the metamorphic testing, a lightweight software-testing strategy, to check the mining results via discovering proper metamorphic relations. The core idea contains five metamorphic relations oriented from two aspects. In the first aspect, we alter maximum thresholds without changing the input data, and in the second aspect, we modify product databases and material items with fixing maximum thresholds. In the experiments, 11 datasets and 76 mutants generated by $\mu $ Java were used to discuss the effects of database parameters, including the average number of materials in products, the number of different material items, the number of products, and the maximum threshold, respectively. The experimental results show that the metamorphic relations deliver good assessment performances in terms of effectiveness and efficiency, especially the fourth metamorphic relation.

There is an indisputable industrial need for highly skilled individuals in the role of software testers. However, little is known about what activities under testers’ responsibilities, competencies, and experiences sought after from employers’ perspectives. For the purpose of this research, a data set of 1000 job ads related to software testing role in the United States was collected and analyzed. Specifically, a thorough analysis was conducted to find the industrial demand for competencies for the software testing role in terms of (i) Level of education, training, and experience; (ii) Testing skills; (iii) Technical skills; and (iv) Soft-skills. Relevant correlated skills that may influence shaping the profile of the software tester were also analyzed. Also, the essential duties that a software tester is expected to perform were investigated. The results from the subsequent quantitative and qualitative analysis are reported.

Context: Flaky tests plague regression testing in Continuous Integration environments by slowing down change releases and wasting testing time and effort. Despite the growing interest in mitigating the burden of test flakiness, how to efficiently and effectively detect flaky tests is still an open problem. Objective: In this study, we present and evaluate FLAST, an approach designed to statically predict test flakiness. FLAST leverages vector-space modeling, similarity search, dimensionality reduction, and $k$ -Nearest Neighbor classification in order to timely and efficiently detect test flakiness. Method: In order to gain insights into the efficiency and effectiveness of FLAST, we conduct an empirical evaluation of the approach by considering 13 real-world projects, for a total of 1,383 flaky and 26,702 non-flaky tests. We carry out a quantitative comparison of FLAST with the state-of-the-art methods to detect test flakiness, by considering a balanced dataset comprising 1,402 real-world flaky and as many non-flaky tests. Results: From the results we observe that the effectiveness of FLAST is comparable with the state-of-the-art, while providing considerable gains in terms of efficiency. In addition, the results demonstrate how by tuning the threshold of the approach FLAST can be made more conservative, so to reduce false positives, at the cost of missing more potentially flaky tests. Conclusion: The collected results demonstrate that FLAST provides a fast, low-cost and reliable approach that can be used to guide test rerunning, or to gate the inclusion of new potentially flaky tests.

Traceability information can be crucial for software maintenance, testing, automatic program repair, and various other software engineering tasks. Customarily, a vast amount of test code is created for systems to maintain and improve software quality. Today’s test systems may contain tens of thousands of tests. Finding the parts of code tested by each test case is usually a difficult and time-consuming task without the help of the authors of the tests or at least clear naming conventions. Recent test-to-code traceability research has employed various approaches but textual methods as standalone techniques were investigated only marginally. The naming convention approach is a well-regarded method among developers. Besides their often only voluntary use, however, one of its main weaknesses is that it can only identify one-to-one links. With the use of more versatile text-based methods, candidates could be ranked by similarity, thus producing a number of possible connections. Textual methods also have their disadvantages, even machine learning techniques can only provide semantically connected links from the text itself, these can be refined with the incorporation of structural information. In this paper, we investigate the applicability of three text-based methods both as a standalone traceability link recovery technique and regarding their combination possibilities with each other and with naming conventions. The paper presents an extensive evaluation of these techniques using several source code representations and meta-parameter settings on eight real, medium-sized software systems with a combined size of over 1.25 million lines of code. Our results suggest that with suitable settings, text-based approaches can be used for test-to-code traceability purposes, even where naming conventions were not followed.

Fuzzing is a widely used technique to discover vulnerabilities in software. However, for programs requiring highly structured inputs, the byte-based mutation strategies in existing fuzzers have difficulties in generating valid inputs. To resolve this challenge, Grammar-Based Fuzzing (GBF) utilizes existing grammar specifications to generate new inputs. Some GBFs perform mutation based on Abstract Syntax Trees (ASTs), which can generate inputs conforming to grammars. However, the existing GBFs neglect using feedback to optimize mutation strategies, and blindly generate inputs without considering the effectiveness of those inputs. In this paper, we use the power schedule and the subtree pool to optimize mutation strategies. Specifically, we first translate input files into ASTs, and extract subtrees from ASTs into a subtree pool. Then, we optimize the power schedule on AST nodes based on a probabilistic model. That is, we adaptively determine the time budget for mutating an AST node. Finally, we replace AST nodes along with their subtrees using the ones we select from the subtree pool. We implement a fuzzing tool to demonstrate our strategies. The experiment results show that our method outperforms the state-of-the-art methods in fuzzing efficiency.

Recently, software testing has become a significant component of information security. The most reliable technique for automated software testing is a fuzzing tool that feeds programs with random test-input and detects software vulnerabilities that are critical to security. Similarly, symbolic execution has gained the most attention as an efficient testing tool for producing smart test-inputs and discovering hard-to-reach bugs using search-based heuristics and compositional approaches. The combination of fuzzing and symbolic execution makes software testing more efficient by mitigating the limitations in each other. Although several studies have been conducted on hybrid fuzzing in recent years, a comprehensive and consistent review of hybrid fuzzing techniques has not been explored. To add coherence to the extensive literature on hybrid fuzzing and to make it reach a large audience, this study provides an overview of key concepts along with the taxonomy of existing hybrid fuzzing tools, problems, and solutions that have been developed in this sphere. It also includes evaluations of the proposed approaches and a number of suggestions for the development of hybrid fuzzing in the future.

We propose a novel technique (TOUR) to improve both bug detection ability and verification speed of ARMC by detecting a target path quickly. The key idea of TOUR is an error location directed search that utilizes the distance to an error location and function call context at runtime. TOUR applies four different distance metrics and a distance metric selection heuristic using static features of a target program. We have extensively evaluated TOUR on 3,042 real-world C programs in a software verification competition benchmark. The experiment results show that TOUR, due to its error location directed search, finds bugs in 20% more programs in 11% less model checking time than the state-of-the-art ARMC technique (i.e., block-abstraction memoization) for 354 buggy programs. Also, TOUR verifies 15% more programs within 15% less model checking time than the block-abstraction memoization for 652 complex clean programs.

Software testing is a vital part of software engineering process. Automated testing makes this process more accurate and more efficient. For automated testing, many different testing tools were introduced. Due to the large number and the variety of testing tools, selecting the appropriate tools became a difficult confusable task. This research aims at developing a comprehensive taxonomy for testing tools that cover a broad range of testing tools criteria. This comprehensive view would help software developers and software vendors to specify the testing tool/s they need/develop accurately. In details, the framework includes two main parts: (1) comprehensive taxonomy of testing tools; (2) multi-criteria selection method. The first part covers different criteria of testing tools. Because these criteria are large in numbers, wide and variant, a taxonomy of these criteria is needed. This taxonomy will help developers distinguish among testing tools based on a wide spectrum of different criteria. The second part of the framework is a multi-criteria selection method; that enables software developers to choose the appropriate testing tool using a systematic and adequate automated manner. The selection method employs scientific two well-known methods of multi-criteria decision-making techniques; Analytic Hierarchy Process (AHP) and Technique for Order Preference by Similarity to Ideal Solution (TOPSIS). The testing tools taxonomy is well validated by academic professionals in software engineering and achieved good scores in terms of significance, usefulness and comprehension. Academics reported that the taxonomy is slightly complex and needs to be simplified. The selection method was validated using different scenarios to prove the quality of selection even in complex cases with many criteria and many alternatives.

Software reliability depends on the performed tests. Bug detection and diagnosis are based on test outcome (oracle) analysis. Most of practical test reports do not provide sufficient information for localizing and correcting bugs. We have found the need to extend the space of test result observation in data and time perspectives. This resulted in tracing supplementary test result features in event logs. They are explored with combined text mining and log parsing techniques. Another important point is correlating test life cycle with project development history journaled in issue tracking and software version control repositories. Dealing with the outlined problems, neglected in the literature, we have introduced original analysis schemes. They focus on assessing test coverage, reasons of low diagnosability, and test result profiles. Multidimensional investigation of test features and their management is supported with the developed test infrastructure. This assures a holistic insight into the test efficiency to identify test scheme deficiencies (e.g., functional inadequacy, aging, insufficient coverage) and possible improvements (test set updates). Our studies have been verified in relevance to a real commercial project and confronted with the experience of testers engaged in other projects.

Software quality can be assured by passing the process of software testing. However, software testing process involve many phases which lead to more resources and time consumption. To reduce these downsides, one of the approaches is to adopt test case prioritization (TCP) where numerous works has indicated that TCP do improve the overall software testing performance. TCP does have several kinds of techniques which have their own strengths and weaknesses. As for this review paper, the main objective of this paper is to examine deeper on machine learning (ML) techniques based on research questions created. The research method for this paper was designed in parallel with the research questions. Consequently, 110 primary studies were selected where, 58 were journal articles, 50 were conference papers and 2 considered as others articles. For overall result, it can be said that ML techniques in TCP has trending in recent years yet some improvements are certainly welcomed. There are multiple ML techniques available, in which each technique has specified potential values, advantages, and limitation. It is notable that ML techniques has been considerably discussed in TCP approach for software testing.

The multiagent path finding (MAPF) problem identifies the scheduling of multiple agents simultaneously, such that all of them can reach their targets efficiently. To date, MAPF systems have been assigned important tasks such as traffics and warehouses. It is essential to conduct testing for MAPF systems to detect potential failures. Namely, in an MAPF system, a test case is a specific MAPF scenario, including the initial locations of the agents and the environment for these agents to play in. By testing, we intend to find the scenarios (i.e., test cases) whose executions reveal failures. Testing MAPF systems is challenging due to the complexity of its input and the interactions among multiple agents. This article proposes the testing approach based on the adaptive random testing (ART) for MAPF systems. ART aims to generate new test cases far from the already executed ones. Particularly, to calculate the distance between each pair of test cases, we introduce two metrics, the initial density distribution and the destination density distribution, to characterize the distribution of the agents’ initial and destination nodes, respectively. Benefit from ART, the diversity of the information generated during testing can be improved. Experimental results show that compared with the random testing, our approach can detect more diverse failure-revealing scenarios. [ABSTRACT FROM AUTHOR]

Mutation testing is basically a fault-based software testing technique, which has been proposed to measure the fault detection effectiveness of a test suite using programs with simulated faults (namely mutants). However, mutation testing is time consuming and computationally expensive because of the normal use of a large amount of mutants. Thus, reducing the mutants is of great significance. To address this problem, various mutant reduction techniques have been proposed. Among them, the identification of redundant mutants aims at removing mutants whose test results can be inferred by other mutants. This article proposes a redundant mutant identification technique based on selective symbolic execution called ReMuSSE for weak mutation testing. Redundant mutants could be revealed by identifying those with similar program execution state changes within a program block involving mutated statements. An empirical study was conducted using 13 C programs from different application domains with varying sizes. The empirical results showed that ReMuSSE could identify up to 31.4% redundant mutants and consequentially save up to 35.2% time cost of weak mutation testing. The results demonstrated that ReMuSSE could effectively identify redundant mutants and thus could significantly improve the efficiency of weak mutation testing. [ABSTRACT FROM AUTHOR]

Artificial intelligence has made significant progress in software testing as it is being incorporated into all stages of the software test lifecycle (STLC). However, formidable challenges remain within software testing activities in the STLC. [ABSTRACT FROM AUTHOR]

In recent years, service oriented architecture (SOA) has been increasingly adopted to develop distributed applications in the context of the Internet. To develop reliable SOA-based applications, an important issue is how to ensure the quality of web services. In this article, we propose a dynamic random testing (DRT) technique for web services, which is an improvement over the widely-practiced random testing (RT) and partition testing (PT) approaches. We examine key issues when adapting DRT to the context of SOA, including a framework, guidelines for parameter settings, and a prototype for such an adaptation. Empirical studies are reported where DRT is used to test three real-life web services, and mutation analysis is employed to measure the effectiveness. Our experimental results show that, compared with the three baseline techniques, RT, Adaptive Testing (AT) and Random Partition Testing (RPT), DRT demonstrates higher fault-detection effectiveness with a lower test case selection overhead. Furthermore, the theoretical guidelines of parameter setting for DRT are confirmed to be effective. The proposed DRT and the prototype provide an effective and efficient approach for testing web services. [ABSTRACT FROM AUTHOR]

It’s April 2018, and I’m stuck inside a hotel room, staring at a laptop, panicking (). I’ve been carefully preparing for months to demonstrate my work at EGX Rezzed—one of the United Kingdom’s most popular videogames expos—and after a lot of hard work and testing, something has gone horribly wrong. [ABSTRACT FROM AUTHOR]

Software is widely used in many application domains. The most popular software are used by millions every day. How to accurately predict and assess the reliability of developed software is becoming increasingly important for project managers and developers. Previous studies have primarily used the software reliability growth model (SRGM) to evaluate and predict software reliability, but prediction results cannot be accurate at particular times or in particular situations. One of the main reasons is that simplified assumptions and abstractions are usually made to simplify the problem when developing SRGMs. Selecting an appropriate SRGM should depend on the key characteristics of the software project. In this article, we propose a deep learning-based approach for software reliability prediction and assessment. Specifically, we clearly demonstrate how to derive mathematical expressions from the computational methods of deep learning models and how to determine the correlation between them and the mathematical formula of SRGMs, and then, we use the back-propagation algorithm to obtain the SRGM parameters. Furthermore, we further integrate some deep learning-based SRGMs and also propose a method for the weighted assignment of combinations. Three real open source software failure datasets are used to evaluate the performance of the proposed models compared to selected SRGMs. The experimental results reveal that our proposed deep learning-based models and their combinations perform better than several classical SRGMs. [ABSTRACT FROM AUTHOR]

Model-based testing has been highlighted in the last few decades. Many improvements have been proposed for this testing method. One improvement is the use of extended regular expressions (EREs) for modeling software behavior, and then using the ERE model to generate test paths. To improve the theory of test generation based on the ERE model, this article presents an algebraic system, named transition algebra, by extending Kleene algebra. Eight operations and their corresponding operational properties, including basic and nonbasic operational properties, are designed for transition algebra, and all such properties are proven. Five examples are given to illustrate the use of ERE modeling and algebraic operations. To ensure the automatic generation of test paths from the ERE model, the article proves that any ERE model constructed by transition algebra can be converted to a set of transition sequences. Compared to Kleene algebra with tests, transition algebra has more operations to build a powerful ERE model and more operational properties to process the ERE model for the generation of test paths. [ABSTRACT FROM AUTHOR]

The Common Weakness Enumeration community publishes an aggregate metric to calculate the "most dangerous software errors." However, the equation highly biases frequency over exploitability and impact. We provide a metric to mitigate this bias and discuss the most significant weaknesses. [ABSTRACT FROM AUTHOR]

To evaluate Internet of Things device security, researchers have attempted to emulate and dynamically analyze firmware. However, this approach cannot deal with complex hardware/environmental diversities. We show that heuristic workarounds can enable firmware emulation and facilitate the discovery of vulnerabilities. [ABSTRACT FROM AUTHOR]

The main task of software testing is the generation of test data with high quality in restricted time. But for software with uncertainties, such as randomness, existing methods of generating test data often lose their effectiveness. For software with randomness, a test datum might cover the test target in one run, but obtains different results in the next run. Therefore, special testing methods must be exploited for this kind of software. In order to test software with randomness, we first propose a novel test adequacy criterion, and then build a mathematical model for generating test data according to the new criterion. Finally, we present a method of solving the optimization model using a set-based genetic algorithm. We apply the proposed method to test 12 programs, and compare with traditional genetic algorithm and the random method. From the experimental results we can see that, the proposed adequacy criterion is available for the software with randomness and the proposed algorithm can effectively generate test data.

Adaptive random testing (ART) has been proposed to enhance the effectiveness of random testing (RT) through more even spreading of the test cases. In particular, restricted random testing (RRT) is an ART algorithm based on the intuition of skipping all the candidate test cases that are within the neighborhoods (or zones) of previously executed test cases. RRT has higher effectiveness than RT in terms of failure detection but incurs a higher time cost. In this paper, we aim to further reduce the time costs for RRT and improve the effectiveness for RT and ART methods. We propose a proactive technique known as “RRT by largest available zone” (RRT-LAZ). Like RRT, RRT-LAZ first defines an exclusion zone around every executed test case in order to determine the available zones. Unlike the original RRT, RRT-LAZ then compares all the available zones to proactively pick the largest one, from which the next test case is randomly generated. Both simulation analyses and empirical studies have been employed to investigate the efficiency and effectiveness of RRT-LAZ in relation to RT and related ART algorithms. The results show that RRT-LAZ has significantly lower time costs than RRT. Furthermore, RRT-LAZ is more effective than RT and related ART methods for block failure patterns in low-dimensional input spaces. In general, since RRT-LAZ employs a proactive technique instead of a passive one in generating next cases, it is much more cost-effective than RRT. RRT-LAZ is also more cost-effective than RT and other ART methods that we have studied.

This pilot study examines how students' performance has evolved in an Object-oriented (OO) programming course and contributes to the learning analytic framework for similar programming courses in university curriculum. First, we briefly introduce the research background, a novel OO teaching practice with consecutive and iterative assignments consisting of programming and testing assignments. We propose a planned quantitative method for assessing students' gains in terms of programming performance and testing performance. Based on real data collected from students who engaged in our course, we use trend analysis to observe how students' performance has improved over the whole semester. By using correlation analysis, we obtain some interesting findings on how students' programming performance correlates with testing performance, which provides persuasive empirical evidence in integrating software testing practices into an Object-oriented programming curriculum. Then, we conduct an empirical study on how students' design competencies are represented by their program code quality changes over consecutive assignments by analyzing their submitted source code in the course system and the GitLab repository. Three different kinds of profiles are found in the students' program quality in the OO design level. The group analysis results reveal several significant differences in their programming performance and testing performance. Moreover, we conduct systematical explanations on how students' programming skill improvement can be attributed to their object-oriented design competency. By performing principal component analysis on software statistical data, a predictive OO metrics suite for both students' programming performance and their testing performance is proposed. The results show that these quality factors can serve as useful predictors of students' learning performance and can provide effective feedback to the instructors in the teaching practices.

The 3D face authentication has become a hot trend for researchers and developers in the recent years, due to its many advantages over the 2D face recognition feature. The 3D reconstruction of a human face using both near-infrared and depth sensors is a complex process on mobile phones. It commonly involves algorithms like face detection, face landmark detection, facial feature extraction, and depth information analysis. The 3D face authentication feature is critical for the user as per as security and also providing a convenient way to authenticate by the correct user. Therefore, the testing of 3D face authentication algorithms and applications in terms of functionality, performance, and stability is critical. However, the research on 3D face authentication application level validation and testing method is lacking in the field. Most testers are still validating the application manually. In this paper, we propose a robotic-arm-based test automation for testing the 3D face authentication feature on mobile phones. We programmed a 6 degree of freedom (6-DOF) robotic arm to perform 3D face authentication automated tests that were executed manually before. Our test automation also benchmarked the performance of an in-house developed 3D face authentication application and a 3rd party application which yielded promising latency and accuracy comparison results under different performance-impacting test scenarios.

This paper considers the problem of designing a test method and robust test automation for evaluating the functionality of computer vision algorithms for object removal and image reconstruction of the removed area. An object removal algorithm is used to select and remove a region in an image. After the removal of the object, the object removal algorithm regenerates the area in the image from which the object was removed, similar to photoshop. We developed an algorithm in MATLAB that accesses the features of the macroblocks from the region that adjoins the removed object and correlates these macroblocks with the actual area that is regenerated. The SSIM (structural similarity index) score is calculated by comparing the ground truth with the image generated from image reconstruction after removal of the object. Finally, we calculate the artifact area present in the regenerated region to determine the algorithm quality. To validate this method, the authors compare a 3rd-party object removal solution with an in-house developed solution using the proposed approach. This comparison shows that the SSIM score is 1.55% higher than the 3rd-party solution and 26.3% lower in artifact areas. Besides the results, this paper also describes in detail the procedure to automate the use case of object removal test automation to scale up the test coverage.

Blockchain technology promises to spark a real revolution. One of most important concepts associated with this technology is smart contracts, which enable the automatic execution of agreements and augur a world without intermediaries. The conditions and rules of “contracts” are established in a computer codes and trust is enforced by consensus among the participants. One relevant feature associated with smart contract is the immutability property, which establishes the non-alteration of blockchain network data after the clauses of the contract are been approved by all parties or entities involved. For this reason, smart contract development requires more effort and care than the development of other common programs. They require systematic mechanisms to collect requirements and functional specifications. In addition, it is necessary to verify and validate the agreed functionality and the implemented code before they are deployed in the blockchain platform. This article presents a systematic literature review of primary studies in the field of Software Development Life Cycle, focusing on model-based software design and testing in the blockchain domain of smart contracts. This research aims to identify gaps and/or opportunities for further research. After carried out this review, it was observed that no clear methodology exists for evaluating and validating the quality either of this software or the overall development process. This means that software developers may implement smart contract code in which bugs and serious security vulnerabilities appear when the software is delivered to their customers.

The potentially many software system input combinations make exhaustive testing practically impossible. To address this issue, combinatorial t-way testing (where t indicates the interaction strength, i.e. the number of interacting parameters (input)) was adopted to minimize the number of cases for testing. Complimentary to existing testing techniques (e.g. boundary value, equivalence partitioning, cause and effect graphing), combinatorial testing helps to detect faults caused by the faulty interaction between input parameters. In the last 15 years, applications of meta-heuristics as the backbone of t-way test suite generation have shown promising results (e.g. Particle Swarm Optimization, Cuckoo Search, Flower Pollination Algorithm, and Hyper-Heuristics (HHH), to name a few). Supporting the No Free Lunch theorem, as well as potentially offering new insights into the whole process of t-way generation, this article proposes a new strategy with constraint support based on the Whale Optimization Algorithm (WOA). Our work is the first attempt to adopt the WOA as part of a search-based software engineering (SBSE) initiative for t-way test suite generation with constraint support. The experimental results of the test-suite generation indicate that WOA produces competitive outcomes compared to some selected single-based and population-based meta-heuristic algorithms.

Under the software quality management mechanism, developers are generally required to review and test their own code firstly to ensure that the submitted code meets specific quality standards. At the same time, with the popularity of test-driven development (TDD) and extreme programming (XP), programming and testing are complementary in the process of software development, i.e., software testing has become as important as programming. Despite its importance, there is no empirical study that investigates the ability relationships between programming and testing. This article presents such a study, where we designed software tasks to investigate the ability of programming and testing. We distributed the program tasks to software vocational students and analyzed the results from multiple dimensions. Our main findings show that (i) almost half of the developers with strong programming ability do not have a good testing ability; (ii) some developers with weak programming ability can do well in testing; (iii) compared with programming ability, testing fundamentals have a greater impact on the testing ability; and (iv) most developers can do well at finding bugs but lack experience in writing test scripts.

To evaluate heap security, researchers have designed evaluation tools that automatically locate heap vulnerabilities. Most of these tools define heap interactions as heap misuses that are bugs, such as overflow in a target heap allocator, and verify whether each combination of heap interactions can be used as an exploit. However, this definition of heap interactions requires preliminary work by a user possessing evaluation tools and specialized knowledge-the user needs to manually do much work to find which heap misuses exist in the target heap allocator. In addition, because the existing heap misuses vary according to target heap allocators and versions, this preliminary work must be performed on each heap implementation. That is, the current definition of heap interaction cannot be generalized to all heap implementations. In this article, we propose a novel heap security evaluation model, called Heap Security Pilot (HS-Pilot), to overcome the preliminary work load and the dependency of heap misuse in heap implementation. In HS-Pilot, a heap interaction is newly defined as the modification of heap metadata, based on the idea that any heap misuse can be represented by a sequence of heap metadata, i.e. combination of heap interactions used by HS-Pilot. Consequently, the heap interactions in HS-Pilot can be applied to all heap implementations without specialized knowledge, and therefore, are more general than that in existing heap evaluation tools. Our evaluation shows that HS-Pilot can cover the analysis range of other evaluation tools, and is able to detect 14 known types of heap exploitation against heap allocator ptmalloc and all types of heap exploitation found by a state-of-the-art evaluation tool.

Fault localization is indeed tedious and costly work during software maintenance. Studies indicate that combining both structural features and behavior characteristics of programs can be beneficial for improving the efficiency of fault locating. In this paper, we proposed a framework, called Mulr4FL, for fault localization using a multivariate logistic regression model that combined both static and dynamic features collected from the program under debugging. Firstly, the hybrid metrics data set, with both program structural features and behavior characteristics combined, is constructed by static program analyzing and dynamically tracing that runs with a designed metrics set. Meanwhile, the fault information of the legacy program is also obtained from the bug tracking system. Secondly, Bivariate logistic analysis is performed to filter the metrics that are significantly related to faults, and then with the selected metrics and their measurements, a multivariate logistic regression model was constructed and trained. Finally, based on the trained logistic model, we conduct the multivariate logistic analysis on the features of the evolved software and predict the buggy class methods. An empirical study was conducted based on a set of benchmarks that are used widely in program debugging research. The results indicate that the Mulr4FL can significantly improve the effectiveness of locating faults in contrast to 5 baseline techniques.

As an important part of software engineering, software testing is a knowledge-intensive work. In the process of software testing, inconsistent knowledge expression, diverse knowledge carriers, and a few experienced people have mastered most of the knowledge, which hinders the transfer and sharing of domain knowledge. Ontology is widely used in various stages of software engineering to define the semantic relationship between relevant information and knowledge. To solve the problem of knowledge silo in the process of software testing, this article forms an Entire Process Ontology on Software Testing (EPOST). EPOST covers the concepts and relationships of software testing process information, software test object information, and software defect information. The concepts and terms in the ontology are extracted from ISTQB, SWEBOK, IEEE std.829-2008 standard, and IEEE std.610.12-1990 standard. We adopt a comprehensive ontology construction method based on Dev. 101 method and Methontology method. The developed ontology is successfully evaluated by using validation and verification tests. Ontology verification uses an improved FOCA evaluation method by adding a cohesion metric. The evaluation result infers that EPOST has a high quality of ontology and good domain coverage, and achieves the purpose of ontology construction. Finally, we make a case study on the role of EPOST in software testing process. The results show that ontology-based application in the software testing process can promote the sharing and transmission of domain knowledge, and improve the testing process and testing quality.

Ethereum is a kind of blockchain platform where developers may develop and run programs called smart contracts. It inherently relies on gas consumption within a specified allowance to constrain code execution, making every instruction along an execution path to be a location for raising an exception. In this paper, we present GasFuzzer, the first work in exploring the effects of gas allowance manipulation to expose gas-oriented exception security vulnerabilities. GasFuzzer consists of two phases. The first phase introduces a gas-greedy strategy to favor transactions having higher gas consumption for mutation to obtain test transactions with different gas consumptions. The second phase introduces a novel notion of fractional gas consumption coverage and a novel gas-leveling strategy. It applies them to mutate the gas allowances of some of these transactions resulting in the highest gas consumptions produced in the first phase followed by applying these allowance-mutated transactions together with those which remained non-mutated to fuzz test the smart contract. We report an evaluation of GasFuzzer via an experiment on 3170 real-world smart contracts deployed on the public Ethereum Blockchain between October 2017 and July 2019. The findings show that GasFuzzer with gas-greedy strategy can detect more Exceptions Disorder kind of security vulnerabilities (7 more cases) than the previous state-of-the-art black-box fuzzer, and GasFuzzer with gas-leveling strategy and gas coverage criterion can detect 6 additional cases of Exceptions Disorder security vulnerabilities, which is significant.

Generating test cases automatically for thread-safe classes is an effective approach to validating their correctness. However, the existing concurrent test generation techniques usually consume a large amount of time and efforts before finding concurrency bugs. To alleviate this problem, we present an automatic and efficient approach which combines the advantages of both the bug-driven and coverage-guided techniques to generate test cases for thread-safe classes. First, method pairs that cannot be executed concurrently are removed by the static analysis. Then, a strategy of the bug-driven grouping of method pairs is designed to divide the remaining method pairs into two groups. One group contains the method pairs with a high priority, and another group contains the method pairs with a low priority. Finally, iterative generation of concurrent test cases, which consists of the coverage-guided generation of concurrent tests and concurrency bug detection, is conducted to find concurrency bugs. Our evaluation is on 20 thread-safe classes. Compared with four state-of-the-art approaches, the results show that our approach can obtain a significant improvement in efficiency without impairing bug finding capacities.

The main goal of software testing is to detect as many hidden bugs as possible in the final software product before release. Generally, a software product is tested by executing a set of test cases, which can be performed manually or automatically. The number of test cases which are required to test a software product depends on several parameters such as the product type, size, and complexity. Executing all test cases with no particular order can lead to waste of time and resources. Test optimization can provide a partial solution for saving time and resources which can lead to the final software product being released earlier. In this regard, test case selection, prioritization, and scheduling can be considered as possible solutions for test optimization. Most of the companies do not provide direct support for ranking test cases on their own servers. In this paper, we introduce, apply, and evaluate sOrTES as our decision support system for manual integration of test scheduling. sOrTES is a Python-based supportive tool which schedules manual integration test cases which are written in a natural language text. The feasibility of sOrTES is studied by an empirical evaluation which has been performed on a railway use-case at Bombardier Transportation, Sweden. The empirical evaluation indicates that around 40 % of testing failure can be avoided by using the proposed execution schedules by sOrTES, which leads to an increase in the requirements coverage of up to 9.6%.

Background: In order to make informed decisions, software engineering practitioners need information from testing. However, with the trend of increased automation, there is exponential growth and increased distribution of this information. This paper aims at exploring the information flow in software testing in the domain of embedded systems. Method: Data was collected through semi-structured interviews of twelve experienced practitioners with an average work experience of more than fourteen years working at five organizations in the embedded software industry in Sweden. Seventeen hours of audio recordings were transcribed and anonymized into 130 pages of text that was analyzed by means of thematic analysis. Results: The flow of information in software testing can be represented as feedback loops involving stakeholders, software artifacts, test equipment, and test results. The six themes that affect the information flow are how organizations conduct testing and trouble shooting, communication, processes, technology, artifacts, and the organization of the company. Seven main challenges for the flow of information in software testing are comprehending the objectives and details of testing; root cause identification; poor feedback; postponed testing; poor artifacts and traceability; poor tools and test infrastructure; and distances. Finally, five proposed approaches for enhancing the flow are: close collaboration between roles; fast feedback; custom test report automation; test results visualization; and the use of suitable tools and frameworks. Conclusions: The results indicate that there are many opportunities to improve the flow of information in software testing: a first mitigation step is to better understand the challenges and approaches. Future work is needed to realize this in practice, for example, on how to shorten feedback cycles between roles, as well as how to enhance exploration and visualization of test results.

Fuzz testing is a widely used technique for software vulnerability detection, but it is still limited in finding bugs nested in the deep program states. Parallel testing is an augmented method aiming to make the best of the computing resource to expose more deep program bugs. However, current parallel testing methods cannot deal well with task slicing so that the parallel nodes show serious duplication with each other, thus decreasing the efficiency in total. For instance, the original parallel mode of the well-known fuzzer American fuzzy lop (AFL) does not split the task and just synchronizes the interesting seeds without any internal execution information. In this paper, we put forward a novel program state sensitive parallel testing method, which: 1) splits the task into low correlated subtasks according to the program states and 2) adjusts the mutation engine to confine one instance's testing among its subtask-related code region as more as possible. Our method is an objective to reduce the testing collision between parallel instances and therefore improve the performance. We developed a new fuzzer called PAFL and implemented some experiments to investigate that if our parallel testing framework is positive when deploying multiple instances and if it shows a better path discovery compared with two state-of-the-art fuzzers, AFL and AFLFast. In the experiments, we employed PAFL with 1/2/4/8/16 to observe their path discovery and conclude that our new parallel framework is positive when using more multiple instances. We also compared PAFL with AFL and AFLFast by employing eight parallel instances for each fuzzer, and the results prove that our tool has the best path discovery among the three fuzzers. Compared with the original parallel AFL, PAFL can achieve averaged performance gains of 3.98, 3.04, 4.18, and 1.45 on the c++filt, objdump, readelf, and tcpdump, respectively. Besides, we took PAFL on binutils and libtiff, and finally, we found ten new bugs.