Search

This systematic literature review examines the effectiveness of automated test case generation, optimization, and prioritization methods based on Unified Modeling Language (UML) diagrams. The review summarizes the methods, main contributions, and limitations, and suggests areas for future research. This paper examines various optimization algorithms, model-based testing methods, and UML diagram validation methods to determine how well they perform. The review highlights some issues with the current situation, such as the fact that it only examines a few types of UML diagrams and does not go into great detail about how they work or compare to other diagrams. However, it also suggests ways in which these issues could be addressed in future research. Some of the suggested directions include researching different modeling languages and devising solutions to handle the complexity of system models. Model-based testing should also be combined with optimization and prioritization methods to increase the flexibility and usefulness of research in this field. This article makes no direct comparisons to UML diagrams, but it does provide a thorough discussion of the current state of the art and a list of strategic priorities to advance the field of automated test case generation, optimization, and prioritization. These reviews are useful for both researchers and practitioners because they demonstrate how things are currently done and how they should be done in the future. [ABSTRACT FROM AUTHOR]

Modern web services widely provide RESTful APIs for clients to access their functionality programmatically. Fuzzing is an emerging technique for ensuring the reliability of RESTful APIs. However, the existing RESTful API fuzzers repeatedly generate invalid requests due to unawareness of errors in the invalid tested requests and lack of effective strategy to generate legal value for the incorrect parameters. Such limitations severely hinder the fuzzing performance. In this paper, we propose DynER, a new test case generation method guided by dynamic error responses during fuzzing. DynER designs two strategies of parameter value generation for purposefully revising the incorrect parameters of invalid tested requests to generate new test requests. The strategies are, respectively, based on prompting Large Language Model (LLM) to understand the semantics information in error responses and actively accessing API-related resources. We apply DynER to the state-of-the-art fuzzer RESTler and implement DynER-RESTler. DynER-RESTler outperforms foREST on two real-world RESTful services, WordPress and GitLab with a 41.21% and 26.33% higher average pass rate for test requests and a 12.50% and 22.80% higher average number of unique request types successfully tested, respectively. The experimental results demonstrate that DynER significantly improves the effectiveness of test cases and fuzzing performance. Additionally, DynER-RESTler finds three new bugs. [ABSTRACT FROM AUTHOR]

With the widespread application of smart contracts, there is a growing concern over the quality assurance of smart contracts. The data flow testing is an important technology to ensure the correctness of smart contracts. We propose an approach named IABC‐TCG (Improved Artificial Bee Colony‐Test Case Generation) to generate test cases for the data flow testing of smart contracts. With a dominance relations‐based fitness function, an improved artificial bee colony algorithm is used to generate test cases, in which the bee colony search coefficient is adaptively adjusted to improve the effectiveness and efficiency of the search. In addition, an improved test case selection and updation strategy is used to avoid unnecessary test cases. The experimental results show that IABC‐TCG achieves 100% coverage for all the test requirements on a dataset of 30 smart contracts and outperforms the baseline approaches in terms of the number of test cases and the execution time. Performing tests with the generated test cases, IABC‐TCG can find more errors with less test cost. [ABSTRACT FROM AUTHOR]

Insufficient safety requirements and the large scale of test cases are the two main problems facing the on-board equipment testing process of the fully automatic operation system. This thesis proposes a security test method based on system-theoretic accident modeling and process theory and combinatorial test for the on-board equipment of the fully automatic operation system. First of all, based on the system-theoretic accident modeling and process theory, a layered control structure model on the on-board equipment is established, security constraints from the perspective of measurement is extracted, and the safety testing characteristics covered by security constraints are analyzed. Then, based on the best coverage of array and seed test cases and combined with security constraints, the genetic algorithm is used to reconstruct the optimal coverage number, and expand the streamlined test cases that meet the constraints. Finally, the fully automatic operation system platform departure scene is taken as an example to verify the effectiveness of the test algorithm of the vehicle equipment. The results show that in the case of the same combined coverage intensity, the test cases in the method proposed in this thesis are reduced by 16. 45% than the test cases generated by the existing mainstream methods, which has certain practical significance for reducing the cost of vehicle equipment testing and improving the test efficiency. [ABSTRACT FROM AUTHOR]

Mobile applications can recognize their computational setting and adjust and respond to actions in the context. This is known as context-aware computing. Testing context-aware applications is difficult due to their dynamic nature, as the context is constantly changing. Most mobile testing tools and approaches focus only on GUI events, adding to the deficient coverage of applications throughout testing. Generating test cases for various context events in Android applications can be achieved using reinforcement learning algorithms. This study proposes an approach for generating Android application test cases based on Expected State-Action-Reward-State-Action (E-SARSA), considering GUI and context events for effective testing. The proposed method was experimentally evaluated on eight Android applications, showing 48- 96% line of code coverage across them, which was higher than Q-testing and SARSA. [ABSTRACT FROM AUTHOR]

Software development would have to include automated testing to ensure the finished product and performs as intended. However, the process of test case generation and maintenance can be time-consuming and error-prone, especially when manual methods are used. This research proposes a new approach to improve the efficiency and accuracy of automated testing using latent semantic analysis (LSA)-based TextRank (TR) and particle swarm optimization (PSO) algorithms. The study aims to evaluate the effectiveness of these algorithms in generating and optimizing test cases based on requirements analysis. To retrieve key information from the criteria, methods including text classification (TC), named entity recognition (NER), and sentiment analysis (SA) are used to evaluate the text. Test cases are then generated using LSA-based TR for text summarization and PSO for optimization. The aim of this work is to identify any limitations that need to be addressed and to evaluate the overall efficiency and accuracy of automated testing (AT) using proposed algorithms. The results of this research are expected to have important implications for the software industry, helps to improve the overall efficiency and accuracy of AT. The findings could guide future research that led to the creation of more advanced and effective tools for AT. [ABSTRACT FROM AUTHOR]

Cooperative software validation aims at having verification and/or testing tools cooperate on the task of correctness checking. Cooperation involves the exchange of information about currently achieved results in the form of (verification) artifacts. These artifacts are typically specialized to the type of analysis performed by the tool, e.g., bounded model checking, abstract interpretation or symbolic execution, and hence require the definition of a new artifact for every new cooperation to be built. In this article, we introduce a unified artifact (called Generalized Information Exchange Automaton, short GIA) supporting the cooperation of over-approximating with under-approximating analyses. It provides information gathered by an analysis to its partner in a cooperation, independent of the type of analysis and usage context within software validation. We provide a formal definition of this artifact in the form of an automaton together with two operators on GIAs. The first operation reduces a program by excluding these parts, where the information that they are already processed is encoded in the GIA. The second operation combines partial results from two GIAs into a single on. We show that computed analysis results are never lost when connecting tools via these operations. To experimentally demonstrate the feasibility, we have implemented two such cooperation: one for verification and one for testing. The obtained results show the feasibility of our novel artifact in different contexts of cooperative software validation, in particular how the new artifact is able to overcome some drawbacks of existing artifacts. [ABSTRACT FROM AUTHOR]

Pairwise testing is a method for identifying defects through combinatorial analysis. It involves testing all possible combinations of input parameters in pairs within a system, ensuring that each pair is tested at least once. The field of test case generation is highly active in the realm of combinatorial interaction testing. Research in this area is particularly encouraged, as it falls under the category of non-deterministic polynomial-time hardness. A big challenge in this field is the combinatorial explosion problem. It is about finding the best test suite that covers all possible combinations of interaction strength. In this paper, we present the task of discovering a pairwise test set as a search problem and introduce an innovative testing tool referred to as pairwise test case generation in harmony search algorithm with seeding and constraint mechanism (PHOSC). Experimental results show that PHOSC performs better compared to some existing pairwise strategies in terms of test suite size. Additionally, PHOSC provides a comprehensive framework and serves as a research platform for the generation of pairwise test sets employing the harmony search algorithm. It adopts an approach that focuses on one parameter at a time (OPAT) and incorporates seeding and constraint mechanisms at the same time, thereby enhancing the efficiency and effectiveness of the testing process. [ABSTRACT FROM AUTHOR]

With the continuous development of Internet technology, the functional requirements of all kinds of business software are increasing, and their complexity is gradually increasing. The reliability and security of software have attracted more and more attention. Software testing is the key technology for software quality assurance. Due to the frequent changes in requirements and rapid version iteration of modern business software products, manually writing test cases for them will consume a lot of manpower costs, especially in agile development processes where regression testing and other tasks require a large number of repeated use cases. A test case automatic generation model based on genetic algorithm and LightGBM model is proposed using machine learning technology. The innovative contribution lies in: 1 Abstracting the test steps into a directed graph model, the test case data were simplified; 2 The genetic algorithm was used to solve reachable paths in directed graphs, replacing manually generated test paths; 3 Using the LightGBM model to accelerate the convergence speed of the genetic algorithm, the effectiveness of the proposed method was experimentally verified, meeting the testing coverage criteria. This model can reduce the workload of testers, accelerate testing speed, and is of great significance for improving project quality and accelerating project progress. [ABSTRACT FROM AUTHOR]

This paper goal is to provide a more effective algorithm for reducing the amount of test cases in a test suit during the regression testing phase. This algorithm divides the entire test suit into equivalence classes at first then apply boundary value coverage to select test case out of repeated test cases with same importance in suit. This algorithm is based on the concept that before selecting best test cases out of repeated test case in test suit to prepare reduced test suit we can divide all test cases in number of equivalence classes to reduce test cases by great extent. This paper proposed a method of experimentation involving test cases from different software application areas. Furthermore, we discuss a case study to verify and check new algorithm for its efficiency, for that we apply our algorithm on one of the program or group of programs. This complete proposed methodology shall be applied to different software applications belonging to soft computing, engineering software, and financial software. This wok has not been done earlier as found in the literature survey performed by us. Minimization in test cases would lead to lesser testing effort and desirable test completion. [ABSTRACT FROM AUTHOR]

Software testing is an important phase as it ensures the software quality. The software testing process comprises of three steps: generation, execution, and evaluation of test cases. Literature claims the usage of single and multiple 'Unified Modeling Language' (UML) diagrams to generate test cases. Using multiple UML diagrams increases test case coverage. However, the existing approaches show limitations in test case generation from UML diagrams. Therefore, in this research study, we propose an approach to generate the test cases using UML State Chart Diagram (SCD), Activity Diagram (AD), and Sequence Diagram (SD). The proposed approach transforms UML diagrams into intermediate forms: SCD Graph, AD Graph, and SD Graph respectively. Furthermore, by integrating these three graphs, a System Testing Graph (STG) is formed. Finally, test cases are identified from STG by using a traversal algorithm such as Depth First Search (DFS) that is an optimization method. The results show that the proposed approach is better compared to existing approaches in terms of coverage and performance. Moreover, the generated test cases have the ability to detect faults at the unit level, integration, and system level testing. [ABSTRACT FROM AUTHOR]

New approaches to software testing are required due to the rising complexity of today’s software applications and the rapid growth of software engineering practices. Among these methods, one that has shown promise is the introduction of Natural Language Processing (NLP) tools to software testing practices. NLP has witnessed a rise in popularity within all IT fields, especially in software engineering, where its use has improved the way we extract information from textual data. The goal of this systematic literature review (SLR) is to provide an in-depth analysis of the present body of the literature on the expanding subject of NLP-based software testing. Through a repeatable process, that takes into account the quality of the research, we examined 24 papers extracted from Web of Science and Scopus databases to extract insights about the usage of NLP techniques in the field of software testing. Requirements analysis and test case generation popped up as the most hot topics in the field. We also explored NLP techniques, software testing types, machine/deep learning algorithms, and NLP tools and frameworks used in the studied body of literature. This study also stressed some recurrent open challenges that need further work in future research such as the generalization of the NLP algorithm across domains and languages and the ambiguity in the natural language requirements. Software testing professionals and researchers can get important insights from the findings of this SLR, which will help them comprehend the advantages and challenges of using NLP in software testing.

In the domain of software testing, the generation of test cases is a critical process for detecting system errors and bugs. However, automated test case generation for smart contracts often encounters challenges related to automation, vulnerability diversity, and coverage. This paper presents a novel method, the self-adaptive learning Genetic Algorithm (self-adaptive learning GA), designed to address these issues. Our research methodology incorporates several construction models, namely the Control Dependence Graph (CDG), Control Flow Graph (CFG), and Application Binary Interface (ABI). Initially, the ABI model provides essential information for generating and executing test cases. The CFG model subsequently visualizes potential execution paths through the functions of smart contracts. Ultimately, the CDG model identifies potential vulnerabilities in smart contracts. Using these models, our method enhances automatic test case generation in smart contracts by improving coverage and reducing execution time. We selected a variety of smart contracts from the Decentralized Finance (DeFi) ecosystem for data collection and comparative analysis. The experimental results show superior performance rates, with an average code coverage rate of 98.1%, a total of 3500 vulnerabilities detected, a vulnerability detection rate of 98.7%, a false positive rate of 1.3%, a recall of 98.2%, precision of 98.8%, a path uniqueness rate of 96.4%, false negative rate of 3.5%, an execution time of 25 s, and test case generation time of 16 s. In conclusion, our proposed approach demonstrates a significant improvement over existing methods for test case generation by providing a promising solution for the robustness of smart contracts and security enhancement in the DeFi ecosystem. [ABSTRACT FROM AUTHOR]

The REST API has become an important way to access and use cloud services, Web, and mobile applications. It is urgent to solve the problem of how to carry out automated testing on these APIs to ensure the security and reliability of services. Although there are numerous research results on REST API automation testing, there is still a lack of comprehensive analysis and summarization of testing techniques. This paper firstly summarized the development history of REST API automation testing, then refined the general process of testing by combining the features of REST API automation testing. Next, this paper elaborated the technical features of existing achievements from four aspects: pre-processing, test case generation, test case execution and monitoring, and result analysis, and compared and analyzed their advantages and disadvantages. Finally, it discussed the shortcomings of the current research, proposed possible solutions, and explored the next research direction. [ABSTRACT FROM AUTHOR]

Context: Computational materials science (CMS) focuses on in silico experiments to compute the properties of known and novel materials, where many software packages are used in the community. The NOMAD Laboratory (Draxl C, Scheffler) offers to store the input and output files in its FAIR data repository. Since the file formats of these software packages are non‐standardized, parsers are used to provide the results in a normalized format. Objective: The main goal of this article is to report experience and findings of using grammar‐based fuzzing on these parsers. Method: We have constructed an input grammar for four common software packages in the CMS domain and performed an experimental evaluation on the capabilities of grammar‐based fuzzing to detect failures in the Novel Materials Discovery (NOMAD) parsers. Results: With our approach, we were able to identify three unique critical bugs concerning service availability, as well as several additional syntactic, semantic, logical, and downstream bugs in the investigated NOMAD parsers. We reported all issues to the developer team prior to publication. Conclusion: Based on the experience gained, we can recommend grammar‐based fuzzing also for other research software packages to improve the trust level in the correctness of the produced results. [ABSTRACT FROM AUTHOR]

Search-based Test Case Generation (TCG) for web applications suffers from unstable performance and suboptimal test suite problems due to diversity loss. However, previous diversity metrics mainly only focus on client-side models or server-side code, which are prone to low robustness and poor generalization in practical applications. We propose a diversity-driven TCG method DTester, which can maximize behavior exploration and minimize the test suite size while covering more server-side vulnerable paths. Three diversity metrics (i.e. phenotypic coupling, intent coupling and competitiveness) are proposed to measure the underlying relationship between test cases from user behavior, code logic and test execution history. Moreover, a 3-dimensional weight graph is designed to model association among metrics, which provides fine-grained guidance for the genetic algorithm to generate diverse test cases from the client-side behavior model. Our empirical evaluation on five web applications shows that DTester can efficiently and robustly generate better test suites than the state-of-the-art TCG method. The maximum improvement is 8 5 % , 6 0 % , 7 1 6 % and 8 8 % in efficiency, test suite size, diversity and robustness. [ABSTRACT FROM AUTHOR]

In applications of software testing, the cause–effect graph method is an approach often used to design test cases by analyzing various combinations of inputs with a graphical approach. However, not all inputs have equal impacts on the results, and approaches based on exhaustive testing are generally time-consuming and laborious. As a statute-based software-testing method, combinatorial testing aims to select a small but effective number of test cases from the large space of all possible combinations of the input values for the software to be tested, and to generate a set of test cases with a high degree of coverage and high error detection capability. In this paper, the reduced ordered binary decision diagram is utilized to simplify the cause–effect graph so as to reduce the numbers of both the inputs and test cases, thereby saving the testing cost. In addition, an improved particle swarm optimization algorithm is proposed to significantly reduce the computation time needed to generate test cases. Experiments on several systems show that the proposed method can generate excellent results for test case generation. [ABSTRACT FROM AUTHOR]

In industry, RESTful APIs are widely used to build modern Cloud Applications. Testing them is challenging, because not only do they rely on network communications, but also they deal with external services like databases. Therefore, there has been a large amount of research sprout in recent years on how to automatically verify this kind of web services. In this article, we present a comprehensive review of the current state-of-the-art in testing RESTful APIs based on the analysis of 92 scientific articles. These articles were gathered by utilizing search queries formulated around the concept of RESTful API testing on seven popular databases. We eliminated irrelevant articles based on our predefined criteria and conducted a snowballing phase to minimize the possibility of missing any relevant paper. This survey categorizes and summarizes the existing scientific work on testing RESTful APIs and discusses the current challenges in the verification of RESTful APIs. This survey clearly shows an increasing interest among researchers in this field, from 2017 onward. However, there are still a lot of open research challenges to overcome. [ABSTRACT FROM AUTHOR]

Various software testing techniques have been shown to be successful in producing high-quality test suites for software where the code is not accessible (black-box approach). Nevertheless, no method has been found to guide combining some of these in a general way. In this study, a test suite generation method for black-box software called CMBMeTest was created to respond to these challenges. It employs several coupled software testing techniques, namely, model-based testing (MBT), constraint programming (CP), and metamorphic testing (MT). CMBMeTest provides step-by-step instructions for using the information available (such as program specifications, inputs and outputs) to create an initial test suite that covers the model obtained, using a combination of MBT and CP (referred to as MBT+CP). Furthermore, using the metamorphic relations (MRs) of MT, a better test suite was produced from that initial test suite. The method allows particular stages to be iterated to improve the results by building new models and new MRs. A comprehensive case study was conducted, employing CMBMeTest to produce encouraging results. Mutation testing was used to evaluate the test suite, and the first round produced a high mutation score. A more detailed model was used to repeat the process, with similar outcomes. [ABSTRACT FROM AUTHOR]

In this paper, we propose a method for the generation of simulated test scenarios for autonomous driving. Based on the requirements of standard regulatory test scenarios, we can generate virtually simulated scenarios and functional scenario libraries for autonomous driving, which can be used for the simulated verification of different ADAS functions. Firstly, the operational design domain (ODD) of a functional scenario is selected, and the weight values of the ODD elements are calculated. Then, a combination test algorithm based on parameter weights is improved to generate virtually simulated autonomous driving test cases for the ODD elements, which can effectively reduce the number of generated test cases compared with the traditional combination test algorithm. Then, the traffic participant elements in each test case are sampled and clustered so as to obtain hazard-specific scenarios. Then, the values of the subelements under the traffic participant element in each test case are sampled and clustered to obtain hazard-specific scenarios. Finally, the specific scenarios are applied to the automatic emergency braking (AEB) system on the model-in-the-loop (MIL) testbed to verify the effectiveness of this scenario generation method. [ABSTRACT FROM AUTHOR]

The convergence rate has been widely accepted as a performance measure for choosing a better metaheuristic algorithm. So, we propose a novel technique to improve the performance of the existing Grey Wolf Optimization (GWO) algorithm in terms of its convergence rate. The proposed approach also prioritizes the test cases that are obtained after executing the input benchmark programs. This paper has three technical contributions. In our first contribution, we generate test cases for the input benchmark programs. Our second contribution prioritizes test cases using an improved version of the existing GWO algorithm (CGWO). Our third contribution analyzes the obtained result and compares it with state-of-the-art metaheuristic techniques. This work is validated after running the proposed model on six benchmark programs. The obtained results show that our proposed approach has achieved 48% better APFD score for the prioritized order of test cases than the non-prioritized order. We also achieved a better convergence rate, which takes around 4000 fewer iterations, when compared with the existing methods on the same platform. [ABSTRACT FROM AUTHOR]

This paper introduces a new variant of a metaheuristic algorithm based on the whale optimization algorithm (WOA), the Q-learning algorithm and the Exponential Monte Carlo Acceptance Probability called (QWOA-EMC). Unlike WOA, QWOA-EMC permits just-in-time adaptive selection of its operators (i.e., between shrinking mechanism, spiral shape mechanism, and random generation) based on their historical performances as well as exploits the Monte Carlo Acceptance probability to further strengthen its exploration capabilities by allowing a poor performing operator to be reselected with probability in the early part of the iteration. Experimental results for constraints combinatorial test generation demonstrate that the proposed QWOA-EMC outperforms WOA and performs competitively against other metaheuristic algorithms. [ABSTRACT FROM AUTHOR]

Summary: Model‐based test design is increasingly being applied in practice and studied in research. Model‐based testing (MBT) exploits abstract models of the software behaviour to generate abstract tests, which are then transformed into concrete tests ready to run on the code. Given that abstract tests are designed to cover models but are run on code (after transformation), the effectiveness of MBT is dependent on whether model coverage also ensures coverage of key functional code. In this article, we investigate how MBT approaches generate tests from model specifications and how the coverage of tests designed strictly based on the model translates to code coverage. We used snowballing to conduct a systematic literature review. We started with three primary studies, which we refer to as the initial seeds. At the end of our search iterations, we analysed 30 studies that helped answer our research questions. More specifically, this article characterizes how test sets generated at the model level are mapped and applied to the source code level, discusses how tests are generated from the model specifications, analyses how the test coverage of models relates to the test coverage of the code when the same test set is executed and identifies the technologies and software development tasks that are on focus in the selected studies. Finally, we identify common characteristics and limitations that impact the research and practice of MBT: (i) some studies did not fully describe how tools transform abstract tests into concrete tests, (ii) some studies overlooked the computational cost of model‐based approaches and (iii) some studies found evidence that bears out a robust correlation between decision coverage at the model level and branch coverage at the code level. We also noted that most primary studies omitted essential details about the experiments. [ABSTRACT FROM AUTHOR]

AbstractThe generation of test cases to verify and validate the actions of software or an application, as per the customers’ requirements, is an indispensable activity in software industries. A tester could construct test cases to suffice various objectives, which could be random or task-oriented at times. Most of the time, test cases are generated based on clients’ specifications or requirements. These requirements are structured in natural language, and manual derivation of test cases from such client-stated requirements could be a cumbersome and time-absorbing activity for testers. Until recently, many practitioners have proposed a natural language processing (NLP)-oriented solution to automate or semi-automate the manual process of generating test cases from requirements; nevertheless, such studies imposed a restriction on how the clients should document or represent their requirements. This study, on the contrary, suggested an NLP solution that considers free-format user requirements and applies text pre-processing, a combination of dependency parser and RAKE process, along with a statistical similarity measure and template-based natural language generation (NLG) to translate them into detailed test cases. Apart from test case generation, with the aid of NLP tactics, this study has also proposed a solution for encoding the historical data of test cases into numerical values. Such numerical scores serve as valuable data and create the proper insight for testers during test case optimization.

Fault localization, a critical process of software debugging, can be implemented by ranking program statements according to their suspiciousness of being faulty, which, in turn, is calculated based on the execution behaviors of test cases. The performance of fault localization will deteriorate if the actual faulty statement is ranked low in suspiciousness. Intuitively speaking, the quality of the used test cases affects the suspiciousness ranking and thus the efficacy of fault localization. As such, it is necessary to generate test cases with "better" quality to improve the chance of faulty statements being ranked as highly suspicious. In this paper, we propose a software fault localization approach based on feedback ranking information, namely FLFR, according to an improved genetic algorithm. The starting point of the new method is the execution of a set of test cases, which gives a preliminary suspiciousness ranking of program statements. The improved genetic algorithm is iteratively applied to generate new test cases. The new method is evaluated through a series of experiments on four C programs and two Java programs. The experimental results show that the test cases automatically generated by the method can improve the suspiciousness ranking of the faulty statement, and thus enhance the performance of fault localization. [ABSTRACT FROM AUTHOR]

Software Industry is evolving at a very fast pace since last two decades. Many software developments, testing and test case generation approaches have evolved in last two decades to deliver quality products and services. Testing plays a vital role to ensure the quality and reliability of software products. In this paper authors attempted to conduct a systematic study of testing tools and techniques. Six most popular e-resources called IEEE, Springer, Association for Computing Machinery (ACM), Elsevier, Wiley and Google Scholar to download 738 manuscripts out of which 125 were selected to conduct the study. Out of 125 manuscripts selected, a good number approx. 79% are from reputed journals and around 21% are from good conference of repute. Testing tools discussed in this paper have broadly been divided into five different categories: open source, academic and research, commercial, academic and open source, and commercial & open source. The paper also discusses several benchmarked datasets viz. Evosuite 10, SF100 Corpus, Defects4J repository, Neo4j, JSON, Mocha JS, and Node JS to name a few. Aim of this paper is to make the researchers aware of the various test case generation tools and techniques introduced in the last 11 years with their salient features.

Test case generation techniques based on adversarial examples are commonly used to enhance the reliability and robustness of image‐based and text‐based machine learning applications. However, efficient techniques for speech recognition systems are still absent. This paper proposes a family of methods that generate targeted adversarial examples for speech recognition systems. All are based on the firefly algorithm (F), and are enhanced with gauss mutations and / or gradient estimation (F‐GM, F‐GE, F‐GMGE) to fit the specific problem of targeted adversarial test case generation. We conduct an experimental evaluation on three different types of speech datasets, including Google Command, Common Voice and LibriSpeech. In addition, we recruit volunteers to evaluate the performance of the adversarial examples. The experimental results show that, compared with existing approaches, these approaches can effectively improve the success rate of the targeted adversarial example generation. The code is publicly available at https://github.com/HanboCai/FGMGE. [ABSTRACT FROM AUTHOR]

The generation of test cases to verify and validate the actions of software or an application, as per the customers' requirements, is an indispensable activity in software industries. A tester could construct test cases to suffice various objectives, which could be random or task-oriented at times. Most of the time, test cases are generated based on clients' specifications or requirements. These requirements are structured in natural language, and manual derivation of test cases from such client-stated requirements could be a cumbersome and time-absorbing activity for testers. Until recently, many practitioners have proposed a natural language processing (NLP)-oriented solution to automate or semi-automate the manual process of generating test cases from requirements; nevertheless, such studies imposed a restriction on how the clients should document or represent their requirements. This study, on the contrary, suggested an NLP solution that considers free-format user requirements and applies text pre-processing, a combination of dependency parser and RAKE process, along with a statistical similarity measure and template-based natural language generation (NLG) to translate them into detailed test cases. Apart from test case generation, with the aid of NLP tactics, this study has also proposed a solution for encoding the historical data of test cases into numerical values. Such numerical scores serve as valuable data and create the proper insight for testers during test case optimization. [ABSTRACT FROM AUTHOR]

Summary: Combinatorial test generation, also called t‐way testing, is the process of generating sets of input parameters for a system under test, by considering interactions between values of multiple parameters. In order to decrease total testing time, there is an interest in techniques that generate smaller test suites. In our previous work, we used graph techniques to produce high‐quality test suites. However, these techniques require a lot of computing power and memory, which is why this paper investigates distributed computing for t‐way testing. We first introduce our distributed graph colouring method, with new algorithms for building the graph and for colouring it. Second, we present our distributed hypergraph vertex covering method and a new heuristic. Third, we show how to build a distributed IPOG algorithm by leveraging either graph colouring or hypergraph vertex covering as vertical growth algorithms. Finally, we test these new methods on a computer cluster and compare them to existing t‐way testing tools. [ABSTRACT FROM AUTHOR]

Machine learning (ML) may enable effective automated test generation. We characterize emerging research, examining testing practices, researcher goals, ML techniques applied, evaluation, and challenges in this intersection by performing. We perform a systematic mapping study on a sample of 124 publications. ML generates input for system, GUI, unit, performance, and combinatorial testing or improves the performance of existing generation methods. ML is also used to generate test verdicts, property‐based, and expected output oracles. Supervised learning—often based on neural networks—and reinforcement learning—often based on Q‐learning—are common, and some publications also employ unsupervised or semi‐supervised learning. (Semi‐/Un‐)Supervised approaches are evaluated using both traditional testing metrics and ML‐related metrics (e.g., accuracy), while reinforcement learning is often evaluated using testing metrics tied to the reward function. The work‐to‐date shows great promise, but there are open challenges regarding training data, retraining, scalability, evaluation complexity, ML algorithms employed—and how they are applied—benchmarks, and replicability. Our findings can serve as a roadmap and inspiration for researchers in this field. [ABSTRACT FROM AUTHOR]

UML designs represent sequential and concurrent behaviors of the systems in terms of activity, sequence, and state machine diagrams. Concurrent systems are non-deterministic in nature. Therefore, test case generation and coverage criterion definitions are challenging for a concurrent system using UML designs. Concurrent system testing is exponential due to the execution of parallel constructs and interleavings of messages inside parallel constructs. Hence, there is a necessity for appropriate concurrency coverage criteria to work with a finite number of test scenarios. This research explores relevant concurrency coverage criteria for UML sequence diagrams and their definitions to work in real-life usages. This paper proposes a message control flow graph (MCFG), a richer form of a control flow graph, as an intermediate representation for UML sequence diagrams. The proposed MCFG facilitates defining new coverage criteria for message interleaving inside concurrency constructs, like parallel and critical region combined fragments and general ordering. We define new coverage criteria, namely interleaving message path coverage, synchronization coverage, and essential concurrent message path coverage. The empirical results show that the proposed scheme provides a good trade-off between cost and defect exposing capability of the concurrent design modeled by UML sequence diagrams. [ABSTRACT FROM AUTHOR]

Automated test case generation technology aims to reduce test costs.Compared with manual test generation,it has higher test efficiency.Most existing testing tools treat all files in the software equally,but in fact,files with defects account for only a small part of the whole code.Therefore,if testers can detect files that are more prone to defects,they can greatly save testing resources.To solve the above problems,this paper designs a predictive guidance test tool AutoUnit,which is based on active learning.We first predict the defect files in the whole file pool to be detected.Next,we use the detection tool to detect the most “suspicious” files.Then we feed back the actual detection results to the prediction model and update the model to enter the next round of prediction.In addition,when the total number of defective files is unknown,AutoUnit can stop in time by setting diffe-rent target recall rates.It can predict the total number of defective files according to the tested files,calculate the current recall rate,judge whether to stop predict guidance and ensure testing efficiency.Experimental analysis shows that when the same number of defect files are tested,the shortest time and the longest time taken by AutoUnit is 70.9% and 80.7% of the current mainstream testing tools,respectively.When the total number of defective files is unknown and the target recall rate is set to 95%,compared with the latest version of Evosuite,AutoUnit only needs to check 29.7% of the source code files to achieve the same detection level,and its test time is only 34.6% of Evosuite,the cost of testing is greatly reduced.Experimental results show that the method effectively improves the efficiency of test.

Web services have been widely used to develop complex distributed software systems in the context of Service Oriented Architecture (SOA). As a standard for describing Web services, the Web Service Description Language (WSDL) provides a universal mechanism to describe the service’s functionalities for the service consumers. However, the current WSDL only provides the description of the interfaces to a Web Service without any restrictions or assumptions on how to properly invoke the service, resulting in divergent understanding of the Web service’s behavior between the service developer and service consumer. A particular challenge is how to make explicit the various behavior assumptions and restrictions of a service (for the user), and make sure that the service implementation conforms to them (for the developer). In this article, we propose a constraint-based model-driven approach to improving the behavior conformance of Web services. In our approach, constraints are introduced in an extended WSDL, called CxWSDL, to formally and explicitly express the implicit restrictions and assumptions on the behavior of a Web service, and then the predefined constraints are used to derive test cases in a model-driven manner to test the service implementation’s conformance to its behavior constraints from the user’s perspective. An empirical study involving four real-life Web services was conducted to evaluate the effectiveness of our approach, and four actual inconsistencies were discovered. [ABSTRACT FROM AUTHOR]