Search

Software Industry is evolving at a very fast pace since last two decades. Many software developments, testing and test case generation approaches have evolved in last two decades to deliver quality products and services. Testing plays a vital role to ensure the quality and reliability of software products. In this paper authors attempted to conduct a systematic study of testing tools and techniques. Six most popular e-resources called IEEE, Springer, Association for Computing Machinery (ACM), Elsevier, Wiley and Google Scholar to download 738 manuscripts out of which 125 were selected to conduct the study. Out of 125 manuscripts selected, a good number approx. 79% are from reputed journals and around 21% are from good conference of repute. Testing tools discussed in this paper have broadly been divided into five different categories: open source, academic and research, commercial, academic and open source, and commercial & open source. The paper also discusses several benchmarked datasets viz. Evosuite 10, SF100 Corpus, Defects4J repository, Neo4j, JSON, Mocha JS, and Node JS to name a few. Aim of this paper is to make the researchers aware of the various test case generation tools and techniques introduced in the last 11 years with their salient features.

This paper serves as a guide for both researchers and students who are new to the research area of Search-Based Package Testing. The application of metaheuristic explore methods in this context specifically pertains to utilizing these algorithms for generating test data. In the realm of software engineering research, software testing emerges as a robust and fertile ground for exploration. The integration of AI methods into software program testing is an evolving research direction. Often, newcomers to this field face challenges due to limited knowledge about the interaction between software testing and artificial intelligence. This paper aims to provide a roadmap for these new researchers or students in the field.

This paper presents an exploratory study with Belbin roles, in particular role types, in an individual activity consisting of code review. The objective is to identify if, in addition to any of the three types of roles, the position of the fault and the gender of the subject influence the activity of detecting faults in the code. To create an experimental context for the review process, during a work session the subjects, who were software engineering students, used a code with injected faults for their review. With respect to the types of roles, the results of the experiment do not show significant differences either in the efficiency index obtained by the subjects in the testing process, or with the confusion index of failures. On the other hand, regarding the position of the fault, the results show significant differences between the faults detected in the first half of the code with respect to the remaining second half. Regarding the gender of the subject, the experiment does not show a significant difference between the detected errors. At the end of this report, researchers should perform a short-term analysis of the errors introduced in the code to obtain a better version of the experimental object, allowing them to perform a second controlled experiment under less restrictive conditions.

The occurrence of faults in software systems represents an inevitable predicament. Testing is the most common means to detect such faults; however, exhaustive testing is not feasible for any nontrivial system. Software fault prediction (SFP), which identifies software components that are more prone to errors, seeks to supplement the testing process. Thus, testing efforts can be focused on such modules. Various approaches exist for SFP, with machine learning (ML) emerging as the prevailing methodology. ML-based SFP relies on a wide range of metrics, ranging from file-level and class-level to method-level and even line-level metrics. More granularized metrics are expected to possess a higher degree of micro-level coverage of the code. The Halstead metric suite offers coverage at the line level and has been extensively employed across diverse domains such as fault prediction, quality assessment, and similarity approximation for the past three decades. In this article, we propose to decompose Halstead base metrics and evaluate their fault prediction capability. The Halstead base metrics consist of operators and operands. In the context of the Java language, we partition operators into five distinct categories, i.e., assignment operators, arithmetic operators, logical operators, relational operators, and all other types of operators. Similarly, operands are classified into two classes: constants and variables. For the purpose of empirical evaluation, two experiments were designed. In the first experiment, the Halstead base metrics were used along with McCabe, Lines of Code (LoC), and Halstead-derived metrics as predictors. In the second experiment, decomposed Halstead base metrics were used along with McCabe, LoC, and Halstead-derived metrics. Five public datasets were selected for the experiments. The ML classifiers used included logistic regression, naïve Bayes, decision tree, multilayer perceptron, random forest, and support vector machines. The ML classifiers’ effectiveness was assessed through metrics such as accuracy, F-measure, and AUC. Accuracy saw an enhancement from 0.82 to 0.97, while F-measure exhibited improvement from 0.81 to 0.99. Correspondingly, the AUC value advanced from 0.79 to 0.99. These findings highlight the superior performance of decomposed Halstead metrics, as opposed to the original Halstead base metrics, in predicting faults across all datasets.

Software systems typically have an input domain that can be subdivided into sub-domains, each of which generates similar or related outputs. Testing it on the boundaries between these sub-domains is critical to ensure high-quality software. Therefore, boundary value analysis and testing have been a fundamental part of the software testing toolbox for a long time and are typically taught early to software engineering students. Despite its many argued benefits, boundary value analysis for a given software specification or application is typically described in abstract terms. This allows for variation in how testers apply it and in the benefits they see. Additionally, its adoption has been limited since it requires a specification or model to be analysed. We propose an automated black-box boundary value detection method to support software testers in performing systematic boundary value analysis. This dynamic method can be utilized even without a specification or model. The proposed method is based on a metric referred to as the program derivative, which quantifies the level of boundariness of test inputs. By combining this metric with search algorithms, we can identify and rank pairs of inputs as good boundary candidates, i.e., inputs that are in close proximity to each other but with outputs that are far apart. We have implemented the AutoBVA approach and evaluated it on a curated dataset of example programs. Furthermore, we have applied the approach broadly to a sample of 613 functions from the base library of the Julia programming language. The approach could identify boundary candidates that highlight diverse boundary behaviours in over 70% of investigated systems under test. The results demonstrate that even a simple variant of the program derivative, combined with broad sampling and search over the input space, can identify interesting boundary candidates for a significant portion of the functions under investigation. In conclusion, we also discuss the future extension of the approach to encompass more complex systems under test cases and datatypes.

An essential part of software engineering is testing. A series of pre-testing tasks should be completed before performing testing tasks. Implementing test suites is one of the pre-testing phases. In this work, a case study has been used to implement the suggested method for implementing test suites. The strategy is based on an analysis of UML (Unified Modelling Language) enhanced state chart diagrams (SCDs). UML state chart diagram analysis, AD (activity diagram) conversion into a graph, AG (activity graph) simplification, and test suite implementation are all steps in the generating process. [ABSTRACT FROM AUTHOR]

The software testing phase usually consumes a lot of the development of software projects time in order to find defects before release. Different strategies have been approached to optimize this phase of the testing stage. Metaheuristics are important in software testing due to their ability to find optimal or near-optimal solutions in complex situations. This research aims to analyze the current status of the application of metaheuristics that assist in software testing phase activities, specifically the most representative Non-bio-inspired algorithms (NBA) are surveyed, being Hill Climbing the most reported. The main activities of the software testing where NBA were implemented, were test case and test data generation and test case prioritization. It was concluded that NBAs used on their own are only viable in some activities of the software testing phase. As future work, it is proposed to investigate the use of hybrid algorithms and approaches in software testing phase. [ABSTRACT FROM AUTHOR]

Cross-browser testing not only is one of the most common non-functional testing methods in the field of software testing, but also the testing method that requires large amounts of resources, in terms of hardware and time. Basically, based on Selenium Grid, Kubernetes and KEDA auto-scaler, a cross-browser testing platform can be quickly built. However, through our empirical study of this style of platform, we observed three significant problems in terms of its reliability and efficiency: the Health-Check problem, the Session-Queue problem, and the Cooldown problem. This paper suggests solutions to these problems. The experimental result shows a 2.27 times improvement in reliability and a decrease in execution time for 61.5%. Moreover, the overall execution time is also 54.2% less comparing with Selenium's Dynamic Grid. [ABSTRACT FROM AUTHOR]

A crucial field in software development and testing is Software Defect Prediction (SDP) because the quality, dependability, efficiency, and cost of the software are all improved by forecasting software defects at an earlier stage. Many existing models predict defects to facilitate software testing process for testers. A comprehensive review of these models from different perspectives is crucial to help new researchers enter this field and learn about its latest developments. Algorithms, method types, datasets, and tools were the only perspectives discussed in the current literature. A comprehensive study that takes into account a wide spectrum of viewpoints hasn't yet been published. Examining the development and advancement of SDP-related studies is the goal of this literature review. It provides a comprehensive and updated state-of-the-art that satisfies all stated criteria. Out of 591 papers retrieved from 6 reputable databases, 73 papers were eligible for analysis. This review addresses relevant research questions regarding techniques & method types, data details, tools, code syntax, semantics, structural and domain information. Motivation to conduct this comprehensive review is to equip the readers with the necessary information and keep them informed about the software defect prediction domain. [ABSTRACT FROM AUTHOR]

Research in software testing often involves the development of software prototypes. Like any piece of software, there are challenges in the development, use and verification of such tools. However, some challenges are rather specific to this problem domain. For example, often these tools are developed by PhD students straight out of bachelor/master degrees, possibly lacking any industrial experience in software development. Prototype tools are used to carry out empirical studies, possibly studying different parameters of novel designed algorithms. Software scaffolding is needed to run large sets of experiments efficiently. Furthermore, when using AI-based techniques like evolutionary algorithms, care needs to be taken to deal with their randomness, which further complicates their verification. The aforementioned represent some of the challenges we have identified for this domain. In this paper, we report on our experience in building the open-source EvoMaster tool, which aims at system-level test case generation for enterprise applications. Many of the challenges we faced would be common to any researcher needing to build software testing tool prototypes. Therefore, one goal is that our shared experience here will boost the research community, by providing concrete solutions to many development challenges in the building of such kind of research prototypes. Ultimately, this will lead to increase the impact of scientific research on industrial practice. [ABSTRACT FROM AUTHOR]

Summary: Data flow testing creates test requirements as definition‐use (DU) associations, where a definition is a program location that assigns a value to a variable and a use is a location where that value is accessed. Data flow testing is expensive, largely because of the number of test requirements. Luckily, many DU‐associations are redundant in the sense that if one test requirement (e.g. node, edge and DU‐association) is covered, other DU‐associations are guaranteed to also be covered. This relationship is called subsumption. Thus, testers can save resources by only covering DU‐associations that are not subsumed by other testing requirements. Although this has the potential to significantly decrease the cost of data flow testing, there are roadblocks to its application. Finding data flow subsumptions correctly and efficiently has been an elusive goal; the savings provided by data flow subsumptions and the cost to find them need to be assessed; and the fault detection ability of a reduced set of DU‐associations and the advantages of data flow testing over node and edge coverage need to be verified. This paper presents novel solutions to these problems. We present algorithms that correctly find data flow subsumptions and are asymptotically less costly than previous algorithms. We present empirical data that show that data flow subsumption is effective at reducing the number of DU‐associations to be tested and can be found at scale. Furthermore, we found that using reduced DU‐associations decreased the fault detection ability by less than 2%, and data flow testing adds testing value beyond node and edge coverage. [ABSTRACT FROM AUTHOR]

Software debugging is the process of detecting and removing bugs during software development. Although the intent of modifications to the software is to remove bugs, one cannot rule out the possibility of introducing new bugs as a result of these modifications. We consider a self‐exciting point process, which can incorporate the case of reliability deterioration due to the potential introduction of new bugs to the software during the development phase. In order to account for the unobservable process of introducing bugs, latent variables are incorporated into the self‐exciting point process models. The models are then applied to two data sets in software reliability and additional insights that can be obtained from these models are discussed. Our results suggest that the self‐exciting processes with latent factors perform better than the standard point process models in describing the behavior of software failures during the debugging process. [ABSTRACT FROM AUTHOR]

The article describes a test system designed for verification of the real-time operating system (RTOS) for embedded systems, which was developed and used at the Scientific Research Institute for System Analysis of the Russian Academy of Sciences (SRISA RAS). This Unix-like operating system is based on the POSIX and ARINC-653 programming standards. Of course, there exists specialized software for automation of testing of Unix-like systems: Avocado, LAVA, Linux Test Project, Linux Distribution Checker, Open POSIX Test Suite, UnixBench, etc. But the use of such ready-made software systems is not always convenient, because they either contain only highly specialized test suites, or support only certain hardware, or do not contain a flexible configuration system. Therefore, the researchers at the SRISA RAS developed their own original test system. The task was to create a convenient testing tool for both software testers and programmers. Many years of experience in using the test system has shown the effectiveness of its use to improve the quality of software products, reduction of time spent on testing and analysis of results, maximally automate software testing process, speed up the process of developing new software versions, and simplify the process of debugging, finding and fixing errors by software developers. [ABSTRACT FROM AUTHOR]

A control flow graph (CFG) is a type of directed graph that shows the execution paths of the programs. It is a mathematical structure that is actively used in software testing. It can be constructed from the source or the executable of the program. Construction of the CFG from the executable is called CFG reconstruction. CFG reconstruction is used in many areas of computer science, like reverse engineering, security analysis, and worst-case execution time analysis. CFG reconstruction can be performed using a static, dynamic, or hybrid approach. This paper introduces a new CFG reconstruction tool named Turna that uses a hybrid approach. Turna works on programs that are compiled for RISC-V architecture. One of the main phases of CFG reconstruction is basic block detection. Therefore, together with Turna, a new rule set and an algorithm for basic block detection from RISC-V executables are also introduced. The CFG reconstruction process and the outputs of Turna are shared and discussed. [ABSTRACT FROM AUTHOR]

Software reliability growth models (SRGMs) are used to estimate future failure rates and the number of residual defects in software. SRGM helps reliability software engineers make test termination decisions. Although more than 250 traditional SRGMs have been proposed for reliability assessment, research to develop more reliable models is still ongoing. Recently, new methods based on neural networks have been developed to improve the accuracy of software reliability assessment. The article develops a neural network algorithm for assessing software reliability. To do this, factors affecting software reliability and covering its life cycle were assessed. Based on them, sets for training a three-layer neural network were created. [ABSTRACT FROM AUTHOR]

This paper presents a novel method for testing Boolean expressions. It is based on spectral, aka Fourier analysis of Boolean functions which is exploited to generate test inputs. The approach has three important contributions: (i) It generates a relatively small test suite with a high capability of fault detection, (ii) The test suite is prioritized such that expected fault detection time is shorter, (iii) It is entirely mathematical relying on a simple and straightforward formula. The proposed method is formulated and evaluations are performed on both synthetic and real expressions. It is also compared with two common test generation criteria, MC/DC and Minimal MUMCUT. Evaluations show that the test suite generated by the spectral approach is relatively small while expressing the capability of a better and quicker fault detection. The approach presented in this paper provides a useful insight into how spectral/Fourier analysis of Boolean functions can be exploited in software testing. [ABSTRACT FROM AUTHOR]

Background: When using deep learning models, one of the most critical vulnerabilities is their exposure to adversarial inputs, which can cause wrong decisions (e.g., incorrect classification of an image) with minor perturbations. To address this vulnerability, it becomes necessary to retrain the affected model against adversarial inputs as part of the software testing process. In order to make this process energy efficient, data scientists need support on which are the best guidance metrics for reducing the adversarial inputs to create and use during testing, as well as optimal dataset configurations. Aim: We examined six guidance metrics for retraining deep learning models, specifically with convolutional neural network architecture, and three retraining configurations. Our goal is to improve the convolutional neural networks against the attack of adversarial inputs with regard to the accuracy, resource utilization and execution time from the point of view of a data scientist in the context of image classification. Method: We conducted an empirical study using five datasets for image classification. We explore: (a) the accuracy, resource utilization, and execution time of retraining convolutional neural networks with the guidance of six different guidance metrics (neuron coverage, likelihood-based surprise adequacy, distancebased surprise adequacy, DeepGini, softmax entropy and random), (b) the accuracy and resource utilization of retraining convolutional neural networks with three different configurations (one-step adversarial retraining, adversarial retraining and adversarial fine-tuning). Results: We reveal that adversarial retraining from original model weights, and by ordering with uncertainty metrics, gives the best model w.r.t. accuracy, resource utilization, and execution time. Conclusions: Although more studies are necessary, we recommend data scientists use the above configuration and metrics to deal with the vulnerability to adversarial inputs of deep learning models, as they can improve their models against adversarial inputs without using many inputs and without creating numerous adversarial inputs. We also show that dataset size has an important impact on the results. [ABSTRACT FROM AUTHOR]

Security attacks abuse software vulnerabilities of IoT devices; hence, detecting and eliminating these vulnerabilities immediately are crucial. Fuzzing is an efficient method to identify vulnerabilities automatically, and many publications have been released to date. However, fuzzing for embedded systems has not been studied extensively owing to various obstacles, such as multi-architecture support, crash detection difficulties, and limited resources. Thus, the article introduces fuzzing techniques for embedded systems and the fuzzing differences for desktop and embedded systems. Further, we collect state-of-the-art technologies, discuss their advantages and disadvantages, and classify embedded system fuzzing tools. Finally, future directions for fuzzing research of embedded systems are predicted and discussed. [ABSTRACT FROM AUTHOR]

Blockchain technology has gained significant prominence in the business world. Its impact has been felt in many sectors, but its integration and interoperability remain complex. Many challenges remain, both for users and business consultants, as well as for IT engineers. One of the challenges identified by the research community is the need to provide mechanisms to specify, verify and validate the requirements and business rules that smart contracts must comply with before they can be deployed in a blockchain network. This paper describes a proposal, based on model-driven and useroriented engineering, that aims to obtain functional verification from smart contract specifications in a systematic way. This proposal has been validated in the SmartISBN project, an R+D+i project whose objectives included ensuring the software quality of smart contracts and, above all, improving the traceability of digital publishing (electronic books and journals) using blockchain technology. In this context, this proposal has facilitated the communication between functional experts (authors, publishers, booksellers, etc.) and IT engineers during the specification phase of the global SmartISBN solution and in particular of the smart contracts, as well as during the definition of the functional tests necessary for the validation of the project. [ABSTRACT FROM AUTHOR]

Software testing is the most importance phase for software development life cycle. However, it is always time consuming and costly. In order to solve this problem, regression testing is required to be conducted since it can verify the software modifications with zero effect to the software actual features. Test Case Prioritization (TCP) is one type of regression testing techniques. It can reduce the cost and time taken. In the area of TCP, there are several algorithms. This study will focus on the comparison analysis of prioritization of test case by using Particle Swarm Optimization (PSO) and Firefly Algorithm (FA). In order to choose an algorithm with better performance between PSO and FA, they are converted into Python code. Then, PSO and FA are implemented into Case Study A and Case Study B. Their result will be compared and analyzed based on the execution time, Big-O, and APFD. The comparison showed that FA outperforms than PSO since FA has the least execution time (0.001 second), less complexity (O(N)) than PSO (O(N³)), and similar APFD values (0.520 and 0.600). Thus, FA has better prioritization performance compared to PSO. [ABSTRACT FROM AUTHOR]

Android is a highly fragmented platform with a diverse set of devices and users. To support the deployment of apps in such a heterogeneous setting, Android has introduced dynamic delivery—a new model of software deployment in which optional, device- or user-specific functionalities of an app, called Dynamic Feature Modules (DFMs), can be installed, as needed, after the app’s initial installation. This model of app deployment, however, has exacerbated the challenges of properly testing Android apps. In this article, we first describe the results of an extensive study in which we formalized a defect model representing the various conditions under which DFM installations may fail. We then present DeltaDroid—a tool aimed at assisting the developers with validating dynamic delivery behavior in their apps by augmenting their existing test suite. Our experimental evaluation using real-world apps corroborates DeltaDroid’s ability to detect many crashes and unexpected behaviors that the existing automated testing tools cannot reveal. [ABSTRACT FROM AUTHOR]