Search

Industrial robotic systems (IRS) consist of industrial robots that automate industrial processes. They accurately perform repetitive tasks, replacing or assisting with dangerous jobs like assembly in the automotive and chemical industries. Failures in these systems can be catastrophic, so it is important to ensure their quality and safety before using them. One way to do this is by applying a software testing process to find faults before they become failures. However, software testing in industrial robotic systems has some challenges. These include differences in perspectives on software testing from people with diverse backgrounds, coordinating and collaborating with diverse teams, and performing software testing within the complex integration inherent in industrial environments. In traditional systems, a well-known development process uses simple, structured sentences in English to facilitate communication between project team members and business stakeholders. This process is called behavior-driven development (BDD), and one of its pillars is the use of templates to write user stories, scenarios, and automated acceptance tests. We propose a software testing (ST) approach called automated acceptance testing for industrial robotic systems (AAT4IRS) that uses natural language to write the features and scenarios to be tested. We evaluated our ST approach through a proof-of-concept, performing a pick-and-place process and applying mutation testing to measure its effectiveness. The results show that the test suites implemented using AAT4IRS were highly effective, with 79% of the generated mutants detected, thus instilling confidence in the robustness of our approach. [ABSTRACT FROM AUTHOR]

Human–Machine Pair Programming (HMPP) is a promising technique in the software development process, which means that software construction can be done in the manner that humans are responsible for developing the program while computer is responsible for monitoring the program in real-time and reporting errors. The Java runtime exceptions in the current version of the software under construction can only be effectively detected by means of its execution. Traditional software testing techniques are suitable for testing completed programs but face a challenge in building a suitable testing environment for testing the partial programs produced during HMPP. In this paper, we put forward a novel technique, called Program Segment Testing (PST) for automatically identifying errors caused by runtime exceptions to support HMPP. We first introduce the relevant involved in this technique to detect index out of bounds exceptions, a representative of runtime exceptions. Then we discuss the methodology of this technique in detail and illustrate its workflow with a simple case study. Finally, we carry out an experiment to evaluate this technique and compare it with three existing fault detection techniques using several programs to demonstrate its effectiveness. [ABSTRACT FROM AUTHOR]

The rapid proliferation of network applications has led to a significant increase in network attacks. According to the OWASP Top 10 Projects report released in 2021, injection attacks rank among the top three vulnerabilities in software projects. This growing threat landscape has increased the complexity and workload of software testing, necessitating advanced tools to support agile development cycles. This paper introduces a novel test prioritization method for SQL injection vulnerabilities to enhance testing efficiency. By leveraging previous test outcomes, our method adjusts defense strength vectors for subsequent tests, optimizing the testing workflow and tailoring defense mechanisms to specific software needs. This approach aims to improve the effectiveness and efficiency of vulnerability detection and mitigation through a flexible framework that incorporates dynamic adjustments and considers the temporal aspects of vulnerability exposure. [ABSTRACT FROM AUTHOR]

Software testing is an effective means of ensuring software quality. The cost of software testing is the main component of the total cost of software development. The generation of test data is very important in testing, because the efficiency of testing depends on the test data used. A significant part of software testing automation is the test data generation. How to automatically generate test datasets is still an open problem. Search-based software testing (SBST) is a process of generating test data that employs meta-heuristic methods to resolve tough NP-hard problems. The test data generation problem belongs to NP-hard problems. This paper focuses on the automatic generation of test data for path coverage based on control flow criteria in structural testing. Some meta-heuristic algorithms have been used to search for software test data. To advance the efficiency and effectiveness of path test data generation, an adaptive simulated annealing particle swarm optimization (ASAPSO) algorithm is proposed. The probabilistic jumping property of simulated annealing (SA) algorithm is introduced into the particle swarm optimization (PSO) algorithm to make the particle accept bad solution at a certain probability during the searching process, and therefore boost the capacity of the proposed algorithm to bounce from the local optimum. The fitness function is constructed by the branch function superposition to properly guide the search process. In addition, to improve the convergence speed of the algorithm, an adaptive adjustment scheme based on inertial weight and an adjustment scheme based on learning factor are proposed as well. The experimental results show that the proposed method can effectually avoid premature convergence and effectively upgrades the efficiency of generating test data automatically. It is also competitive in solving other complicated optimization problems. [ABSTRACT FROM AUTHOR]

The purpose of this study is to identify and classify studies published on software testing techniques applied to blockchain systems. Previously published reviews in related areas have a narrow focus and/or do not follow a systematic review protocol. We conducted a systematic mapping based on an initial selection of 1025 studies. A rigorous selection process resulted in a final pool of 17 primary studies. These studies are categorized with respect to the employed testing methods, considered quality attributes, and functionality. We observe that most of the publications focus on testing functional correctness or security, whereas the testing of runtime performance attracts less attention. Existing approaches mostly employ fuzz testing or mutation testing. Search-based testing is usually combined with these techniques. The application of model-based testing is rare. The adaptability of fuzz testing and model-based testing techniques to changing blockchain platforms and languages remains a concern. On the other hand, performance and scalability issues are noted for search-based techniques and mutation testing. The use and integration of multiple testing techniques also stand out as a viable research direction. [ABSTRACT FROM AUTHOR]

Software applications (apps) have been playing an increasingly important role in various aspects of society. In particular, mobile apps and web apps are the most prevalent among all applications and are widely used in various industries as well as in people's daily lives. To help ensure mobile and web app quality, many approaches have been introduced to improve app GUI testing via automated exploration, including random testing, model-based testing, learning-based testing, and so on. Despite the extensive effort, existing approaches are still limited in reaching high code coverage, constructing high-quality models, and being generally applicable. Reinforcement learning-based approaches, as a group of representative and advanced approaches for automated GUI exploration testing, are faced with difficult challenges, including effective app state abstraction, reward function design, and so on. Moreover, they heavily depend on the specific execution platforms (i.e., Android or Web), thus leading to poor generalizability and being unable to adapt to different platforms. This work specifically tackles these challenges based on the high-level observation that apps from distinct platforms share commonalities in GUI design. Indeed, we propose PIRLTest, an effective platform-independent approach for app testing. Specifically, PIRLTest utilizes computer vision and reinforcement learning techniques in a novel, synergistic manner for automated testing. It extracts the GUI widgets from GUI pages and characterizes the corresponding GUI layouts, embedding the GUI pages as states. The app GUI state combines the macroscopic perspective (app GUI layout) and the microscopic perspective (app GUI widget) and attaches the critical semantic information from GUI images. This enables PIRLTest to be platform-independent and makes the testing approach generally applicable on different platforms. PIRLTest explores apps with the guidance of a curiosity-driven strategy, which uses a Q-network to estimate the values of specific state-action pairs to encourage more exploration in uncovered pages without platform dependency. The exploration will be assigned with rewards for all actions, which are designed considering both the app GUI states and the concrete widgets, to help the framework explore more uncovered pages. We conduct an empirical study on 20 mobile apps and 5 web apps, and the results show that PIRLTest is zero-cost when being adapted to different platforms, and can perform better than the baselines, covering 6.3–41.4% more code on mobile apps and 1.5–51.1% more code on web apps. PIRLTest is capable of detecting 128 unique bugs on mobile and web apps, including 100 bugs that cannot be detected by the baselines. [ABSTRACT FROM AUTHOR]

Natural language processing (NLP) has gained widespread adoption in the development of real-world applications. However, the black-box nature of neural networks in NLP applications poses a challenge when evaluating their performance, let alone ensuring it. Recent research has proposed testing techniques to enhance the trustworthiness of NLP-based applications. However, most existing works use a single, aggregated metric (i.e., accuracy) which is difficult for users to assess NLP model performance on fine-grained aspects, such as LCs. To address this limitation, we present ALiCT, an automated testing technique for validating NLP applications based on their LCs. ALiCT takes user-specified LCs as inputs and produces diverse test suite with test oracles for each of given LC. We evaluate ALiCT on two widely adopted NLP tasks, sentiment analysis and hate speech detection, in terms of diversity, effectiveness, and consistency. Using Self-BLEU and syntactic diversity metrics, our findings reveal that ALiCT generates test cases that are 190% and 2213% more diverse in semantics and syntax, respectively, compared to those produced by state-of-the-art techniques. In addition, ALiCT is capable of producing a larger number of NLP model failures in 22 out of 25 LCs over the two NLP applications. [ABSTRACT FROM AUTHOR]

The increasing number of software applications incorporating machine learning (ML) solutions has led to the need for testing techniques. However, testing ML software requires tremendous human effort to design realistic and relevant test inputs and to judge software output correctness according to human common sense. Even when misbehavior is exposed, it is often unclear whether the defect is inside ML API or the surrounding code and how to fix the implementation. This article tackles these challenges by proposing Keeper, an automated testing and fixing tool for ML software. The core idea of Keeper is designing pseudo-inverse functions that semantically reverse the corresponding ML task in an empirical way and proxy common human judgment of real-world data. It incorporates these functions into a symbolic execution engine to generate tests. Keeper also detects code smells that degrade software performance. Once misbehavior is exposed, Keeper attempts to change how ML APIs are used to alleviate the misbehavior. Our evaluation on a variety of applications shows that Keeper greatly improves branch coverage, while identifying 74 previously unknown failures and 19 code smells from 56 out of 104 applications. Our user studies show that 78% of end-users and 95% of developers agree with Keeper's detection and fixing results. [ABSTRACT FROM AUTHOR]

Large Language Models (LLMs) have received much recent attention due to their human-level accuracy. While existing works mostly focus on either improving accuracy or testing accuracy robustness, the computation efficiency of LLMs, which is of paramount importance due to often vast generation demands and real-time requirements, has surprisingly received little attention. In this article, we make the first attempt to understand and test potential computation efficiency robustness in state-of-the-art LLMs. By analyzing the working mechanism and implementation of 20,543 public-accessible LLMs, we observe a fundamental property in LLMs that could be manipulated in an adversarial manner to reduce computation efficiency significantly. Our interesting observation is that the output length determines the computation efficiency of LLMs instead of the input, where the output length depends on two factors: an often sufficiently large yet pessimistic pre-configured threshold controlling the max number of iterations and a runtime-generated end of sentence (EOS) token. Our key motivation is to generate test inputs that could sufficiently delay the generation of EOS such that LLMs would have to go through enough iterations to satisfy the pre-configured threshold. We present LLMEffiChecker, which can work under both white-box setting and black-box setting. In the white-box scenario, LLMEffiChecker develops a gradient-guided technique that searches for a minimal and unnoticeable perturbation at character-level, token-level, and structure-level. In the black-box scenario, LLMEffiChecker employs a causal inference-based approach to find critical tokens and similarly applies three levels of imperceptible perturbation to them. Both the white-box and black-box settings effectively delay the appearance of EOS, compelling these inputs to reach the naturally unreachable threshold. To demonstrate the effectiveness of LLMEffiChecker, we conduct a systematic evaluation on nine publicly available LLMs: Google T5, AllenAI WMT14, Helsinki-NLP translator, Facebook FairSeq, UNICAMP-DL translator, MarianMT, Google FLAN-T5, MBZUAI LaMini-GPT, and Salesforce CodeGen. Experimental results show that LLMEffiChecker can increase on average LLMs' response latency and energy consumption by 325% to 3,244% and 344% to 3,616%, respectively, by perturbing just one character or token in the input sentence. Our case study shows that inputs generated by LLMEffiChecker significantly affect the battery power in real-world mobile devices (i.e., drain more than 30 times battery power than normal inputs). [ABSTRACT FROM AUTHOR]

Metamorphic testing is a valuable approach to verifying machine learning programs where traditional oracles are unavailable or difficult to apply. This paper proposes a technique to prioritize metamorphic relations (MRs) in metamorphic testing for machine learning and deep learning systems, aiming to enhance early fault detection. We introduce five metrics based on diversity in source and follow-up test cases to prioritize MRs. The effectiveness of our proposed prioritization methods is evaluated on three machine learning and one deep learning algorithm implementation. We compare our approach against random-based, fault-based, and neuron activation coverage-based MR ordering. The results show that our data diversity-based prioritization performs comparably to fault-based prioritization, reducing fault detection time by up to 62% compared to random MR execution. Our proposed metrics outperformed neuron activation coverage-based prioritization, providing 5–550% higher fault detection effectiveness. Overall, our approach to prioritizing metamorphic relations leads to increased fault detection effectiveness and reduced average fault detection time. This improvement in efficiency can result in significant time and cost savings when applying metamorphic testing to machine learning and deep learning systems. [ABSTRACT FROM AUTHOR]

Software testing is a critical phase due to misconceptions about ambiguities in the requirements during specification, which affect the testing process. Therefore, it is difficult to identify all faults in software. As requirement changes continuously, it increases the irrelevancy and redundancy during testing. Due to these challenges; fault detection capability decreases and there arises a need to improve the testing process, which is based on changes in requirements specification. In this research, we have developed a model to resolve testing challenges through requirement prioritization and prediction in an agile-based environment. The research objective is to identify the most relevant and meaningful requirements through semantic analysis for correct change analysis. Then compute the similarity of requirements through case-based reasoning, which predicted the requirements for reuse and restricted to error-based requirements. Afterward, the apriori algorithm mapped out requirement frequency to select relevant test cases based on frequently reused or not reused test cases to increase the fault detection rate. Furthermore, the proposed model was evaluated by conducting experiments. The results showed that requirement redundancy and irrelevancy improved due to semantic analysis, which correctly predicted the requirements, increasing the fault detection rate and resulting in high user satisfaction. The predicted requirements are mapped into test cases, increasing the fault detection rate after changes to achieve higher user satisfaction. Therefore, the model improves the redundancy and irrelevancy of requirements by more than 90% compared to other clustering methods and the analytical hierarchical process, achieving an 80% fault detection rate at an earlier stage. Hence, it provides guidelines for practitioners and researchers in the modern era. In the future, we will provide the working prototype of this model for proof of concept. [ABSTRACT FROM AUTHOR]

Faulty Interaction Localization (FIL) is a process to identify which combination of input parameter values induced test failures in combinatorial testing. An accurate and fast FIL provides helpful information to fix defects causing the test failure. One type of conventional FIL approach, which analyzes test results of whole test cases and estimates the suspiciousness of each combination, has two main concerns; (1) the accuracy is not enough, (2) the huge time cost is sometimes needed. In this paper, we propose two novel approaches to improve those concerns. FROGa attempts to estimate suspiciousness more accurately using logistic regression analysis. FROGb attempts to estimate failure-inducing combinations at high speed by estimating the subsets of them using logistic regression analysis and exploring just their supersets. Through evaluation experiments using a large number of artificial test results based on several real software systems, we observed that FROGa has very high accuracy, and FROGb can drastically reduce time cost for targets that have been difficult to complete by the conventional method. [ABSTRACT FROM AUTHOR]

The internet key exchange (IKE) protocol, integral to the authentication and key negotiation process within the Internet Protocol Security (IPSec) framework, has been utilized for the protection of IP communications. Given the complex protocol logic, security vulnerabilities in the implementation of the IKE protocol are inevitably present. Fuzz testing, recognized as an effective means of detecting potential vulnerabilities in protocol implementations, has been conventionally applied. However, the direct application of existing fuzzing tools to the IKE protocol has been found to present limitations, such as the generation of low-quality test cases and difficulty in exploring deep states. To address these issues, a mutation strategy based on the grammar of the IKE protocol was designed, aiming to reduce the generation of invalid test cases while increasing the diversity of generated test cases. Additionally, an evolutionary strategy-based mutation scheduling scheme was introduced, which automatically optimized the probability distribution of mutation operators, further increasing the likelihood of generating high-quality test cases. A message handler was designed to maintain protocol interaction context information and perform cryptographic operations, thereby supporting testing of the IKE protocol under black-box conditions. This enabled the exploration of deep protocol interaction behavior and state space. Utilizing the aforementioned methods, an IKE protocol stateful fuzz testing tool named IKEChecker was implemented, supporting testing of both IKEv1 and IKEv2 protocols. Testing was conducted on two widely used open-source IKE protocol implementations, strongSwan and Libreswan, resulting in the revelation of 4 undisclosed vulnerabilities. By comparing IKEChecker with other fuzz testing tools, its efficiency in vulnerability detection was evaluated.

In this research, authors give a literature analysis of the methods used to detect and anticipate security risks in software testing by using a number of deep learning models. The purpose of this study is to conduct a literature review on the use of deep learning models in software testing for the purpose of detecting and predicting security issues. Moreover, the motive of this work is to analyze the exiting techniques related as software testing model and its application oriented performance. A thorough search of the available literature across several databases and resources forms the basis of this evaluation. There are a total of 69 publications found via the search, 34 of which are original research. Convolutional neural networks, long short-term memory networks, generative adversarial networks, and capsule networks are only some of the state-of-the-art methods for identifying and predicting security risks that are the topic of this research study. Datasets, performance indicators, and assessment criteria are all dissected for this examination. Finally, the benefits and drawbacks of using deep learning models to detect and anticipate software security flaws during testing are outlined in this article. Software testing experts and academics will benefit from this study since it will shed light on the present state of the art in the area and provide direction for the creation of innovative approaches to the detection and forecasting of security risks in software testing. [ABSTRACT FROM AUTHOR]

Few mobile robot developers already test their software on simulated robots in virtual environments or sceneries. However, the majority still shy away from simulation-based test campaigns because it remains challenging to specify and execute suitable testing scenarios, that is, models of the environment and the robots' tasks. Through developer interviews, we identified that managing the enormous variability of testing scenarios is a major barrier to the application of simulation-based testing in robotics. Furthermore, traditional CAD or 3Dmodelling tools such as SolidWorks, 3ds Max, or Blender are not suitable for specifying sceneries that vary significantly and serve different testing objectives. For some testing campaigns, it is required that the scenery replicates the dynamic (e.g., opening doors) and static features of real-world environments, whereas for others, simplified scenery is sufficient. Similarly, the task and mission specifications used for simulation-based testing range from simple point-to-point navigation tasks to more elaborate tasks that require advanced deliberation and decision-making. We propose the concept of composable and executable scenarios and associated tooling to support developers in specifying, reusing, and executing scenarios for the simulation-based testing of robotic systems. Our approach differs from traditional approaches in that it offers a means of creating scenarios that allow the addition of new semantics (e.g., dynamic elements such as doors or varying task specifications) to existing models without altering them. Thus, we can systematically construct richer scenarios that remain manageable. We evaluated our approach in a small simulation-based testing campaign, with scenarios defined around the navigation stack of a mobile robot. The scenarios gradually increased in complexity, composing new features into the scenery of previous scenarios. Our evaluation demonstrated how our approach can facilitate the reuse of models and revealed the presence of errors in the configuration of the publicly available navigation stack of our SUT, which had gone unnoticed despite its frequent use. [ABSTRACT FROM AUTHOR]

The enhancement of software system quality is achieved through a process called software testing, which is a time and cost-intensive stage of software development. As a result, automating software tests is recognized as an effective solution that can simplify time-consuming and arduous testing activities. Generating test data with maximum branch coverage and fault discovery capability is an NP-complete optimization problem. Various methods based on heuristics and evolutionary algorithms have been suggested to create test suites that provide the most feasible coverage. The main disadvantages of past approaches include inadequate branching coverage, fault detection rate, and unstable results. The main objectives of the current research are to improve the branch coverage rate, fault detection rate, success rate, and stability. This research has suggested an efficient technique to produce test data automatically utilizing a hybrid version of Olympiad Optimization Algorithms (OOA) in conjunction with genetic algorithm (GA) operators theory. Maximum coverage, fault detection capability, and success rate are the main characteristics of produced test data. Various experiments have been conducted on the nine standard benchmark programs. Regarding the results, the suggested method provides 99.92% average coverage, a success rate of 99.20%, an average generation of 5.76, and an average time of 7.97 s. Based on the fault injection experiment's results, the proposed method can discover about 89% of the faults injected by mutation testing tools such as MuJava. [ABSTRACT FROM AUTHOR]

Quality and productivity needs are considered together in software. For this reason, any existing software should be tested automatically with test automation. Software test automations is automated software testing activities. Automating constantly manually written tests, on the other hand, saves time, reduces error rates, produces better quality software, and reduces costs. This study aims to produce an automatic unit testing framework that is planned to work in run-time on software products. This developed application performs unit test transformations that can respond to the desired test scenarios on the product being studied.. Java agent is used as the basis of all these transformations. All information about the objects, methods, and variables of the sample java classes to be worked on is converted into data in run-time using byte code. During this transformation, information is saved in the database, and unit tests are created automatically through the template engine. Compared to the products developed on automatic unit test generation in the literature, the opcode parsing method was developed for this study. This method reads a byte code at run-time, uses the properties of the java class it belongs to, and automatically creates the unit test class and test methods. The study can also examine different object definitions and conditional and loop structures within a method and produce alternative test scenarios. The automatic unit test scenario produced has been turned into a flexible framework that can encounter minimum errors at run-time. Considering the scarcity of studies in the field of national software testing; It is thought that the automatic unit test generation product developed within the scope of this study, using byte code, will contribute to the work area. [ABSTRACT FROM AUTHOR]

As software projects evolve and grow in size and complexity, so do their test suites. Test suite reduction (TSR) aims at reducing the size of a test suite by removing redundant and obsolete test cases based on a coverage metric while preserving its fault detection capabilities. The contributions of this paper are twofold: (1) We examine a lesser‐known coverage criterion, that is, checked coverage. Checked coverage not only investigates if a part of the code was executed but also if it was checked by a test oracle. In an empirical evaluation, we performed TSR based on different reduction algorithms, coverage metrics, and open‐source Java projects with our own TSR tool to determine the most effective and efficient combination of metric and method. (2) Given the results of the first evaluation, we further investigate the potential of parameter optimization in regard to a genetic reduction algorithm. In particular, we focus on finding a general setting for the parameters crossover rate and mutation rate such that test suites can be reduced in a reasonable time while maintaining a high fault detection power. [ABSTRACT FROM AUTHOR]

Summary: To ensure the quality of software systems, software engineers can make use of a variety of quality assurance approaches, for example, software testing, modern code review, automated static analysis, and build automation. Each of these quality assurance practices have been studied in depth in isolation, but there is a clear knowledge gap when it comes to our understanding of how these approaches are being used in conjunction, or not. In our study, we broadly investigate whether and how these quality assurance approaches are being used in conjunction in the development of 1454 popular open source software projects on GitHub. Our study indicates that typically projects do not follow all quality assurance practices together with high intensity. In fact, we only observe weak correlation among some quality assurance practices. In general, our study provides a deeper understanding of how existing quality assurance approaches are currently being used in Java‐based open source software development. Besides, we specifically zoom in on the more mature projects in our dataset, and generally we observe that more mature projects are more intense in their application of the quality assurance practices, with more focus on their ASAT usage, and code reviewing, but no strong change in their CI usage. [ABSTRACT FROM AUTHOR]

In software that is free from defects or errors, a fixed method and technique of software testing is required. For this step, an Information System is needed to help eliminate software defects, so that testing or testing work is not a burden on software costs but is a perfection of software development work so that Zero Defect Application Software can be realized. From several software testing methods and techniques, a strategy suitable for the size of a software development project is required. So even though the software development project is small, it still requires a suitable test. And for software development testing to have reliable quality, it is necessary to measure and criteria for each software testing practice, so that a software development project can be determined in quality. To support software testing work, an Information System is needed that supports the need for processors to assess a quality software development test. With the Software Defect Elimination Information System, an objective quality of software development testing can be produced. [ABSTRACT FROM AUTHOR]

As software vulnerabilities can cause cybersecurity threats and have severe consequences, it is necessary to develop effective techniques to discover such vulnerabilities. Fuzzing is one of the most widely employed approaches that has been adapted for software testing. The mutation-based fuzzing approach is currently the most popular. The state-of-the-art American Fuzzy Lop (AFL) selects mutations randomly and lacks knowledge of mutation operations that are more helpful in a particular stage. This study performs a systematic review to identify and analyze existing approaches that optimize the selection of mutation operations. The main contributions of this work are to draw attention to the importance of mutation operator selection, identify optimization algorithms for mutation operator selection, and investigate their impact on fuzzing testing in terms of code coverage and finding new vulnerabilities. The investigation shows the effectiveness and advantages of optimizing the selection of mutation operations to achieve higher code coverage and find more vulnerabilities. [ABSTRACT FROM AUTHOR]

Software is a part of most complex systems used for different purposes in today's world. The reliability of these systems must be verified before putting them to use, because their malfunction may lead to catastrophic failure, the loss of human lives, or huge costs. Software reliability can be verified by performing black‐box testing, without knowledge of the complex internal structure of these systems. In this work, an embedded IoT alarm system for detection of intruders and potential robbery is proposed. It is composed of three subsystems: embedded IoT, web‐server, and mobile phone subsystems. Cause‐effect graphing technique was applied for generating three types of black‐box test suites, by using the forward‐propagation and two different minimization algorithms. These test suites were used for black‐box testing the system, which was performed after injecting a maximum of 12 faults into the system in 100 iterations. The resulting software quality and reliability metrics showed that the forward‐propagation algorithm always achieves maximum fault detection rate (FDR), and this algorithm should be used when the highest demand for reliability must be met. The optimized minimization should only be used in cases where testing is very expensive, because it did not find faults in all testing iterations. The basic optimization retained high FDR, while reducing the impact of the minimized test suite and achieved a significant increase of test effect coverage values. [ABSTRACT FROM AUTHOR]

Testing Autonomous Driving Systems (ADSs) is crucial to ensure their reliability when navigating complex environments. ADSs may exhibit unexpected behaviours when presented, during operation, with driving scenarios containing features inadequately represented in the training dataset. To address this shift from development to operation, developers must acquire new data with the newly observed features. This data can be then utilised to fine tune the ADS, so as to reach the desired level of reliability in performing driving tasks. However, the resource-intensive nature of testing ADSs requires efficient methodologies for generating targeted and diverse tests. In this work, we introduce a novel approach, DeepAtash-LR, that incorporates a surrogate model into the focused test generation process. This integration significantly improves focused testing effectiveness and applicability in resource-intensive scenarios. Experimental results show that the integration of the surrogate model is fundamental to the success of DeepAtash-LR. Our approach was able to generate an average of up to 60× more targeted, failure-inducing inputs compared to the baseline approach. Moreover, the inputs generated by DeepAtash-LR were useful to significantly improve the quality of the original ADS through fine tuning. [ABSTRACT FROM AUTHOR]

Metamorphic Testing (MT) alleviates the oracle problem by defining oracles based on metamorphic relations (MRs) that govern multiple related inputs and their outputs. However, designing MRs is challenging, as it requires domain-specific knowledge. This hinders the widespread adoption of MT. We observe that developer-written test cases can embed domain knowledge that encodes MRs. Such encoded MRs could be synthesized for testing not only their original programs but also other programs that share similar functionalities. In this article, we propose MR-Scout to automatically synthesize MRs from test cases in open-source software (OSS) projects. MR-Scout first discovers MR-encoded test cases (MTCs), and then synthesizes the encoded MRs into parameterized methods (called codified MRs), and filters out MRs that demonstrate poor quality for new test case generation. MR-Scout discovered over 11,000 MTCs from 701 OSS projects. Experimental results show that over 97% of codified MRs are of high quality for automated test case generation, demonstrating the practical applicability of MR-Scout. Furthermore, codified-MRs-based tests effectively enhance the test adequacy of programs with developer-written tests, leading to 13.52% and 9.42% increases in line coverage and mutation score, respectively. Our qualitative study shows that 55.76% to 76.92% of codified MRs are easily comprehensible for developers. [ABSTRACT FROM AUTHOR]

Quantum computing has emerged as a promising field with the potential to revolutionize various domains by harnessing the principles of quantum mechanics. As quantum hardware and algorithms continue to advance, developing high-quality quantum software has become crucial. However, testing quantum programs poses unique challenges due to the distinctive characteristics of quantum systems and the complexity of multi-subroutine programs. This article addresses the specific testing requirements of multi-subroutine quantum programs. We begin by investigating critical properties by surveying existing quantum libraries and providing insights into the challenges of testing these programs. Building upon this understanding, we focus on testing criteria and techniques based on the whole testing process perspective, spanning from unit testing to integration testing. We delve into various aspects, including IO analysis, quantum relation checking, structural testing, behavior testing, integration of subroutine pairs, and test case generation. We also introduce novel testing principles and criteria to guide the testing process. We conduct comprehensive testing on typical quantum subroutines, including diverse mutants and randomized inputs, to evaluate our proposed approach. The analysis of failures provides valuable insights into the effectiveness of our testing methodology. Additionally, we present case studies on representative multi-subroutine quantum programs, demonstrating the practical application and effectiveness of our proposed testing principles and criteria. [ABSTRACT FROM AUTHOR]

Voice-based virtual assistants are becoming increasingly popular. Such systems provide frameworks to developers for building custom apps. End-users can interact with such apps through a Voice User Interface (VUI), which allows the user to use natural language commands to perform actions. Testing such apps is not trivial: The same command can be expressed in different semantically equivalent ways. In this article, we introduce VUI-UPSET, an approach that adapts chatbot-testing approaches to VUI-testing. We conducted an empirical study to understand how VUI-UPSET compares to two state-of-the-art approaches (i.e., a chatbot testing technique and ChatGPT) in terms of (i) correctness of the generated paraphrases, and (ii) capability of revealing bugs. To this aim, we analyzed 14,898 generated paraphrases for 40 Alexa Skills. Our results show that VUI-UPSET generates more bug-revealing paraphrases than the two baselines with, however, ChatGPT being the approach generating the highest percentage of correct paraphrases. We also tried to use the generated paraphrases to improve the skills. We tried to include in the voice interaction models of the skills (i) only the bug-revealing paraphrases, (ii) all the valid paraphrases. We observed that including only bug-revealing paraphrases is sometimes not sufficient to make all the tests pass. [ABSTRACT FROM AUTHOR]

In today's rapidly evolving software environment, the graphical user interface (GUI) plays a crucial role in providing intuitive, user-friendly interaction. However, traditional intrusive GUI testing methods often face challenges such as interrupting user workflows, requiring significant manual effort and insufficient test scenario coverage. Non-intrusive testing methods, such as Robotic Process Automation (RPA), offer a solution to validate GUI functionality without modifying the application's code or affecting the user experience. RPA systems automate repetitive tasks by simulating user interactions, becoming valuable tools in GUI testing. However, challenges like limited computational resources, time constraints, or restricted exploration capabilities may limit the efficiency of individual RPA agents, thus restricting coverage and effectiveness. To address this issue, this study explores the performance of a single RPA agent versus an RPA cluster under different testing conditions, using three popular testing methods: Monkey, Stoat and Q-testing. Experimental results indicate that an RPA cluster outperforms a single RPA in GUI coverage and error detection, making a significant contribution to the field of non-intrusive GUI exploration testing. The findings of this study provide directions for future research to ensure the delivery of high-quality mobile applications. [ABSTRACT FROM AUTHOR]

Summary Natural language inference (NLI) is a task to infer the relationship between the premise and hypothesis sentences, whose models have essential applications in the many natural language processing (NLP) fields, for example, machine reading comprehension and recognizing textual entailment. Due to the data‐driven programming paradigm, bugs inevitably occur in NLI models during the application process, which calls for novel automatic testing techniques to deal with NLI testing challenges. The main difficulty in achieving automatic testing for NLI models is the oracle problem; that is, it may be too expensive to label NLI model inputs manually and hence be too challenging to verify the correctness of model outputs. To tackle the oracle problem, this study proposes a novel automatic testing method hybrid mutation driven testing (HMT), which extends the mutation idea applied in other NLP domains successfully. Specifically, as there are two sets of sentences, that is, premise and hypothesis, to be mutated, we propose four mutation operators to achieve the hybrid mutation strategy, which mutate the premise and the hypothesis sentences jointly or individually. We assume that the mutation would not affect the outputs; that is, if the original and mutated outputs are inconsistent, inconsistency bugs could be detected without knowing the true labels. To evaluate our method HMT, we conduct experiments on two widely used datasets with two advanced models and generate more than 520,000 mutations by applying our mutation operators. Our experimental results show that (a) our method, HMT, can effectively generate mutated testing samples, (b) our method can effectively trigger the inconsistency bugs of the NLI models, and (c) all four mutation operators can independently trigger inconsistency bugs. [ABSTRACT FROM AUTHOR]

With the rapid development of information technology, software has become a part of social life. However, for complex software systems, traditional design and testing methods face a series of problems such as low testing coverage and low efficiency. The main purpose of this article is to study the optimization of software design and testing processes using intelligent algorithms, in order to improve the quality and efficiency of software. This article established a real-time optimization and feedback adjustment mechanism during the software design and testing phase, enabling intelligent algorithms to respond promptly to the environment and requirements. Monitoring and control strategies were designed to ensure the algorithm runs stably for a long time. Test method 1 (traditional method) had 100 test cases, a total execution time of 1200 seconds, and an average execution time of 12 seconds per test case; test method 2 (Intelligent Algorithm Optimization) had 100 test cases, a total execution time of 800 seconds, and an average execution time of 8 seconds per test case. The research results of this article play a good promoting role in improving software testing coverage and software structure optimization, and also provide new ideas and methods for the promotion of intelligent algorithms in software engineering. [ABSTRACT FROM AUTHOR]

Test Case Generation (TCG) generates various types of tests, including functional tests, performance tests, security tests, and reliability tests to ensure software quality, while Test Case Prioritization (TCP) prioritizes the generated tests. However, the previous studies had challenges, including resource constraints, detecting crucial requirements, and automating the Test Case (TC) process efficiently. Additionally, the process is costlier and takes a maximum time duration that affects the effective performance. Therefore, an effective framework is proposed to overcome such issues by optimizing TCG and TCP processes effectively. The proposed work starts with the generation of a Unified Modeling Language (UML) diagram from historical project source code, which is then converted into a Comma-Separated Value (CSV) format. Then, the feature extraction is performed on this CSV file, followed by optimal TCG using the Entropy-based Locust Swarm Optimization Algorithm (Ent-LSOA). Additionally, factors are extracted and reduced from the historical project source code using Pearson Correlation Coefficient-Generalized Discriminant Analysis (PCC-GDA). Finally, the optimal TCs and selected factors are prioritized with the highest accuracy and recall of 96.89% and 96.92%, respectively using an Interpolated Multiple Time scale Recurrent Neural Network (IMTRNN). Thus, the proposed work outperformed the existing techniques by providing an efficient solution for TCG and TCP in software testing. [ABSTRACT FROM AUTHOR]

Fault localization is a complex, costly and time‐consuming task in software debugging. Numerous automated techniques have been developed to expedite this process. Mutation‐based fault localization (MBFL) is one of the most widely studied techniques which uses mutation analysis to generate mutants for revealing potential faults in the program. However, our theoretical analysis exposes an inherent conflict between the fundamental assumption and the essential meaning of existing MBFL suspiciousness. This conflict is caused by mutant bias. Intuitively, the suspiciousness can be corrected by eliminating the mutant bias for more accurately measuring the faulty probability of the corresponding mutant statement. In this paper, we introduce Delta4Ms, a fault localization approach designed to eliminate mutant bias. Delta4Ms integrates the principles of signal theory, modelling the actual suspiciousness and mutant bias as the desired and false signal components, respectively. Based on theoretical derivation, the average suspiciousness of mutants serves as an estimate of mutant bias. Delta4Ms effectively mitigates mutant bias, extracting the desired signal and yielding corrected suspiciousness for fault localization. To precisely estimate mutant bias, higher order mutants (HOMs) are incorporated. We conduct an extensive experimental evaluation of Delta4Ms on 320 real‐fault programs from Codeflaws. The results indicate that our model significantly outperforms existing SBFL and MBFL techniques, showing a considerable improvement in fault localization effectiveness. We further assessed the robustness of Delta4Ms by examining different HOM ratios and HOM generation strategies. Moreover, Delta4Ms achieves a substantial reduction in mutation execution cost and minimal accuracy loss through the implementation of test case reduction. Finally, we perform preliminary experiments on 15 real‐fault programs from the Defects4J benchmark to assess the generalization of the model's fault localization effectiveness. [ABSTRACT FROM AUTHOR]

Machine translation systems have been widely adopted in our daily life, making life easier and more convenient. Unfortunately, erroneous translations may result in severe consequences, such as financial losses. This requires to improve the accuracy and the reliability of machine translation systems. However, it is challenging to test machine translation systems because of the complexity and intractability of the underlying neural models. To tackle these challenges, we propose a novel metamorphic testing approach by syntactic tree pruning (STP) to validate machine translation systems. Our key insight is that a pruned sentence should have similar crucial semantics compared with the original sentence. Specifically, STP (1) proposes a core semantics-preserving pruning strategy by basic sentence structures and dependency relations on the level of syntactic tree representation, (2) generates source sentence pairs based on the metamorphic relation, and (3) reports suspicious issues whose translations break the consistency property by a bag-of-words model. We further evaluate STP on two state-of-the-art machine translation systems (i.e., Google Translate and Bing Microsoft Translator) with 1,200 source sentences as inputs. The results show that STP accurately finds 5,073 unique erroneous translations in Google Translate and 5,100 unique erroneous translations in Bing Microsoft Translator (400% more than state-of-the-art techniques), with 64.5% and 65.4% precision, respectively. The reported erroneous translations vary in types and more than 90% of them are not found by state-of-the-art techniques. There are 9,393 erroneous translations unique to STP, which is 711.9% more than state-of-the-art techniques. Moreover, STP is quite effective in detecting translation errors for the original sentences with a recall reaching 74.0%, improving state-of-the-art techniques by 55.1% on average. [ABSTRACT FROM AUTHOR]

Software-defined networks (SDN) enable flexible and effective communication systems that are managed by centralized software controllers. However, such a controller can undermine the underlying communication network of an SDN-based system and thus must be carefully tested. When an SDN-based system fails, in order to address such a failure, engineers need to precisely understand the conditions under which it occurs. In this article, we introduce a machine learning-guided fuzzing method, named FuzzSDN, aiming at both (1) generating effective test data leading to failures in SDN-based systems and (2) learning accurate failure-inducing models that characterize conditions under which such system fails. To our knowledge, no existing work simultaneously addresses these two objectives for SDNs. We evaluate FuzzSDN by applying it to systems controlled by two open-source SDN controllers. Furthermore, we compare FuzzSDN with two state-of-the-art methods for fuzzing SDNs and two baselines for learning failure-inducing models. Our results show that (1) compared to the state-of-the-art methods, FuzzSDN generates at least 12 times more failures, within the same time budget, with a controller that is fairly robust to fuzzing and (2) our failure-inducing models have, on average, a precision of 98% and a recall of 86%, significantly outperforming the baselines. [ABSTRACT FROM AUTHOR]

Test smells, detrimental coding practices that hinder high-quality test code development, pose a significant challenge in software testing and maintenance. Software refactoring, traditionally a powerful approach for addressing code smells and improving software quality without changing functionality, often focuses on production code, leaving test code overlooked. Despite extensive research in test smell refactoring, understanding the efficacy of existing refactoring operations on test code quality remains limited. Investigating real-world developer refactoring practices is crucial to bridge this knowledge gap. In this study, we investigate refactorings performed by developers to address test smells, resulting in a comprehensive catalog of test smells and their corresponding test-specific refactorings. Two test-specific refactorings closely tied to JUnit5 and seven version-agnostic refactorings for various JUnit versions have been identified. While many of these test-specific refactorings are documented in the literature, this analysis unveils new test-specific refactorings aimed at dealing with the "Inappropriate Assertion" test smell. This research provides insights into the challenges faced by developers and prevailing practices for effectively refactoring test code, thereby enhancing software testing and maintenance. [ABSTRACT FROM AUTHOR]

Boundary value analysis (BVA) is a common technique in software testing that uses input values that lie at the boundaries where significant changes in behavior are expected. This approach is widely recognized and used as a natural and effective strategy for testing software. Test coverage is one of the criteria to measure how much the software execution paths are covered by the set of test cases. This paper focuses on evaluating test coverage with respect to BVA by defining a metric called boundary coverage distance (BCD). The BCD metric measures the extent to which a test set covers the boundaries. In addition, based on BCD, we consider the optimal test input generation to minimize BCD under the random testing scheme. We propose three algorithms, each representing a different test input generation strategy, and evaluate their fault detection capabilities through experimental validation. The results indicate that the BCD-based approach has the potential to generate boundary values and improve the effectiveness of software testing. [ABSTRACT FROM AUTHOR]

Pairwise testing is a method for identifying defects through combinatorial analysis. It involves testing all possible combinations of input parameters in pairs within a system, ensuring that each pair is tested at least once. The field of test case generation is highly active in the realm of combinatorial interaction testing. Research in this area is particularly encouraged, as it falls under the category of non-deterministic polynomial-time hardness. A big challenge in this field is the combinatorial explosion problem. It is about finding the best test suite that covers all possible combinations of interaction strength. In this paper, we present the task of discovering a pairwise test set as a search problem and introduce an innovative testing tool referred to as pairwise test case generation in harmony search algorithm with seeding and constraint mechanism (PHOSC). Experimental results show that PHOSC performs better compared to some existing pairwise strategies in terms of test suite size. Additionally, PHOSC provides a comprehensive framework and serves as a research platform for the generation of pairwise test sets employing the harmony search algorithm. It adopts an approach that focuses on one parameter at a time (OPAT) and incorporates seeding and constraint mechanisms at the same time, thereby enhancing the efficiency and effectiveness of the testing process. [ABSTRACT FROM AUTHOR]

Industrial robotic systems (IRS) consist of industrial robots that automate industrial processes. They accurately perform repetitive tasks, replacing or assisting with dangerous jobs like assembly in the automotive and chemical industries. Failures in these systems can be catastrophic, so it is important to ensure their quality and safety before using them. One way to do this is by applying a software testing process to find faults before they become failures. However, software testing in industrial robotic systems has some challenges. These include differences in perspectives on software testing from people with diverse backgrounds, coordinating and collaborating with diverse teams, and performing software testing within the complex integration inherent in industrial environments. In traditional systems, a well-known development process uses simple, structured sentences in English to facilitate communication between project team members and business stakeholders. This process is called behavior-driven development (BDD), and one of its pillars is the use of templates to write user stories, scenarios, and automated acceptance tests. We propose a software testing (ST) approach called automated acceptance testing for industrial robotic systems (AAT4IRS) that uses natural language to write the features and scenarios to be tested. We evaluated our ST approach through a proof-of-concept, performing a pick-and-place process and applying mutation testing to measure its effectiveness. The results show that the test suites implemented using AAT4IRS were highly effective, with 79% of the generated mutants detected, thus instilling confidence in the robustness of our approach.

Software testing is an important step in software development where inputs are administered repeatedly to detect bugs present in the software. In this paper, we have considered the estimation of total number of bugs and software reliability as a size-biased sampling problem by introducing the concept of eventual bug size as a latent variable. We have developed a Bayesian generalised linear mixed model (GLMM) using software testing detection data to estimate software reliability and stopping phase. The model uses size-biased approach where the probability of detecting a bug is an increasing function of eventual size of the bug which is as an index for the potential number of inputs that may eventually pass through the bug. We have tested the sensitivity of the reliability estimates by varying the number of inputs and detection probability via a simulation study and have found that the key parameters could be accurately estimated. Further, we have applied our model to two empirical data sets – one from a commercial software and the other from ISRO launch mission software testing data set. The hierarchical modelling approach provides a unified modelling framework that may find applications in other fields (e.g. hydrocarbon explorations) apart from software management. [ABSTRACT FROM AUTHOR]

An issue regarding the incorporation of software reliability within the nuclear power plant (NPP) probabilistic risk assessment model has emerged in the licensing processes of digitalized NPPs. Since software failure induces common-cause failure of the processor modules, the reliability of the software used in the NPP safety-critical instrumentation and control systems must be quantified and verified with proper test cases and environments. In this study, a software testing method based on the minimal cut set (MCS)-based exhaustive test case generation scheme is proposed where the software logic model is developed from available information on the software development and the MCSs that represent the necessary and sufficient conditions for the software variables' states to produce safety software outputs are generated. The MCSs are then converted into the test cases, which can be used as inputs to the test bed to verify that the test cases produce correct outputs after software execution. The effectiveness of the proposed method is demonstrated with the safety-critical trip logic software of the APR-1400 reactor protection system. The method provides a systematic way to conduct exhaustive software testing and prove the functionality of the nuclear safety software based on the test result without uncertainties. [ABSTRACT FROM AUTHOR]

C is a dominant programming language for implementing system and low-level embedded software. Unfortunately, the unsafe nature of its low-level control of memory often leads to memory errors. Dynamic analysis has been widely used to detect memory errors at runtime. However, existing monitoring algorithms for dynamic analysis are not yet satisfactory, as they cannot deterministically and completely detect some types of errors, such as segment confusion errors, sub-object overflows, use-after-frees and memory leaks. We propose a new monitoring algorithm, namely Smatus, short for smart status, that improves memory safety by performing comprehensive dynamic analysis. The key innovation is to maintain at runtime a small status node for each memory object. A status node records the status value and reference count of an object, where the status value denotes the liveness and segment type of this object, and the reference count tracks the number of pointer variables pointing to this object. Smatus maintains at runtime a pointer metadata for each pointer variable, to record not only the base and bound of a pointer's referent but also the address of the referent's status node. All the pointers pointing to the same referent share the same status node in their pointer metadata. A status node is smart in the sense that it is automatically deleted when it becomes useless (indicated by its reference count reaching zero). To the best of our knowledge, Smatus represents the most comprehensive approach of its kind. We have evaluated Smatus by using a large set of programs including the NIST Software Assurance Reference Dataset, MSBench, MiBench, SPEC and stress testing benchmarks. In terms of effectiveness (detecting different types of memory errors), Smatus outperforms state-of-the-art tools, Google's AddressSanitizer, SoftBoundCETS and Valgrind, as it is capable of detecting more errors. In terms of performance (the time and memory overheads), Smatus outperforms SoftBoundCETS and Valgrind in terms of both lower time and memory overheads incurred, and is on par with AddressSanitizer in terms of the time and memory overhead tradeoff made (with much lower memory overheads incurred). [ABSTRACT FROM AUTHOR]

Regarding the caliber of newly-graduated professionals, the software industry expresses discontent. Therefore, software industries spend on training these individuals to improve software development process-related skills. According to various research studies, one reason for the lack of skilled graduates is a gap between industry and academia. One of the vital areas of software development where that gap exacerbates is software testing. The study aims to find skills (testing skills, Soft skills, and technical skills) that the software industry demands from novice software testers and assess their relevant importance. To find the skills that the software industry requires from fresh graduates, we have analyzed 160 job advertisements. The results revealed that the software industry focuses on testing skills and demands technical and soft skills, as 98% of advertisements require testing-related skills. Regarding soft skills, 92% of advertisements demanded soft skills. The novice tester must have strong technical skills as 78% of job advertisements require different technical skills. Moreover, the industry at present is also considering personality traits and educational attainment when hiring novice software testers. This study intends to identify the skill set for the novice software tester required by the Pakistan software industry. Results of the study indicate that software companies want to hire fresh graduates who are proficient in testing skills along with technical and soft skills. Therefore, there is a need to incorporate identified skills in software testing courses to better prepare computing graduates for the role of a novice software tester. [ABSTRACT FROM AUTHOR]

This paper presents Bayesian techniques for conservative claims about software reliability, particularly when evidence suggests the software's executions are not statistically independent. We formalise informal notions of "doubting" that the executions are independent, and incorporate such doubts into reliability assessments. We develop techniques that reveal the extent to which independence assumptions can undermine conservatism in assessments, and identify conditions under which this impact is not significant. These techniques – novel extensions of conservative Bayesian inference (CBI) approaches – give conservative confidence bounds on the software's failure probability per execution. With illustrations in two application areas – nuclear power‐plant safety and autonomous vehicle (AV) safety – our analyses reveals: (1) the confidence an assessor should possess before subjecting a system to operational testing. Otherwise, such testing is futile – favourable operational testing evidence will eventually decrease one's confidence in the system being sufficiently reliable; (2) the independence assumption supports conservative claims sometimes; (3) in some scenarios, observing a system operate without failure gives less confidence in the system than if some failures had been observed; (4) building confidence in a system is very sensitive to failures – each additional failure means significantly more operational testing is required, in order to support a reliability claim. [ABSTRACT FROM AUTHOR]

Testing and debugging have been the most significant steps of software development since it is tricky for engineers to create error-free software. Software testing takes place after coding with the goal of finding flaws. If errors are found, debugging would be done to identify the source of the errors so that they may be fixed. Detecting as well as locating defects are thus two essential stages in the creation of software. We have created a unique approach with the following two working phases to generate a minimized test suite that is capable of both detecting and localizing faults. In the initial test suite minimization process, the cases were generated and minimized based on the objectives such as D-score and coverage by the utilization of the proposed Blue Monkey Customized Black Widow (BMCBW) algorithm. After this test suite minimization, the fault validation is done which includes the process of fault detection and localization. For this fault validation, we have utilized an improved Long Short-Term Memory (LSTM). At 90% of the learning rate the accuracy of the presented work is 0.97%, 2.20%, 2.52%, 0.97% and 2.81% is better than the other extant models like AOA, COOT, BES, BMO and BWO methods. The results obtained proved that our Blue Monkey Customized Black Widow Optimization-based fault detection and localization approach can provide superior outcomes. [ABSTRACT FROM AUTHOR]

Software quality can be ensured by passing the process of software testing before the software is released. However, the software testing process involves many phases which leads to more resources and time consumption. Test Case Prioritization (TCP) has gained widespread acceptance because it prioritizes the tasks and reduces the number of phases and also produces results in good quality software free from defects. The coverage-based prioritization can be useful to distinguish each test case and result in a better prioritization process by using some algorithm. In this work, we propose a coverage-based prioritized test case generation using an Embedded Auto Encoder (EAE) algorithm which will produce an ordered sequence of the prioritized test cases. Initially, the code coverage for each benchmark has been extracted from the source code repository and is further processed to eliminate the noise in the data. The processed data will be given to the embedded autoencoder which consists of an autoencoder and a sparse autoencoder. Once the modeling is done, the model has been trained with the data generated by the Keras Data Generator class. The efficiency of the proposed EAE technique has been evaluated by using the APFD metric and the observations clearly show that the EAE framework proves to provide an APFD metric of 0.72 on average which is a good value in comparison to the previously deployed methodologies. [ABSTRACT FROM AUTHOR]

The objective of the research was to find if it was possible to apply fuzz testing, a technique that can be used to test software to discover vulnerabilities, on an embedded avionic software using an open-source fuzz tool. The open-source fuzz tool AFL++ was applied to an NH90 Airbus Helicopter embedded software component to find the vulnerabilities. The proposed setup was able to find a few crashes related to data parsing associated with Ada strong typing declaration requirement. Moreover, the experiments outline practical guidelines and considerations for implementing fuzz testing on embedded avionic software. [ABSTRACT FROM AUTHOR]

The reliability of AI-based object detection models has gained interest with their increasing use in safety-critical systems and the development of new regulations on artificial intelligence. To meet the need for robustness evaluation, several authors have proposed methods for testing these models. However, applying these methods in industrial settings can be difficult, and several challenges have been identified in practice in the design and execution of tests. There is, therefore, a need for clear guidelines for practitioners. In this paper, we propose a method and guidelines for assessing the robustness of AI-based 2D object detection systems, based on the Goal Question Metric approach. The method defines the overall robustness testing process and a set of recommended metrics to be used at each stage of the process. We developed and evaluated the method through action research cycles, based on two industrial cases and feedback from practitioners. Thus, the resulting method addresses issues encountered in practice. A qualitative evaluation of the method by practitioners was also conducted to provide insights that can guide future research on the subject. [ABSTRACT FROM AUTHOR]

Current testing methods for autonomous driving systems primarily focus on simple traffic scenarios, generating test cases based on traffic accidents, while research on generating edge test cases for complex driving environments by traffic regulations is not adequately comprehensive. Therefore, we propose a method for scenario modeling and violation testing using an autonomous driving system based on traffic regulations named TraModeAVTest. Initially, TraModeAVTest constructs a Petri net model for complex scenarios based on the combination relationships of basic traffic regulation scenarios and verifies the consistency of the model's design with traffic regulation requirements using formal methods, to provide a representation of traffic regulation scenario models for the violation testing of autonomous driving systems. Subsequently, based on the coverage criteria of the Petri net model, it utilizes a search strategy to generate model paths that represent traffic regulations, and employs a parameter combination method to generate test cases that cover the model paths, to test the violation behaviors of autonomous driving systems. Finally, simulation experiment results on the Baidu Apollo demonstrate that the test cases representing traffic regulations generated by TraModeAVTest can effectively identify the behaviors of autonomous vehicles violating traffic regulations, and TraModeAVTest can effectively improve the efficiency of generating different types of violation scenarios. [ABSTRACT FROM AUTHOR]

The software testing phase requires considerable time, effort, and cost, particularly when there are many faults. Thus, developers focus on the evolution of Software Fault Prediction (SFP) to predict faulty units in advance, therefore, improving software quality significantly. Forecasting the number of faults in software units can efficiently direct software testing efforts. Previous studies have employed several machine learning models to determine whether a software unit is faulty. In this study, a new, simple deep neural network approach that can adapt to the type of input data was designed, utilizing Convolutional Neural Networks (CNNs) and Multi-Layer Perceptron (MLP), to predict the number of software faults. Twelve open-source software project datasets from the PROMISE repository were used for testing and validation. As data imbalance can negatively impact prediction accuracy, the new version of synthetic minority oversampling technique (SMOTEND) was used to resolve data imbalance. In experimental results, a lower error rate was obtained for MLP, compared to CNN, reaching 0.195, indicating the accuracy of this prediction model. The proposed approach proved to be effective when compared with two of the best machine learning models in the field of prediction. The code will be available on GitHub. [ABSTRACT FROM AUTHOR]

Fuzzing is an important technique in software and security testing that involves continuously generating a large number of test cases against target programs to discover unexpected behaviors such as bugs, crashes, and vulnerabilities. Recently, fuzzing has advanced considerably owing to the emergence of new methods and corresponding tools. However, it still suffers from low coverage, ineffective detection of specific vulnerabilities, and difficulty in deploying complex applications. Therefore, to comprehensively survey the development of fuzzing techniques and analyze their advantages and existing challenges, this paper provides a comprehensive survey of the development of fuzzing techniques, summarizes the main research issues, and provides a categorized overview of the latest research advances and applications. The paper first introduces the background and related work on fuzzing. Research issues are subsequently addressed and summarized, along with the latest research developments. Furthermore, various customized fuzzing techniques in different applications are presented. Finally, the paper discusses future research directions. [ABSTRACT FROM AUTHOR]

Many cloud services are relying on distributed key-value stores such as ZooKeeper, Cassandra, HBase, etc. However, distributed key-value stores are notoriously difficult to design and implement without any mistakes. Because data consistency is the contract for clients that defines what the correct values to read are for a given history of operations under a specific consistency model, consistency violations can confuse client applications by showing invalid values. As a result, serious consequences such as data loss, data corruption, and unexpected behavior of client applications can occur. Software bugs are one of main reasons why consistency violations may occur. Formal verification techniques may be used to make designs correct and minimize the risks of having bugs in the implementation. However, formal verification is not a panacea due to limitations such as the cost of verification, inability to verify existing implementations, and human errors involved. Implementation-level model checking has been heavily explored by researchers for the past decades to formally verify whether the underlying implementation of distributed systems have bugs or not. Nevertheless, previous proposals are limited because their invariant checking is not versatile enough to check for the wide spectrum of consistency models, from eventual consistency to strong consistency. In this work, consistency oracles are employed for consistency invariant checking that can be used by implementation-level model checkers to formally verify data consistency model implementations of distributed key-value stores. To integrate consistency oracles with implementation-level distributed system model checkers, the partial-order information obtained via API is leveraged to avoid the exhaustive search during consistency invariant checking. Our evaluation results show that, by using the proposed method for consistency invariant checking, our prototype model checker, VConMC, can detect consistency violations caused by several real-world software bugs in a well-known distributed key-value store, ZooKeeper. [ABSTRACT FROM AUTHOR]