Search

The purpose of this study is to identify and classify studies published on software testing techniques applied to blockchain systems. Previously published reviews in related areas have a narrow focus and/or do not follow a systematic review protocol. We conducted a systematic mapping based on an initial selection of 1025 studies. A rigorous selection process resulted in a final pool of 17 primary studies. These studies are categorized with respect to the employed testing methods, considered quality attributes, and functionality. We observe that most of the publications focus on testing functional correctness or security, whereas the testing of runtime performance attracts less attention. Existing approaches mostly employ fuzz testing or mutation testing. Search-based testing is usually combined with these techniques. The application of model-based testing is rare. The adaptability of fuzz testing and model-based testing techniques to changing blockchain platforms and languages remains a concern. On the other hand, performance and scalability issues are noted for search-based techniques and mutation testing. The use and integration of multiple testing techniques also stand out as a viable research direction. [ABSTRACT FROM AUTHOR]

Metamorphic testing is a valuable approach to verifying machine learning programs where traditional oracles are unavailable or difficult to apply. This paper proposes a technique to prioritize metamorphic relations (MRs) in metamorphic testing for machine learning and deep learning systems, aiming to enhance early fault detection. We introduce five metrics based on diversity in source and follow-up test cases to prioritize MRs. The effectiveness of our proposed prioritization methods is evaluated on three machine learning and one deep learning algorithm implementation. We compare our approach against random-based, fault-based, and neuron activation coverage-based MR ordering. The results show that our data diversity-based prioritization performs comparably to fault-based prioritization, reducing fault detection time by up to 62% compared to random MR execution. Our proposed metrics outperformed neuron activation coverage-based prioritization, providing 5–550% higher fault detection effectiveness. Overall, our approach to prioritizing metamorphic relations leads to increased fault detection effectiveness and reduced average fault detection time. This improvement in efficiency can result in significant time and cost savings when applying metamorphic testing to machine learning and deep learning systems. [ABSTRACT FROM AUTHOR]

Faulty Interaction Localization (FIL) is a process to identify which combination of input parameter values induced test failures in combinatorial testing. An accurate and fast FIL provides helpful information to fix defects causing the test failure. One type of conventional FIL approach, which analyzes test results of whole test cases and estimates the suspiciousness of each combination, has two main concerns; (1) the accuracy is not enough, (2) the huge time cost is sometimes needed. In this paper, we propose two novel approaches to improve those concerns. FROGa attempts to estimate suspiciousness more accurately using logistic regression analysis. FROGb attempts to estimate failure-inducing combinations at high speed by estimating the subsets of them using logistic regression analysis and exploring just their supersets. Through evaluation experiments using a large number of artificial test results based on several real software systems, we observed that FROGa has very high accuracy, and FROGb can drastically reduce time cost for targets that have been difficult to complete by the conventional method. [ABSTRACT FROM AUTHOR]

In this research, authors give a literature analysis of the methods used to detect and anticipate security risks in software testing by using a number of deep learning models. The purpose of this study is to conduct a literature review on the use of deep learning models in software testing for the purpose of detecting and predicting security issues. Moreover, the motive of this work is to analyze the exiting techniques related as software testing model and its application oriented performance. A thorough search of the available literature across several databases and resources forms the basis of this evaluation. There are a total of 69 publications found via the search, 34 of which are original research. Convolutional neural networks, long short-term memory networks, generative adversarial networks, and capsule networks are only some of the state-of-the-art methods for identifying and predicting security risks that are the topic of this research study. Datasets, performance indicators, and assessment criteria are all dissected for this examination. Finally, the benefits and drawbacks of using deep learning models to detect and anticipate software security flaws during testing are outlined in this article. Software testing experts and academics will benefit from this study since it will shed light on the present state of the art in the area and provide direction for the creation of innovative approaches to the detection and forecasting of security risks in software testing. [ABSTRACT FROM AUTHOR]

Few mobile robot developers already test their software on simulated robots in virtual environments or sceneries. However, the majority still shy away from simulation-based test campaigns because it remains challenging to specify and execute suitable testing scenarios, that is, models of the environment and the robots' tasks. Through developer interviews, we identified that managing the enormous variability of testing scenarios is a major barrier to the application of simulation-based testing in robotics. Furthermore, traditional CAD or 3Dmodelling tools such as SolidWorks, 3ds Max, or Blender are not suitable for specifying sceneries that vary significantly and serve different testing objectives. For some testing campaigns, it is required that the scenery replicates the dynamic (e.g., opening doors) and static features of real-world environments, whereas for others, simplified scenery is sufficient. Similarly, the task and mission specifications used for simulation-based testing range from simple point-to-point navigation tasks to more elaborate tasks that require advanced deliberation and decision-making. We propose the concept of composable and executable scenarios and associated tooling to support developers in specifying, reusing, and executing scenarios for the simulation-based testing of robotic systems. Our approach differs from traditional approaches in that it offers a means of creating scenarios that allow the addition of new semantics (e.g., dynamic elements such as doors or varying task specifications) to existing models without altering them. Thus, we can systematically construct richer scenarios that remain manageable. We evaluated our approach in a small simulation-based testing campaign, with scenarios defined around the navigation stack of a mobile robot. The scenarios gradually increased in complexity, composing new features into the scenery of previous scenarios. Our evaluation demonstrated how our approach can facilitate the reuse of models and revealed the presence of errors in the configuration of the publicly available navigation stack of our SUT, which had gone unnoticed despite its frequent use. [ABSTRACT FROM AUTHOR]

Quality and productivity needs are considered together in software. For this reason, any existing software should be tested automatically with test automation. Software test automations is automated software testing activities. Automating constantly manually written tests, on the other hand, saves time, reduces error rates, produces better quality software, and reduces costs. This study aims to produce an automatic unit testing framework that is planned to work in run-time on software products. This developed application performs unit test transformations that can respond to the desired test scenarios on the product being studied.. Java agent is used as the basis of all these transformations. All information about the objects, methods, and variables of the sample java classes to be worked on is converted into data in run-time using byte code. During this transformation, information is saved in the database, and unit tests are created automatically through the template engine. Compared to the products developed on automatic unit test generation in the literature, the opcode parsing method was developed for this study. This method reads a byte code at run-time, uses the properties of the java class it belongs to, and automatically creates the unit test class and test methods. The study can also examine different object definitions and conditional and loop structures within a method and produce alternative test scenarios. The automatic unit test scenario produced has been turned into a flexible framework that can encounter minimum errors at run-time. Considering the scarcity of studies in the field of national software testing; It is thought that the automatic unit test generation product developed within the scope of this study, using byte code, will contribute to the work area. [ABSTRACT FROM AUTHOR]

As software projects evolve and grow in size and complexity, so do their test suites. Test suite reduction (TSR) aims at reducing the size of a test suite by removing redundant and obsolete test cases based on a coverage metric while preserving its fault detection capabilities. The contributions of this paper are twofold: (1) We examine a lesser‐known coverage criterion, that is, checked coverage. Checked coverage not only investigates if a part of the code was executed but also if it was checked by a test oracle. In an empirical evaluation, we performed TSR based on different reduction algorithms, coverage metrics, and open‐source Java projects with our own TSR tool to determine the most effective and efficient combination of metric and method. (2) Given the results of the first evaluation, we further investigate the potential of parameter optimization in regard to a genetic reduction algorithm. In particular, we focus on finding a general setting for the parameters crossover rate and mutation rate such that test suites can be reduced in a reasonable time while maintaining a high fault detection power. [ABSTRACT FROM AUTHOR]

Copyright of Chinese Journal of Network & Information Security is the property of Beijing Xintong Media Co., Ltd. and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Copyright of Computer Measurement & Control is the property of Magazine Agency of Computer Measurement & Control and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

As software vulnerabilities can cause cybersecurity threats and have severe consequences, it is necessary to develop effective techniques to discover such vulnerabilities. Fuzzing is one of the most widely employed approaches that has been adapted for software testing. The mutation-based fuzzing approach is currently the most popular. The state-of-the-art American Fuzzy Lop (AFL) selects mutations randomly and lacks knowledge of mutation operations that are more helpful in a particular stage. This study performs a systematic review to identify and analyze existing approaches that optimize the selection of mutation operations. The main contributions of this work are to draw attention to the importance of mutation operator selection, identify optimization algorithms for mutation operator selection, and investigate their impact on fuzzing testing in terms of code coverage and finding new vulnerabilities. The investigation shows the effectiveness and advantages of optimizing the selection of mutation operations to achieve higher code coverage and find more vulnerabilities. [ABSTRACT FROM AUTHOR]

In software that is free from defects or errors, a fixed method and technique of software testing is required. For this step, an Information System is needed to help eliminate software defects, so that testing or testing work is not a burden on software costs but is a perfection of software development work so that Zero Defect Application Software can be realized. From several software testing methods and techniques, a strategy suitable for the size of a software development project is required. So even though the software development project is small, it still requires a suitable test. And for software development testing to have reliable quality, it is necessary to measure and criteria for each software testing practice, so that a software development project can be determined in quality. To support software testing work, an Information System is needed that supports the need for processors to assess a quality software development test. With the Software Defect Elimination Information System, an objective quality of software development testing can be produced. [ABSTRACT FROM AUTHOR]

Summary: To ensure the quality of software systems, software engineers can make use of a variety of quality assurance approaches, for example, software testing, modern code review, automated static analysis, and build automation. Each of these quality assurance practices have been studied in depth in isolation, but there is a clear knowledge gap when it comes to our understanding of how these approaches are being used in conjunction, or not. In our study, we broadly investigate whether and how these quality assurance approaches are being used in conjunction in the development of 1454 popular open source software projects on GitHub. Our study indicates that typically projects do not follow all quality assurance practices together with high intensity. In fact, we only observe weak correlation among some quality assurance practices. In general, our study provides a deeper understanding of how existing quality assurance approaches are currently being used in Java‐based open source software development. Besides, we specifically zoom in on the more mature projects in our dataset, and generally we observe that more mature projects are more intense in their application of the quality assurance practices, with more focus on their ASAT usage, and code reviewing, but no strong change in their CI usage. [ABSTRACT FROM AUTHOR]

Copyright of Ordnance Industry Automation is the property of Editorial Board for Ordnance Industry Automation and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Software is a part of most complex systems used for different purposes in today's world. The reliability of these systems must be verified before putting them to use, because their malfunction may lead to catastrophic failure, the loss of human lives, or huge costs. Software reliability can be verified by performing black‐box testing, without knowledge of the complex internal structure of these systems. In this work, an embedded IoT alarm system for detection of intruders and potential robbery is proposed. It is composed of three subsystems: embedded IoT, web‐server, and mobile phone subsystems. Cause‐effect graphing technique was applied for generating three types of black‐box test suites, by using the forward‐propagation and two different minimization algorithms. These test suites were used for black‐box testing the system, which was performed after injecting a maximum of 12 faults into the system in 100 iterations. The resulting software quality and reliability metrics showed that the forward‐propagation algorithm always achieves maximum fault detection rate (FDR), and this algorithm should be used when the highest demand for reliability must be met. The optimized minimization should only be used in cases where testing is very expensive, because it did not find faults in all testing iterations. The basic optimization retained high FDR, while reducing the impact of the minimized test suite and achieved a significant increase of test effect coverage values. [ABSTRACT FROM AUTHOR]

Metamorphic Testing (MT) alleviates the oracle problem by defining oracles based on metamorphic relations (MRs) that govern multiple related inputs and their outputs. However, designing MRs is challenging, as it requires domain-specific knowledge. This hinders the widespread adoption of MT. We observe that developer-written test cases can embed domain knowledge that encodes MRs. Such encoded MRs could be synthesized for testing not only their original programs but also other programs that share similar functionalities. In this article, we propose MR-Scout to automatically synthesize MRs from test cases in open-source software (OSS) projects. MR-Scout first discovers MR-encoded test cases (MTCs), and then synthesizes the encoded MRs into parameterized methods (called codified MRs), and filters out MRs that demonstrate poor quality for new test case generation. MR-Scout discovered over 11,000 MTCs from 701 OSS projects. Experimental results show that over 97% of codified MRs are of high quality for automated test case generation, demonstrating the practical applicability of MR-Scout. Furthermore, codified-MRs-based tests effectively enhance the test adequacy of programs with developer-written tests, leading to 13.52% and 9.42% increases in line coverage and mutation score, respectively. Our qualitative study shows that 55.76% to 76.92% of codified MRs are easily comprehensible for developers. [ABSTRACT FROM AUTHOR]

Testing Autonomous Driving Systems (ADSs) is crucial to ensure their reliability when navigating complex environments. ADSs may exhibit unexpected behaviours when presented, during operation, with driving scenarios containing features inadequately represented in the training dataset. To address this shift from development to operation, developers must acquire new data with the newly observed features. This data can be then utilised to fine tune the ADS, so as to reach the desired level of reliability in performing driving tasks. However, the resource-intensive nature of testing ADSs requires efficient methodologies for generating targeted and diverse tests. In this work, we introduce a novel approach, DeepAtash-LR, that incorporates a surrogate model into the focused test generation process. This integration significantly improves focused testing effectiveness and applicability in resource-intensive scenarios. Experimental results show that the integration of the surrogate model is fundamental to the success of DeepAtash-LR. Our approach was able to generate an average of up to 60× more targeted, failure-inducing inputs compared to the baseline approach. Moreover, the inputs generated by DeepAtash-LR were useful to significantly improve the quality of the original ADS through fine tuning. [ABSTRACT FROM AUTHOR]

Quantum computing has emerged as a promising field with the potential to revolutionize various domains by harnessing the principles of quantum mechanics. As quantum hardware and algorithms continue to advance, developing high-quality quantum software has become crucial. However, testing quantum programs poses unique challenges due to the distinctive characteristics of quantum systems and the complexity of multi-subroutine programs. This article addresses the specific testing requirements of multi-subroutine quantum programs. We begin by investigating critical properties by surveying existing quantum libraries and providing insights into the challenges of testing these programs. Building upon this understanding, we focus on testing criteria and techniques based on the whole testing process perspective, spanning from unit testing to integration testing. We delve into various aspects, including IO analysis, quantum relation checking, structural testing, behavior testing, integration of subroutine pairs, and test case generation. We also introduce novel testing principles and criteria to guide the testing process. We conduct comprehensive testing on typical quantum subroutines, including diverse mutants and randomized inputs, to evaluate our proposed approach. The analysis of failures provides valuable insights into the effectiveness of our testing methodology. Additionally, we present case studies on representative multi-subroutine quantum programs, demonstrating the practical application and effectiveness of our proposed testing principles and criteria. [ABSTRACT FROM AUTHOR]

Voice-based virtual assistants are becoming increasingly popular. Such systems provide frameworks to developers for building custom apps. End-users can interact with such apps through a Voice User Interface (VUI), which allows the user to use natural language commands to perform actions. Testing such apps is not trivial: The same command can be expressed in different semantically equivalent ways. In this article, we introduce VUI-UPSET, an approach that adapts chatbot-testing approaches to VUI-testing. We conducted an empirical study to understand how VUI-UPSET compares to two state-of-the-art approaches (i.e., a chatbot testing technique and ChatGPT) in terms of (i) correctness of the generated paraphrases, and (ii) capability of revealing bugs. To this aim, we analyzed 14,898 generated paraphrases for 40 Alexa Skills. Our results show that VUI-UPSET generates more bug-revealing paraphrases than the two baselines with, however, ChatGPT being the approach generating the highest percentage of correct paraphrases. We also tried to use the generated paraphrases to improve the skills. We tried to include in the voice interaction models of the skills (i) only the bug-revealing paraphrases, (ii) all the valid paraphrases. We observed that including only bug-revealing paraphrases is sometimes not sufficient to make all the tests pass. [ABSTRACT FROM AUTHOR]

In today's rapidly evolving software environment, the graphical user interface (GUI) plays a crucial role in providing intuitive, user-friendly interaction. However, traditional intrusive GUI testing methods often face challenges such as interrupting user workflows, requiring significant manual effort and insufficient test scenario coverage. Non-intrusive testing methods, such as Robotic Process Automation (RPA), offer a solution to validate GUI functionality without modifying the application's code or affecting the user experience. RPA systems automate repetitive tasks by simulating user interactions, becoming valuable tools in GUI testing. However, challenges like limited computational resources, time constraints, or restricted exploration capabilities may limit the efficiency of individual RPA agents, thus restricting coverage and effectiveness. To address this issue, this study explores the performance of a single RPA agent versus an RPA cluster under different testing conditions, using three popular testing methods: Monkey, Stoat and Q-testing. Experimental results indicate that an RPA cluster outperforms a single RPA in GUI coverage and error detection, making a significant contribution to the field of non-intrusive GUI exploration testing. The findings of this study provide directions for future research to ensure the delivery of high-quality mobile applications. [ABSTRACT FROM AUTHOR]

The internet key exchange (IKE) protocol, integral to the authentication and key negotiation process within the Internet Protocol Security (IPSec) framework, has been utilized for the protection of IP communications. Given the complex protocol logic, security vulnerabilities in the implementation of the IKE protocol are inevitably present. Fuzz testing, recognized as an effective means of detecting potential vulnerabilities in protocol implementations, has been conventionally applied. However, the direct application of existing fuzzing tools to the IKE protocol has been found to present limitations, such as the generation of low-quality test cases and difficulty in exploring deep states. To address these issues, a mutation strategy based on the grammar of the IKE protocol was designed, aiming to reduce the generation of invalid test cases while increasing the diversity of generated test cases. Additionally, an evolutionary strategy-based mutation scheduling scheme was introduced, which automatically optimized the probability distribution of mutation operators, further increasing the likelihood of generating high-quality test cases. A message handler was designed to maintain protocol interaction context information and perform cryptographic operations, thereby supporting testing of the IKE protocol under black-box conditions. This enabled the exploration of deep protocol interaction behavior and state space. Utilizing the aforementioned methods, an IKE protocol stateful fuzz testing tool named IKEChecker was implemented, supporting testing of both IKEv1 and IKEv2 protocols. Testing was conducted on two widely used open-source IKE protocol implementations, strongSwan and Libreswan, resulting in the revelation of 4 undisclosed vulnerabilities. By comparing IKEChecker with other fuzz testing tools, its efficiency in vulnerability detection was evaluated.