Search

This paper presents an improved mathematical model for calculating the solar test factor (STF) and solar reliability factor (SRF) of a photovoltaic (PV) automated equipment. By employing a unified metrics system and a combined testing suite encompassing various energy-efficient testing techniques, the aim of this paper is to determine a general fault coverage and improve the global SRF of a closed-loop dual-axis solar tracking system. Accelerated testing coupled with reliability analysis are essential tools for assessing the performance of modern solar tracking devices since PV system malfunctioning is directly connected to economic loss, which is an important aspect for the solar energy domain. The experimental results show that the unified metrics system is potentially suitable for assessing the reliability evaluation of many types of solar tracking systems. Additionally, the proposed combined testing platform proves efficient regarding fault coverage (overall coverage of 66.35% for all test scenarios), test time (an average of 275 min for 2864 test cycles), and power consumption (zero costs regarding electricity consumption for all considered test cases) points of view.

As an important asset in software testing, measuring quality attributes of the test suite is important to describe the quality of software. This research proposes a method to measure the test suite quality attributes for white-box testing. The attributes are usability, efficiency, reliability, functionality, portability, and maintainability that are selected from 28 attributes in software quality. By using the proposed method, the test suite quality attributes are calculated with various results of level of quality. The result of test suite quality attribute measurement then proves the validity of its result by the reliability analysis. It is used Cohen’s kappa coefficient to validating the result of test suite quality attributes measurement based on the level of agreement between the result of measurement and expert assessment. Reliability analysis on test suite quality attribute finds the attribute that strongly related based on the minimum percentage of level of agreement value are usability, reliability and functionality. Hence, our proposed method is useful to measure test suite quality attributes.

Object-oriented software systems contain large number of modules which make unit testing, integration testing, and system testing very difficult and challenging. While the aim of unit testing is to show that individual modules are working properly and the aim of the system testing is to determine whether the whole system meets its specifications, the aim of integration testing is to uncover errors in the interactions between system modules. However, it is generally impossible to test all connections between modules because of time and budget constraints. Thus, it is important to focus the testing on the connections presumed to be more error-prone. The goal of this work is to guide software testers where in a software system to focus when performing integration testing to save time and resources. This paper proposes a new approach to predict and rank error-prone connections. We use method level metrics that capture both dependencies and internal complexity of methods. We performed experiments on several Java applications and used error seeding techniques for evaluation. The experimental results showed that our approach is effective for selecting the test focus in integration testing.

External failures like network changes can affect system operation negatively. Mitigation requirements try to prevent or reduce their effects. This paper presents a black box testing approach that tests fail-safe behavior in web applications. Failure mitigation tests are built based on a functional test suite. Mitigation requirements are used to build mitigation models and mitigation tests paths through them. A genetic algorithm is used to generate failure scenarios (failure mitigation test requirements). Weaving rules describe how mitigation test paths are combined with behavioral test paths to create failure mitigation test paths. These are then transformed into an executable test suite. The paper also evaluates the genetic algorithm used to generate test requirements with respect to efficiency and effectiveness. A large case study, a commercial mortgage lending system, is used to explore applicability, scalability, and effectiveness of the approach.

Static testing is used to detect software defects in the earlier phases of the software development lifecycle, which makes the total costs caused by defects lower and the software development project less risky. Different types of static testing have been introduced and are used in software projects. In this paper, we focus on static testing related to data consistency in a software system. In particular, we propose extensions to contemporary static testing techniques based on CRUD matrices, employing cross-verifications between various types of CRUD matrices made by different parties at various stages of the software project. Based on performed experiments, the proposed static testing technique significantly improves the consistency of Data Cycle Test cases. Together with this trend, we observe growing potential of test cases to detect data consistency defects in the system under test, when utilizing the proposed technique.

Software testing being one of the major phases of the software development life cycle is critical in delivering reliable software products. Traditional manual GUI testing has severe limitations like insufficient test coverage, labor intensiveness, complexity involved and cost ineffectiveness. “How much testing is enough or sufficient?” still remains a challenging question. Coverage analysis helps to guide the test engineers regarding test coverage and is used extensively to determine the effectiveness of selected testing methodology. The problems mentioned above related to manual GUI testing prompted the need for automation of GUI testing and coverage analysis. With the rapid emergence of GUI based context free applications, automated testing tools seems less effective. Recently, it has been learnt that concurrent and distributed machines based cluster can be used to reduce the required effort to test GUI in context of time. Proposed system use two different evolutionary algorithms concurrently to gain multiple objectives, based on the fitness functions of maximization of GUI test path coverage (measured as the number of events in an event sequence tested by a particular test case) and simultaneously minimizing the number of test cases. For coverage analysis, two of the very well-known multi objective algorithms; NSGA II and MOPSO from evolutionary domain have been employed. Experiments results of coverage analysis show that accuracy to the tune of 85% was achieved in MOPSO and NSGA II. This high level of accuracy is an indicator of usefulness of proposed fitness function. Both algorithms execute more than 90% of test paths.

The test oracle —a judge of the correctness of the system under test (SUT)—is a major component of the testing process. Specifying test oracles is challenging for some domains, such as real-time embedded systems, where small changes in timing or sensory input may cause large behavioral differences. Models of such systems, often built for analysis and simulation, are appealing for reuse as test oracles. These models, however, typically represent an idealized system, abstracting away certain issues such as non-deterministic timing behavior and sensor noise. Thus, even with the same inputs, the model's behavior may fail to match an acceptable behavior of the SUT, leading to many false positives reported by the test oracle. We propose an automated steering framework that can adjust the behavior of the model to better match the behavior of the SUT to reduce the rate of false positives. This model steering is limited by a set of constraints (defining the differences in behavior that are acceptable) and is based on a search process attempting to minimize a dissimilarity metric. This framework allows non-deterministic, but bounded, behavioral differences, while preventing future mismatches by guiding the oracle—within limits—to match the execution of the SUT. Results show that steering significantly increases SUT-oracle conformance with minimal masking of real faults and, thus, has significant potential for reducing false positives and, consequently, testing and debugging costs while improving the quality of the testing process.

Testing activities for software product lines should be different from that of single software systems, due to significant differences between software product line engineering and single software system development. The cost of testing in software product line is generally higher compared with single software systems; therefore, there should exist a certain balance between cost, quality of final products, and the time of performing testing activities. As decreasing testing cost is an important challenge in software product line integration testing, the contribution of this paper is in introducing a method for early integration testing in software product lines based on feature model (FM) by prioritizing test cases in order to decrease integration testing costs in SPLs. In this method, we focus on reusing domain engineering artifacts and prioritized selection and execution of integration test cases. It also uses separation of concerns and pruning techniques on FMs to help prioritize the test cases. The method shows to be promising when applied to some case studies in the sense that it decreases the costs of performing integration test by about 82% and also detects about 44% of integration faults in domain engineering.

Almost all existing studies consider that product release results in the end of testing phase. However, the modern practices in software industry perceive a post release testing during which organization treats the remaining bugs present in the software to improve product experience for customer and at the same time trying to lower down the possibilities of software failure before testing ends. However, continuing testing after software release is not considered cost-effective for firms. Hence, it is essential to decide the optimal testing stop time under the budget. We comprehend this scenario to design a framework for optimal scheduling policy by considering cost and reliability as two attribute using multi-attribute utility theory. Validation of the obtained results is done on real-life software failure data set of tandem computers.

Software reliability means operational reliability. If we assume that hardware & input values have no error, then the probability that a software system will fulfills its assigned task in a certain environment for predefined number of input case is known as software reliability. There are many approaches for checking the reliability of software, such as object oriented approach, regression testing approach, load testing, stress testing, black box testing, white box testing and many other methods. From several years software researchers have proposed many methods to assess the software reliability, but none of the methods is fit for assessing the reliability in all cases. Some methods are good for assessing the reliability in one case but are worst in other case. In this paper we will describe the artificial neural network based approach to assess the reliability of software. The time allotted to testing team is very less in comparison to time given to coding team which further deteriorates the reliability of software. We will use simulation criteria and queuing modeling in this paper.

Software testing activities are an important process in the software testing life cycle. Therefore, test case generation has become one of the essential steps of testing for software to obtain accuracy guarantee and quality of software. However, the changing of requirements often occurs in the development phase, which causes an effect on the software testing.To solve this problem, this paper proposes a framework for impact analysis of test cases based on changes of use case based requirement specification. The proposed approach verifies and validates the difference of requirement specification which is described in use case pattern on two versions of requirement specification document. Consequently, the patterns of the variable requirements changing to be classified and analysed. These results enable the existing test cases completely reused, partly updated as well as additionally generated. In addition, generating new test case completely supports testing coverage and eliminates test case redundancy by using the effective CCTM method.

Software cost estimation is the important activity while the development of the software. Expenditure assessment is bit complex task as it can be affected by many factors. This factors aids in the calculating of maintenance cost of software. In this paper we have implemented the function point analysis and test point analysis in order to discover the maintenance cost. This is accomplished by using the various techniques to calculate the function point analysis and test point analysis. Along with the calculation of maintenance cost we have also presented the module to assess the reliability of the software from the context of white box testing. Software reliability growth models are aids to evaluate the reliability of the software. This paper presented the analysis of code based SRGM to estimate the reliability.

For several decades both practitioners as well as researchers have contributed immensely towards solving one of the prime concerns of software firms regarding how long to continue testing? Although, efficient testing ensures reliable software but costs a lot to developer in terms of higher testing and market opportunity cost. Hence it’s imperative to obtain optimal testing time of software. Product release marking the end to software testing has been the phenomenon commonly observed in both practice and theory but often leaves customer dissatisfied either in the form of delayed release (prolonged testing phase) or lower reliability (shortened testing phase). To address this issue, current practices in software industry involve a post release testing phase during which organization aims at treating remaining software faults and subsequently enhance product experience for customers. In this paper we present a generalized framework of optimal scheduling policy to determine optimal software release and testing stop time separately, to minimize overall testing cost under reliability and budget constraints. One of the biggest advantages of early release is capitalizing market opportunity while post release testing ensures adequate software reliability. We also propose that software testing & operational phases are governed by different distribution functions in our proposed cost model. Numerical illustration is provided at the end of the paper to validate the proposed cost model.

We extend the theory of input-output conformance (IOCO) testing to accommodate behavioral models of software product lines (SPLs). We present the notions of residual and spinal testing. These notions allow for structuring the test process for SPLs by taking variability into account and extracting separate test suites for common and specific features of an SPL. The introduced notions of residual and spinal test suites allow for focusing on the newly introduced behavior and avoiding unnecessary re-test of the old one. Residual test suites are very conservative in that they require retesting the old behavior that can reach to new behavior. However, spinal test suites more aggressively prune the old tests and only focus on those test sequences that are necessary in reaching the new behavior. We show that residual testing is complete but does not usually lead to much reduction in the test-suite. In contrast, spinal testing is not necessarily complete but does reduce the test-suite. We give sufficient conditions on the implementation to guarantee completeness of spinal testing. Finally, we specify and analyze an example regarding the Ceiling Speed Monitoring Function from the European Train Control System. (C) 2016 The Author(s). Published by Elsevier Inc. AUTO-CAAS, KK HÖG Project EFFEMBAC, VR Project

Modern computer software usually consists of a number of interacting components. Defects pertaining to component interactions may not be visible until the components are integrated. To exercise component interactions, we propose an integration testing approach based on the core concept of contracts (i.e., preconditions and postconditions of component calls in first-order logic). To enable correctness analysis and generation of coverage-based tests (including state coverage, transition or component coverage, and reachability coverage) of a contract-based test model, we transform the test model into an operational model. We have implemented this approach into a framework that has the capacity to generate executable test code in a large collection of programming languages and to run the generated test code in a rich variety of test execution environments. To demonstrate the cost-effectiveness of our approach, we have conducted a series of experiments in controlled environments to measure the performance of test generation and the fault detection capabilities of coverage-based tests. The results have clearly demonstrated that: 1) our approach can quickly generate coverage-based tests when there is enough memory for dealing with the state space of the given test models; 2) reachability coverage is more capable of detecting faults than state coverage and transition coverage; 3) postcondition-based test oracles are more capable of detecting faults than state-based test oracles; and 4) robustness tests, which exercise the situations when the contracts are broken, are important for improving fault detection capability.

Software testers should prioritize test cases so that important ones are run earlier in the regression testing process to reduce the cost of regression testing. Test case prioritization techniques schedule test cases for execution in an order that improves the performance of regression testing. One of the performance goals i.e. the fault detection rate, measures how quickly faults are detected during the testing process. Improved rate of fault dependency detection can provide faster feedback on software and let developers debug the leading faults at first that cause other faults to appear later. Another performance goal i.e. severity detection rate among faults, measures how quickly more severe faults are detected earlier during testing process. Previous studies addressed the second goal, but did not consider dependency among faults. In this paper an algorithm is proposed to prioritize test cases based on rate of severity detection associated with dependent faults. The aim is to detect more severe leading faults earlier with least amount of execution time and to identify the effectiveness of prioritized test case. Index Terms—Software testing, Regression testing, Test case prioritization, Fault dependency, Software quality.

Testing plays a vital role in the evolution and establishment of any quality product as well as any quality system. Testing is essential to prove the correctness (valid output when input is valid, and proper handling techniques when input is invalid) of the system and it is crucial to prove the compatibility of the system with the operating environment. In component-based software systems, various components interact with each other to access as well as provide required functionalities. In such complex systems testing is one of the most important activities. Since component-based software engineering relies on the concept of “use of pre-built and pre-tested components”, our focus here, is on functional testing rather than structural testing. Functional testing emphasizes the behavioural attributes of the components when they interact with each other. Components interact through operands and parameters. In this paper we suggest functional testing strategy and test case generation technique for component-based software. When two components are integrated then they generate some specific effect. This strategy is named Integration-effect graph. This strategy is a black-box technique as it covers the input and output domains only. We have used the graph theory notations to show the integration and interaction among the components.

Background/Objectives: When client wants some change in functionality or in look/design of software, then after implementation of requirement, tester has to retest it. For this tester has to perform regression testing. It is done to verify that any other issue is not pushed into the system, and also verify existing issue if any, So, Regression testing not only is very important process when software change made but also it is very costly too. Methods: This paper proposes a new method based on ratio between number of issues found and run time of Test cases. This paper gives predilection to the original test suite so that the new suite, which is run within a time-constrained execution environment, will have a superior rate of fault detection. Findings: This paper presents a technique that prioritizes regression test cases so that the new ordered test case produces a high productivity. Application/Improvements: This paper assumes data for pre-executed test cases in the form of number of issues found and run time.

Praktek Kerja Industri (PRAKERIN) information system is a system used to record grades and other student data relating to the implementation of work practices in industry. This information system must be error free. If an error occurs can cause losses to stakeholders. To provide assurance that the information system developed has been free from errors, it is necessary to test. Testing is one level of the program implementation process that must be passed in order that an error can be found. Tests that can be said to be good are if the tests were initially found no errors and then errors can be found, even if only a small error. In testing the Praktek Kerja Industri (PRAKERIN) information system it is proposed as a Black Box. Black Box testing has various techniques. Of the many testing methods currently available, the current testing will use the Boundary Value Analysis method. The technique of determining the limits that exist in the lower and upper values to be able to produce the actual value. The test results show that the information system developed has been error free and has good quality.

Over the last decades, white-box search-based techniques have been applied to automate the design and the execution of test cases. While most of the research effort has been devoted to unit-level testing, integration-level test case generation requires to solve several open challenges, such as the combinatorial explosion of conditions or pre-condition failures. This paper summarizes these challenges in white-box testing and highlights possible research directions to overcome them.

Model-based testing is a promising technology for black-box software and hardware testing, in which test cases are generated automatically from high-level specifications. Nowadays, systems typically consist of multiple interacting components and, due to their complexity, testing presents a considerable portion of the effort and cost in the design process. Exploiting the compositional structure of system specifications can considerably reduce the effort in model-based testing. Moreover, inferring properties about the system from testing its individual components allows the designer to reduce the amount of integration testing. In this paper, we study compositional properties of the IOCO-testing theory. We propose a new approach to composition and hiding operations, inspired by contract-based design and interface theories. These operations preserve behaviors that are compatible under composition and hiding, and prune away incompatible ones. The resulting specification characterizes the input sequences for which the unit testing of components is sufficient to infer the correctness of component integration without the need for further tests. We provide a methodology that uses these results to minimize integration testing effort, but also to detect potential weaknesses in specifications. While we focus on asynchronous models and the IOCO conformance relation, the resulting methodology can be applied to a broader class of systems.

Cleanroom software engineering (CSE) reference model is a rigorous incremental model that focuses on defect prevention using sound mathematical principles combined with statistical usage testing (Linger, Trammell, in cleanroom software engineering reference model, 1996, [1]). Similar to the concept of hardware cleanrooms, this model is also used for the development of zero defect and extremely reliable software (Mills, Poore, in Quality Progress, 1988, [2]). Statistical usage testing (SUT) is a technique defined for testing as a part of CSE model [1]. The technique works by performing usage modelling and assigning usage probabilities (Runeson, Wohlin in IEEE Trans Softw Eng 20(6): 494–499, 1994, [3]). Next statistical tests are carried out on the usage models [3]. CSE relies on SUT for testing, and unit testing is not defined in the CSE process (Hausler et al. in IBM Syst J 33(1): 89, 109, 1994, [4]). However, additional testing can be carried out along with SUT depending on the need (Prowell et al. in cleanroom software engineering technology and process, 1999, [5]). The paper presents the usefulness and advantages of applying SUT along with various white box testing techniques. The white box testing techniques used in the paper are data flow testing, control flow testing and mutation testing.

Unit testing is an attractive quality management tool in the software development process, however, practical obstacles make it difficult to use unit tests for performance testing. We present Stochastic Performance Logic, a formalism for expressing performance requirements, together with interpretations that facilitate performance evaluation in the unit test context. The formalism and the interpretations are implemented in a performance testing framework and evaluated in multiple experiments, demonstrating the ability to identify performance differences in realistic unit test scenarios.

Many large-scale software systems must service thousands or millions of concurrent requests. These systems must be load tested to ensure that they can function correctly under load (i.e., the rate of the incoming requests). In this paper, we survey the state of load testing research and practice. We compare and contrast current techniques that are used in the three phases of a load test: (1) designing a proper load, (2) executing a load test, and (3) analyzing the results of a load test. This survey will be useful for load testing practitioners and software engineering researchers with interest in the load testing of large-scale software systems.

Software testing is an important part of project development. Depending on system type and size, it is performed variously. Unit testing is one of the available approaches that is used to ensure that behavior of small software parts is consistent with requirements. It allows to improve software quality and decrease overall costs. Despite the fact that such an approach is commonly judged as a vital concept, it is not usual in control software. In this paper, the comprehensive approach to test the IEC 61131-3 software using unit tests is presented. It supports to create tests in two ways—either in textual and graphical IEC 61131-3 languages or in the CPTest+ dedicated test definition language. The latter is equipped with many advanced features, such as test fixtures and inclusions, parameterized and analog signal extensions, mock objects, as well as a few kinds of suites. The overall solution runs on the developer and testing station; hence, it does not have significant impact on performance of the control program and tests are more reliable and repeatable. To explain the concept, the simple running example is presented in this paper. The described solution has been introduced in the CPDev engineering environment for programming controllers.

This paper introduces a testing strategy that is suitable for testing service-based applications. We describe an architecture that responds to changes of service operation, operation arguments and service composition changes. Our proof-of-concept test system performs runtime testing on our model atomic and composite web services using a random testing technique. A novel change identification method was developed to capture changes at the service interface. The test system is able to identify changes that occur in service operations and operational arguments in a service description of a test candidate. Our approach uses a new method to detect changes in a service inventory. Automated reconfiguration is used to support the continuous operation of the testing systems during a test candidate change.

present paper introduces a new testing process for embedded software development activities in the field of hybrid electric vehicles. The proposed process addresses some deficiencies of the existing testing models in order to improve the performance of HEV embedded software testing activities. The value added is given by the clear split of the test tasks and the level of independence between the test vehicles involved in the test process. The motivation of the research is given by the fact that the automotive sector is heading towards electric vehicles and the level of software complexity in these vehicles is continuously increasing. The advantage of this method is that will allow a smoother offload of testing tasks between locations, depending on project needs. The test model contributes to the practical need of developing software for hybrid and electric vehicles at higher quality standards, faster and with a better cost. Keywordstesting process; software

In this article we propose a technique, called abstract testing, which replaces traditional test cases by abstract test cases. By doing so, fewer test cases are needed, and they are linked more closely to the requirements. Abstract tests can be considered as verification scenarios on the source code level which are derived from the requirements. Checking verification scenarios against the source code is done automatically using a software model checker. We also suggest a migration path from traditional tests to abstract test cases, which provides a smooth transition towards this new technique. Finally, we demonstrate feasibility of abstract testing by a case study from the automotive systems domain.

Summary This paper includes the importance of performance testing of web applications and analyses the application's bottleneck based on hardware, software and resource utilization. The main focus of this paper is performance testing based on load test. It has been seen that everyone wants a very fast application, but at the same time, reliability of the application takes an important role; so, customer's satisfaction is the push for performance testing of a given application. Performance testing determines how fast some aspects of the system perform under a pre-defined workload. The performance testing is calculated by analyzing the production, which comes from the application hosted on the server. The main performance is calculated when business is at its peak by its hits.

Testing is a key activity to assure the quality of concurrent applications. In recent years, a variety of different mechanisms have been proposed to test concurrent software. However, a persistent problem is the high testing cost because of the large number of different synchronization sequences that must be tested. When structural testing criteria are adopted, a large number of infeasible synchronization sequences is generated, increasing the testing cost. Although the use of reachability testing reduces the number of infeasible combination because only feasible synchronization sequences are generated, many synchronization combinations are also generated, and this again results in a testing cost with exponential behavior. This paper presents a new composite approach that uses reachability testing to guide the selection of the synchronization sequences tests according to a specific structural testing criterion. This new composite approach is empirically evaluated in the context of message-passing concurrent programs developed with MPI. The experimental study evaluates both the cost and effectiveness of proposed composite approach in comparison with traditional reachability testing and structural testing. The results confirm that the use of the new composite approach has advantages for testing of concurrent applications. Copyright © 2015 John Wiley & Sons, Ltd.

Failures in reactive embedded systems are often unacceptable. Moreover, effective testing of such systems to detect potential critical failures is a difficult task. We present an automated black box test suite generation technique for reactive systems. The technique is based on dynamic mining of specifications, in form of a finite state machine (FSM), from initial runs. The set of test cases thus produced contain several redundant test cases, many of which are eliminated by a simple greedy test suite reduction algorithm to give the final test suite. The effectiveness of tests generated by our technique was evaluated using five case studies from the reactive embedded domain. Results indicate that a test suite generated by our technique is promising in terms of effectiveness and scalability. While the test suite reduction algorithm removes redundant test cases, the change in effectiveness of test suites due to this reduction is examined in the experimentation. We present our specification mining based test suite generation technique, the test suite reduction technique and results on industrial case studies.

Software Testing is one of the expensive activities in Software development lifecycle but it used in to assure the quality of the Software. One of the important focuses in software engineering process is lessen the effort of software testing and cut down the cost & time of testing. This paper presents a unified method for reducing the test cases by adopting the hypothesis of Genetic algorithm such as crossover, mutation and crowding distance to elucidate the given problem and results in the minimum set of test cases with the ability to cover all the faults in minimum time.

 Abstract—Software testing plays a vital role in developing software that is freedom from bugs and defects. Manual test is a cost and time consuming process although it may find many defects in a software application. If the testing process could be automated, the cost of developing software could be reduced obviously. The most critical part of the testing process is generation of test cases. Model based test path generation approaches identify faults in the implementation at early design phase, reduce the software development time, and inspire developer to improve design quality. The proposed approach focuses on Model Based testing for automatic best test path generation. In this approach, the proposed system generates all possible test paths and optimizes a test path from these paths. The proposed system will reduce the processing time by using XMI based activity diagram, validate the generated test paths with Cyclomatic Complexity and choose the best test path without dependence on others.

Cyber-Physical Systems (CPS) have become one of the core-enabling technologies for multiple domains, such as manufacturing, healthcare, energy and transportation. Furthermore, these domains are demanding CPS to be highly-configurable in order to respond to multiple and changing market requirements. Testing these Highly-Configurable Cyber-Physical Systems (HCCPS) is challenging. First, when working with CPSs, considerable time is required in order to tackle physical processes during testing. And secondly, in highly-configurable systems, a large number of system variants need to be tested. Consequently, reducing HCCPS testing time is essential.In this context, a research work is presented to reduce the overall testing time of HCCPS, focusing on a merged strategy of product and test cases optimisation. In particular, two approaches are proposed in order to achieve the testing time reduction. The first approach aims to reduce the HCCPS testing time by an iterative allocation of products and test cases. The second approach aims to reduce the HCCPS testing time by a feedback driven dynamic and iterative allocation of products and test cases.A preliminary experiment has been undertaken to test the iterative allocation approach. In this experiment, products to be tested are selected and prioritised. Next, multiple testing iterations are perform until the time-budget is consumed. In each iteration a small number of test cases are allocated for each of the products to be tested. The experiment was evaluated with an academic HCCPS and preliminary results suggest that the proposed approach reduces the fault detection time when compared with traditional approaches.

Background: In the era of digitalization, context awareness has become more important to allow software systems adaptation. Therefore, the quality assurance of such systems must consider the variation of context. However, there is a lack of software quality technologies doing so, which can increase the risk of failure of these systems. Objective/Aim: To evidence recent advances regarding the testing of context-aware software systems (CASS), focusing on the abstraction that context should freely vary during test execution as it does in production environments. Method: Based on knowledge acquired with quasi-Systematic Literature Reviews, we evaluate current testing approaches and discuss benefits and limitations of applying the observation above about CASS.Results: It was not possible to observe any software testing technology supporting the unconstrained variation of context during testing a CASS. The practitioners, meanwhile, can use three main evidence-based strategies to test CASS: to assure functional correctness before turning to testing context-aware requirements; to design test cases to target context variables and to take advantage of automatic testing tools as much as possible.Conclusions: To allow context to vary during testing freely, seems plausible. It provides a new frame of thought to enable the design of novel technologies to improve our capacity of testing CASS.

Testing of software is a worthwhile aspect of software development life cycle. Effective and efficient test cases must be designed to test the software which will reduce the testing cost, time and effort. Nowadays, testing an aspect-oriented program is becoming a challenge for the testers. This paper proposes a novel approach to generate test case scenarios for an aspect oriented program derived from “Unified Modeling Language” (UML). Combined heuristic approach using Genetic and Fuzzy Clustering Algorithm is being employed to reduce or minimize the number of test case scenarios. This paper caters to save time, cost and effort by efficiently generating optimal and minimal test sequences in order to obtain the aspectual branch coverage criteria for testing the behavior of AOP. This approach has been applied on a well-known aspect oriented software testing problem namely, “Heating kettle problem”. Effectiveness of proposed technique is justified by analyzing it with GA and Fuzzy method individually. The results show that proposed approach tremendously reduces the number of test case scenarios by satisfying aspectual branch coverage criteria.

Testing is a fundamental task to ensure software quality. Regression testing aims to ensure that changes to software do not introduce new failures. As resources are often limited and testing comprises a vast amount of test cases, different regression strategies have been proposed to reduce testing effort by selecting or prioritizing important test cases, e.g., code coverage (to ensure a sufficient testing depth). However, in system testing, source code is often not available creating a black-box system. In this paper, we introduce an automated, multi-objective test case selection technique in black-box systems using genetic algorithms. We define seven different objectives, based on meta-data, allowing a flexible test case selection for a variety of systems. For evaluation, we apply our technique on two different subject systems assessing the feasibility and suitability of our test case selection approach. Results indicate that our approach is applicable based on different data available and is able to outperform random test case selection and retest-all.

This presents a new model to support empirical failure probability estimation for a software-intensive system. The new element of the approach is that it combines the results of testing using a simulated hardware platform with results from testing on the real platform. This approach addresses a serious practical limitation of a technique known as statistical testing. This limitation will be called the test time expansion problem (or simply the 'time problem'), which is that the amount of testing required to demonstrate useful levels of reliability over a time period T is many orders of magnitude greater than T. The time problem arises whether the aim is to demonstrate ultra-high reliability levels for protection system, or to demonstrate any (desirable) reliability levels for continuous operation ('high demand') systems. Specifically, the theoretical feasibility of a platform simulation approach is considered since, if this is not proven, questions of practical implementation are moot. Subject to the assumptions made in the paper, theoretical feasibility is demonstrated.

Statistical testing based on a Markov chain usage model, as a rigorous testing method, has been around for more than two decades. Through the comprehensive application of statistical science to the testing of software, it provides audit trails of evidence to support correctness arguments for a software-intensive system as well as a decision that the system is of requisite quality for its intended use. This paper reports a real-world case study in which we applied standard statistical testing practices to the phone flag copy testing problem at the site of our industrial collaborator, and presents our solution from problem formalization, usage modeling and model analysis to test case generation and analysis. Our results helped evaluate the coverage of a heuristically generated test suite, and shed light on what other test cases to craft/generate to improve test coverage and adequacy, as well as reliability estimates both at the arc (usage event) level and at the path (system) level.

Combinatorial interaction testing (CIT) stands as one of the efficient testing techniques that have been used in different applications recently. The technique is useful when there is a need to take the interaction of input parameters into consideration for testing a system. The key insight the technique is that not every single parameter may contribute to the failure of the system and there could be interactions among these parameters. Hence, there must be combinations of these input parameters based on the interaction strength. This technique has been used in many applications to assess its effectiveness. In this paper, we are addressing the effectiveness of CIT for a real-world case study using model-based mutation testing experiments. The contribution of the paper is threefold: First we introduce an effective testing application for CIT; Second, we address the effectiveness of increasing the interaction strength beyond the pairwise (i.e., interaction of more than two parameters); Third, model-based mutation testing is used to mutate the input model of the program in contrast to the traditional code-based mutation testing process. Experimental results showed that CIT is an effective testing technique for this kind of application. In addition, the results also showed the usefulness of model-based mutation testing to assess CIT applications. For the subject of this case study, the results also indicate that 3-way test suite (i.e., interaction of three parameters) could detect new faults that can not be detected by pairwise.

Fuzz testing and symbolic test generation both face their own challenges. While symbolic testing has scalability issues, fuzzing cannot uncover faults which require carefully engineered inputs. In this paper I propose a combination of both approaches, compensating weaknesses of each approach with the strength of the other approach. I present my plans for evaluation, which include applications of the hybrid tool to programs which neither of the approaches can handle on its own.

Regression testing is a process used to measure the validity of the system during software maintenance. Regression testing process is very expensive and must be introduced each time a modification occurs in software to ensure that the system still work and that the new modification doesn't cause any bugs, this process depends on selecting test cases from a test suite. Selection of test cases is very critical since it affect the regression testing time and effort, so that many algorithms exist to enhance regression testing process. One of the methods used to make enhancements is to select test cases using prioritization testing techniques. Prioritization techniques find the bugs early to improve regression testing efficiency by prioritizing the test cases. In this paper many regression testing prioritization techniques were reviewed and analyzed.

Software testing is the process of testing, verifying and validating the user's requirements. During whole software development, testing is an ongoing process. Black Box, White Box testing and grey box testing are the three main types of software testing. In Black box testing, user doesn't know intrinsic logics and design of system. In white box testing, Tester knows the intrinsic logic of code. In Grey box testing, Tester has little bit knowledge about the internal structure, code and working of the system. It is commonly used in case of Integration testing. Load testing is the type of testing which helps us to analyze the performance of the system under heavy load or under Zero load. With the help of a automated load Testing Tools, we can do it in better way. The intention for writing this research is to carry out a comparison of four automated load testing tools i.e. Apache JMeter, HP LoadRunner, Microsoft Visual Studio (TFS), Siege based on certain criteria i.e. test scripts generation, plug-in support, result reports, application support and cost. The main focus is to study and analyze these load testing tools and identify which tool is best and more efficient [10]. We assume this comparative analysis can help in selecting the most appropriate tool and motivates the use of open source load testing tools.

Nowadays, any communicating or autonomous systems rely on high-quality software-based components. To ensure a sufficient level of quality, these components must be thoroughly verified before being released and being deployed in operational settings. Regression testing is a crucial verification process that executes any new release of a software-based component against previous versions of the component, with existing test cases. However, the selection of test cases in regression testing is challenging as the time available for testing is limited and some selection criteria must be respected. This problem, coined as Test Suite Reduction (TSR), is usually addressed by validation engineers through manual analysis or by using approximation techniques. Even if the underlying optimization problem is untractable in theory, solving it in practice is crucial when there are pressing needs to release high-quality components while at the same time reducing the time-to-market of new software releases. In this paper, we address the TSR problem with sound Artificial intelligence techniques such as Constraint Programming (CP) and global constraints. By associating each test case a cost-value aggregating distinct criteria, such as execution time, priority or importance due to the error-proneness of each test case, we propose several constraint optimization models to find a subset of test cases covering all the test requirements and optimizing the overall cost of selected test cases. Our models are based on a combination of NVALUE, GLOBALCARDINALITY, and SCALAR_PRODUCT, three well-known global constraints that can faithfully encode the coverage relation between test cases and test requirements. Our contribution includes the reuse of existing preprocessing rules to simplify the problem before solving it and the design of structure-aware heuristics, which take into account the notion of costs, associated with test cases. The work presented in this paper has been motivated by an industrial application in the communication domain. Our overall goal is to develop a constraint-based approach of test suite reduction that can be deployed to test a complete product line of conferencing systems in continuous delivery mode. By implementing this approach in a software prototype tool and experimentally evaluated it on both randomly generated instances and industrial instances, we hope to foster a quick adoption of the technology.

Conference of 10th IEEE International Conference on Software Testing, Verification and Validation, ICST 2017 ; Conference Date: 13 March 2017 Through 17 March 2017; Conference Code:127845; International audience; Automated white-box testing is a major issue in software engineering. In previous work, we introduced LTest, a generic and integrated toolkit for automated white-box testing of C programs. LTest supports a broad class of coverage criteria in a unified way (through the label specification mechanism) and covers most major parts of the testing process - including coverage measurement, test generation and detection of infeasible test objectives. However, the original version of LTest was unable to handle several major classes of coverage criteria, such as MCDC or dataflow criteria. Moreover, its practical applicability remained barely assessed. In this work, we present a significantly extended version of LTest that supports almost all existing testing criteria, including MCDC and some software security properties, through a native support of recently proposed hyperlabels. We also provide a more realistic view on the practical applicability of the extended tool, with experiments assessing its efficiency and scalability on real-world programs.

Exhaustive testing of highly configurable software developed in continuous integration is rarely feasible in practice due to the configuration space of exponential size on the one hand, and strict time constraints on the other. This entails using selective testing techniques to determine the most failure-inducing test cases, conforming to highly-constrained time budget. These challenges have been well recognized by researchers, such that many different techniques have been proposed. In practice, however, there is a lack of efficient tools able to reduce high testing effort, without compromising software quality. In this paper we propose a test suite optimization technology TITAN, which increases the time-and cost-efficiency of testing highly configurable software developed in continuous integration. The technology implements practical test prioritization and minimization techniques, and provides test traceability and visualization for improving the quality of testing. We present the TITAN tool and discuss a set of methodological and technological challenges we have faced during TITAN development. We evaluate TITAN in testing of Cisco's highly configurable software with frequent high quality releases, and demonstrate the benefit of the approach in such a complex industry domain.

To test a Software Product Line (SPL), the test artifacts and the techniques must be extended to support variability. In general, when new SPL products are developed, more tests are generated to cover new or modified features. A dominant source of extra effort for such tests is the concretization of newly generated tests. Thus, minimizing the amount of new nonconcretized tests required to perform conformance testing on new products reduces the overall test effort. In this paper, we propose a test reuse strategy for conformance testing of SPL products that aims at reducing test effort. We use incremental test generation methods based on finite state machines (FSMs) to maximize test reuse. We combine these methods with a selection algorithm used to identify non-redundant concretized tests. We illustrate our strategy using examples and a case study with an embedded mobile SPL. The results indicate that our strategy can save up to 36% of test effort in comparison to current test reuse strategies for the same fault detection capability.

The base-choice criterion was proposed as a suitable technique for testing software based on its nominal choice of input parameters. Test cases are created based on this strategy by varying the values of one input parameter at a time while keeping the values of the other parameters fixed on the base choice. However, this strategy might not be as effective when used on industrial control software for testing timed behaviour. We propose to incorporate time as another parameter when generating and executing tests by defining the timed base-choice coverage criterion. We performed an empirical evaluation using 11 industrial programs written in the IEC 61131-3 programming language. We found that tests generated for timed base-choice criterion show better code coverage (7% improvement) and fault detection (27% improvement) in terms of mutation score than tests satisfying base-choice coverage criterion. The results demonstrate the feasibility of applying timed base-choice criterion for testing industrial control software. &nbsp