Search

System level testing of industrial data processing software poses several challenges. Input data can be very large, even in the order of gigabytes, and with complex constraints that define when an input is valid. Generating the right input data to stress the system for robustness properties (e.g. to test how faulty data is handled) is hence very complex, tedious and error prone when done manually. Unfortunately, this is the current practice in industry. In previous work, we defined a methodology to model the structure and the constraints of input data by using UML class diagrams and OCL constraints. Tests were automatically derived to cover predefined fault types in a fault model. In this paper, to obtain more effective system level test cases, we developed a novel search-based test generation tool. Experiments on a real-world, large industrial data processing system show that our automated approach can not only achieve better code coverage, but also accomplishes this using significantly smaller test suites.

This paper presents the result of our experience with the application of runtime verification, testing and static analysis techniques to several industrial projects. We discuss the eight most relevant challenges that we experienced, and the strategies that we elaborated to face them.

Automatic testing, although useful, is still quite ineffective against faults that do not cause crashes or uncaught exceptions. In the majority of the cases automatic tests do not include oracles, and only in some cases they incorporate assertions that encode the observed behavior instead of the intended behavior, that is if the application under test produces a wrong result, the synthesized assertions will encode wrong expectations that match the actual behavior of the application. In this paper we present ZoomIn, a technique that extends the fault-revealing capability of test case generation techniques from crash-only faults to faults that require non-trivial oracles to be detected. ZoomIn exploits the knowledge encoded in the manual tests written by developers and the similarity between executions to automatically determine an extremely small set of suspicious assertions that are likely wrong and thus worth manual inspection. Early empirical results show that ZoomIn has been able to detect 50% of the analyzed non-crashing faults in the Apache Commons Math library requiring the inspection of less than 1.5% of the assertions automatically generated by EvoSuite.

System level testing of industrial data processing software poses several challenges. Input data can be very large, even in the order of gigabytes, and with complex constraints that define when an input is valid. Generating the right input data to stress the system for robustness properties (e.g. to test how faulty data is handled) is hence very complex, tedious and error prone when done manually. Unfortunately, this is the current practice in industry. In previous work, we defined a methodology to model the structure and the constraints of input data by using UML class diagrams and OCL constraints. Tests were automatically derived to cover predefined fault types in a fault model. In this paper, to obtain more effective system level test cases, we developed a novel search-based test generation tool. Experiments on a real-world, large industrial data processing system show that our automated approach can not only achieve better code coverage, but also accomplishes this using significantly smaller test suites.

Testing the correct behaviour of data processing systems in the presence of faulty data is extremely expensive. The data structures processed by these systems are often complex, with many data fields and multiple constraints among them. Software engineers, in charge of testing these systems, have to handcraft complex data files or databases, while ensuring compliance with the multiple constraints to prevent the generation of trivially invalid inputs. In addition, assessing test results often means analysing complex output and log data. Though many techniques have been proposed to automatically test systems based on models, little exists in the literature to support the testing of systems where the complexity is in the data consumed in input or produced in output, with complex constraints between them. In particular, such systems often need to be tested with the presence of faults in the input data, in order to assess the robustness and behaviour of the system in response to such faults. This paper presents an automated test technique that relies upon six generic mutation operators to automatically generate faulty data. The technique receives two inputs: field data and a data model, i.e. a UML class diagram annotated with stereotypes and OCL constraints. The annotated class diagram is used to tailor the behaviour of the generic mutation operators to the fault model that is assumed for the system under test and the environment in which it is deployed. Empirical results obtained with a large data acquisition system in the satellite domain show that our approach can successfully automate the generation of test suites that achieve slightly better instruction coverage than manual testing based on domain expertise.

We present UMTG, a toolset for automatically generating executable and traceable system test cases from use case specifications. UMTG employs Natural Language Processing (NLP), a restricted form of use case specifications, and constraint solving. Use cases are expected to follow a template with restriction rules that reduce imprecision and enable NLP. NLP is used to capture the control flow implicitly described in use case specifications. Finally, to generate test input, constraint solving is applied to OCL constraints referring to the domain model of the system. UMTG is integrated with two tools that are widely adopted in industry, IBM Doors and Rhapsody. UMTG has been successfully evaluated on an industrial case study.

Automatic testing, although useful, is still quite ineffective against faults that do not cause crashes or uncaught exceptions. In the majority of the cases automatic tests do not include oracles, and only in some cases they incorporate assertions that encode the observed behavior instead of the intended behavior, that is if the application under test produces a wrong result, the synthesized assertions will encode wrong expectations that match the actual behavior of the application. In this paper we present ZoomIn, a technique that extends the fault-revealing capability of test case generation techniques from crash-only faults to faults that require non-trivial oracles to be detected. ZoomIn exploits the knowledge encoded in the manual tests written by developers and the similarity between executions to automatically determine an extremely small set of suspicious assertions that are likely wrong and thus worth manual inspection. Early empirical results show that ZoomIn has been able to detect 50% of the analyzed non-crashing faults in the Apache Commons Math library requiring the inspection of less than 1.5% of the assertions automatically generated by EvoSuite.

In safety critical domains, system test cases are often derived from functional requirements in natural language (NL) and traceability between requirements and their corresponding test cases is usually mandatory. The definition of test cases is therefore time-consuming and error prone, especially so given the quickly rising complexity of embedded systems in many critical domains. Though considerable research has been devoted to automatic generation of system test cases from NL requirements, most of the proposed approaches re- quire significant manual intervention or additional, complex behavioral modelling. This significantly hinders their appli- cability in practice. In this paper, we propose Use Case Modelling for System Tests Generation (UMTG), an approach that automatically generates executable system test cases from use case spec- ifications and a domain model, the latter including a class diagram and constraints. Our rationale and motivation are that, in many environments, including that of our industry partner in the reported case study, both use case specifica- tions and domain modelling are common and accepted prac- tice, whereas behavioural modelling is considered a difficult and expensive exercise if it is to be complete and precise. In order to extract behavioral information from use cases and enable test automation, UMTG employs Natural Language Processing (NLP), a restricted form of use case specifica- tions, and constraint solving.

We present UMTG, a toolset for automatically generating executable and traceable system test cases from use case specifications. UMTG employs Natural Language Processing (NLP), a restricted form of use case specifications, and constraint solving. Use cases are expected to follow a template with restriction rules that reduce imprecision and enable NLP. NLP is used to capture the control flow implicitly described in use case specifications. Finally, to generate test input, constraint solving is applied to OCL constraints referring to the domain model of the system. UMTG is integrated with two tools that are widely adopted in industry, IBM Doors and Rhapsody. UMTG has been successfully evaluated on an industrial case study.

Testing the correct behaviour of data processing systems in the presence of faulty data is extremely expensive. The data structures processed by these systems are often complex, with many data fields and multiple constraints among them. Software engineers, in charge of testing these systems, have to handcraft complex data files or databases, while ensuring compliance with the multiple constraints to prevent the generation of trivially invalid inputs. In addition, assessing test results often means analysing complex output and log data. Though many techniques have been proposed to automatically test systems based on models, little exists in the literature to support the testing of systems where the complexity is in the data consumed in input or produced in output, with complex constraints between them. In particular, such systems often need to be tested with the presence of faults in the input data, in order to assess the robustness and behaviour of the system in response to such faults. This paper presents an automated test technique that relies upon six generic mutation operators to automatically generate faulty data. The technique receives two inputs: field data and a data model, i.e. a UML class diagram annotated with stereotypes and OCL constraints. The annotated class diagram is used to tailor the behaviour of the generic mutation operators to the fault model that is assumed for the system under test and the environment in which it is deployed. Empirical results obtained with a large data acquisition system in the satellite domain show that our approach can successfully automate the generation of test suites that achieve slightly better instruction coverage than manual testing based on domain expertise.

This chapter presents RADAR, a dynamic analysis technique, which analyzes regression problems and automatically identifies failure causes in C and C++ software.

In this chapter we present Verification-Aided Regression Testing, a novel extension of regression testing that is significantly less sensitive to the completeness of the validation test suite due to the use of model checking. We extend the use of test case executions from conventional direct fault discovery to generation of behavioral properties specific to the new version by (i) automatically producing properties that are proved to hold for the base version of a program, (ii) automatically identifying and checking on the upgraded program only the properties that, according to the developers’ intention, must be preserved by the upgrade, and (iii) reporting the faults and the corresponding counterexamples that are not revealed by the regression tests. Our empirical study on both open-source and industrial software systems show that Verification-Aided Regression Testing produces properties that can be extremely beneficial in increasing the effectiveness of regression testing by timely and automatically detecting faults unnoticed by existing test suites.

In this paper we present Veri cation-Aided Regression Testing (VART), a novel extension of regression testing that uses model checking to increase the fault revealing capability of existing test suites. The key idea in VART is to extend the use of test case executions from the conventional direct fault discovery to the generation of behavioral properties speci fic to the upgrade, by (i) automatically producing properties that are proved to hold for the base version of a program, (ii) automatically identifying and checking on the upgraded program only the properties that, according to the developers'intention, must be preserved by the upgrade, and (iii) reporting the faults and the corresponding counter-examples that are not revealed by the regression tests. Our empirical study on both open source and industrial software systems shows that VART automatically produces properties that increase the e ffectiveness of testing by automatically detecting faults unnoticed by the existing regression test suites.

Automated debugging techniques aim to help developers locate and understand the cause of a failure, an extremely challenging yet fundamental task. Most state-of-the-art approaches suffer from two problems: they require a large number of passing and failing tests and report possible faulty code with no explanation. To mitigate these issues, we present MIMIC, a novel automated debugging technique that combines and extends our previous input generation and anomaly detection techniques. MIMIC (1) synthesizes multiple passing and failing executions similar to an observed failure and (2) uses these executions to detect anomalies in behavior that may explain the failure. We evaluated MIMIC on six failures of real world programs with promising results: for five of these failures, MIMIC identified their root causes while producing a limited number of false positives. Most importantly, the anomalies identified by MIMIC provided information that may help developers understand (and ultimately eliminate) such root causes.

Source Code Management (SCM) systems with extensive support to branching, parallel development, and merging, such as Git and Mercurial, are increasingly popular. Despite the popularity of advanced SCM systems, merging multiple branches is still extremely painful and time consuming because it requires manually resolving a number of conflicts generated while developing branches independently. This is due to the limitations of currently available conflict detection mechanisms: conflicts are identified late and are limited to textual conflicts, while the conflicts that are most expensive to fix are the ones that cause misbehaviors in the program without causing textual conflicts. This paper introduces a multi-branch server-side dynamic analysis that can automatically detect conflicts that produce misbehaviors as soon as they are introduced, regardless the syntactical changes that occur in the program. The results of the analysis can dramatically improve the rate of conflicts that are detected and resolved early in the process. As a consequence, the cost and the effort required to complete merge operations will drastically decrease and the capability to timely evolve software will significantly improve.

Integrating tools implemented as plug-ins is a complex and time-consuming activity that requires good programming skills and a deep understanding of the underlying plug-in architecture for successful completion. Even when tools are implemented as plug-ins for a same integrated development environment (IDE), users frequently prefer to manually perform the same inefficient operations every time they use the tools rather than implementing automated tool integration.In our vision, IDE users must be able to flexibly execute plug-ins and easily integrate their results by designing workflows that can be persisted, automatically executed, and reused in other workflows.This paper therefore presents MASH, a framework that extends IDEs with task-based plug-ins (TB-plug-ins) and workflows. A TB-plug-in is a plug-in that exposes its functionalities as executable tasks in a workflow. TB-plug-in workflows are processes that automatically execute multiple tools and integrate their results. IDE users can turn regular plug-ins into TB-plug-ins by writing simple scripts or using the GUI capturing feature that MASH offers.We validated our idea with two case studies examining the design of two data-driven analyses as tool integration. We discovered that workflows can be easily designed by knowing little about the IDE or plug-ins API, saving significant effort otherwise devoted to implementing additional plug-ins and glue code, and they produce analyses that can be quickly modified and reused. Copyright © 2012 John Wiley & Sons, Ltd. Copyright © 2012 John Wiley & Sons, Ltd.

Several debugging techniques can be used to automatically identify the code fragments or the runtime events likely responsible of a failure. These techniques are useful, but can help reducing the debugging effort only to a given extent. In fact, even when these techniques are successful, software developers still have to invest a lot of effort in understanding if and why something detected as suspicious is really wrong. In this paper we present the tool implementing the AVA technique. AVA, compared to other approaches dedicated to automatic debugging, in addition to automatically identifying the events likely responsible of a failure, generates an explanation about why these events have been considered suspicious. This explanation can be used by developers to quickly discard imprecise outputs and more effectively work on the relevant anomalies

Multiple tools can assist developers when debugging programs, but only a few solutions specifically target the common case of regression failures, to provide a more focused and effective support to debugging. In this paper we present RADAR, a tool that combines change identification and dynamic analysis to automatically explain regression problems with a list of suspicious differences in the behavior of the base and upgraded version of a program. The output produced by the tool is particularly beneficial to understand why an application failed. A demo video is available at http://www.youtube.com/watch?v= DMGUgALG-yE

Despite the recent advances in test generation, fully automatic software testing remains a dream: Ultimately, any generated test input depends on a test oracle that determines correctness, and, except for generic properties such as ``the program shall not crash'', such oracles require human input in one form or another. CrowdSourcing is a recently popular technique to automate computations that cannot be performed by machines, but only by humans. A problem is split into small chunks, that are then solved by a crowd of users on the Internet. In this paper we investigate whether it is possible to exploit CrowdSourcing to solve the oracle problem: We produce tasks asking users to evaluate CrowdOracles -- assertions that reflect the current behavior of the program. If the crowd determines that an assertion does not match the behavior described in the code documentation, then a bug has been found. Our experiments demonstrate that CrowdOracles are a viable solution to automate the oracle problem, yet taming the crowd to get useful results is a difficult task.

Regression testing techniques are commonly used to validate the correctness of upgrades. When a regression test fails, testers must understand the erroneous behaviors that caused the failure and identify the fault that originated these erroneous behaviors. In many cases, identifying the causes of a failure is difficult and time consuming. The analysis of regression problems provides interesting opportunities to validation and verification techniques. In fact, by comparing the execution of the base version and the upgraded version of the same program it is possible to automatically deduce information about incorrect behavior of the program. In this paper we present RADAR (Regression Analysis with Diff And Recording), a dynamic analysis technique, which analyzes regression problems and automatically identifies the chain of erroneous events that lead to a failure in C/C++ programs. RADAR exploits information about changes and the availability of multiple versions of the same program to automatically distinguish correct and suspicious events. Empirical experience with industrial and open source cases shows that RADAR can effectively support testers in the investigation of regression problems. Thus, RADAR can drive and simplify the debugging process. © 2012 IEEE.

Software systems evolve during development and maintenance, and many test cases designed for the early versions of the system become obsolete during the software lifecycle. Repairing test cases that do not compile due to changes in the code under test and generating new test cases to test the changed code is an expensive and time consuming activity that could benefit from automated approaches. In this paper we propose an approach for automatically repairing and generating test cases during software evolution. Differently from existing approaches to test case generation, our approach uses information available in existing test cases, defines a set of heuristics to repair test cases invalidated by changes in the software, and generate new test cases for evolved software. The results obtained with a prototype implementation of the technique show that the approach can effectively maintain evolving test suites, and perform well compared to competing approaches.

Most of the modern Integrated Development Environments are developed with plug-in based architectures that can be extended with additional functionalities and plug-ins, according to user needs. However, extending an IDE is still a possibility restricted to developers with deep knowledge about the specific development environment and its architecture. In this paper we present MASH, a tool that eases the programming of Integrated Development Environments. The tool supports the definition of workflows that can be quickly designed to integrate functionalities offered by multiple plugins, without the need of knowing anything about the internal architecture of the IDE. Workflows can be easily reshaped every time an analysis must be modified, without the need of producing Java code and deploying components in the IDE. Early results suggest that this approach can effectively facilitate programming of IDEs. © 2012 IEEE.

The majority of IDEs implement a concept of plug-in that nicely supports the integration of tools within the IDEs. Plug-ins dramatically simplify the structural integration of multiple tools, but provide little support to the design of the dynamic of the integration, which must be entirely coded by programmers from plug-ins' API. Manually integrating plug-ins is costly, complex and requires a deep understanding of the underlying environment. The implementation of tools as plug-ins and the integration of the results produced by different plug-ins are still difficult, expensive and error-prone activities. This paper presents the concepts of Task Based Plug-in (TB-plug-in) and workflow of TB-plug-ins. In our vision, IDE users must be able to execute plug-ins and integrate their results by designing workflows that can be persisted, executed and re-used in other workflows. We validated our idea by refactoring a set of Eclipse plug-ins for log-file analysis into TB-plug-ins, and designing several workflows that integrate plug-in tasks. We compared the effort necessary to implement these analyses from plug-ins with the effort necessary to design the workflows from TB-plug-ins. We discovered that workflows can be easily designed with little knowledge about the IDE and the plug-ins' API, save significant effort otherwise devoted to the implementation of additional plug-ins and glue-code, and produce analyses that can be quickly modified and reused. Copyright 2011 ACM.

Many software design and verification techniques assume the availability of some kind of behavioral models of the systems under analysis [2,4,10,11,28]. Unfortunately, manually specifying and maintaining behavioral models is expensive and error prone, and requires specific skills that are not always available in development teams. This reduces the applicability of model-based approaches in industrial projects. The problem of generating behavioral models can be solved with techniques that automatically generate behavioral models by mining program executions and thus reducing the effort required to generate models [3, 5, 18,20,24]

Many software components are provided with incomplete specifications and little access to the source code. Reusing such gray-box components can result in integration faults that can be difficult to diagnose and locate. In this paper, we present Behavior Capture and Test (BCT), a technique that uses dynamic analysis to automatically identify the causes of failures and locate the related faults. BCT augments dynamic analysis techniques with model-based monitoring. In this way, BCT identifies a structured set of interactions and data values that are likely related to failures (failure causes), and indicates the components and the operations that are likely responsible for failures (fault locations). BCT advances scientific knowledge in several ways. It combines classic dynamic analysis with incremental finite state generation techniques to produce dynamic models that capture complementary aspects of component interactions. It uses an effective technique to filter false positives to reduce the effort of the analysis of the produced data. It defines a strategy to extract information about likely causes of failures by automatically ranking and relating the detected anomalies so that developers can focus their attention on the faults. The effectiveness of BCT depends on the quality of the dynamic models extracted from the program. BCT is particularly effective when the test cases sample the execution space well. In this paper, we present a set of case studies that illustrate the adequacy of BCT to analyze both regression testing failures and rare field failures. The results show that BCT automatically filters out most of the false alarms and provides useful information to understand the causes of failures in 69 percent of the case studies. © 2006 IEEE.

Software faults have a relevant impact on today economy. Modern software systems are often composed by different third-party components, of which developers have only a partial knowledge because of the absence of source code or complete specifications. The lack of complete knowledge about the integrated components is the principal cause of faults, and makes fault diagnosis difficult as well. The increase of maintenance costs and services unavailability is one of the consequences of difficult faults diagnosis. Different automatic diagnosis techniques have been developed to support developers in fault diagnosis activities. Unfortunately the existing techniques present several limitations: Log analysis techniques often require log specifications which are often not available; Fault localization techniques require source code which is often not available too; Automated debugging techniques detect faults only if their effects are visible from the application state; Anomaly detection techniques force developers to manually inspect and interpret a huge quantity of suspicious behaviors thus resulting not effective in practice. This PhD Thesis presents a framework for the diagnosis of software functional faults that advances the state of the art by: (1) Identifying and correlating different kinds of software misbehaviors, thus effectively describing faults with heterogeneous causes; (2) Automatically filtering false positives, thus permitting developers to inspect only anomalous behaviors useful to debug the application; (3) Identifying the cause effect relationships between multiple misbehaviors, thus simplifying the diagnosis activity; (4) Automatically interpreting software misbehaviors, thus permitting developers to immediately diagnose the fault; (5) Evaluating the solution with multiple third-party and industrial case studies, thus demonstrating the effectiveness of the solution.

When software systems evolve, for example due to fault fixes, modification of functionalities or refactoring activities, test cases may become obsolete thus generating wrong results or even not being executable or compilable. Maintaining test cases is expensive and time consuming, and often test cases are discarded by software developers due to high maintenance costs. This paper presents TestCareAssistant, a technique that combines data-flow analysis with program diffing for automatically repairing test cases that become obsolete because of changes in method declarations (addition, removal, or type modification of parameters or return values). The paper illustrates the efficacy of TestCareAssistant by analyzing the impact of method declarations changes on the executability of test cases, and by presenting the preliminary results of applying TestCareAssistant to repair 22 test cases

Heterogeneity, mobility, complexity and new application domains raise new software reliability issues that cannot be met cost-effectively only with classic software engineering approaches. Self-healing systems can successfully address these problems, thus increasing software reliability while reducing maintenance costs. Self-healing systems must be able to automatically identify runtime failures, locate faults, and find a way to bring the system back to an acceptable behavior. This paper discusses the challenges underlying the construction of self-healing systems with particular focus on functional failures, and presents a set of techniques to build software systems that can automatically heal such failures. It introduces techniques to automatically derive assertions to effectively detect functional failures, locate the faults underlying the failures, and identify sequences of actions alternative to the failing sequence to bring the system back to an acceptable behavior.

Classic fault localization techniques can automatically provide information about the suspicious code blocks that are likely responsible for observed failures. This information is useful, but not sufficient to completely understand the causes of failing executions, which still require further (time-consuming) investigations to be exactly identified. A useful and comprehensive source of information is frequently given by the set of unexpected events that have been observed during failures. Sequences of unexpected events are usually simple to be interpret, and testers can guess the expected correct sequences of events from the faulty sequences. In this paper, we present a tool that automatically identifies anomalous events that likely caused failures, filters the possible false positives, and presents the resulting data by building views that show chains of cause-effect relations, i.e., views that show when anomalous events are caused by other anomalous events. The use of the technique to investigate a fault in the Tomcat application server is also presented in the paper. © 2009 IEEE.

Dynamic analysis techniques have been extensively adopted to discover causes of observed failures. In particular, anomaly detection techniques can infer behavioral models from observed legal executions and compare failing executions with the inferred models to automatically identify the likely anomalous events that caused observed failures. Unfortunately the output of these techniques is limited to a set of independent suspicious anomalous events that does not capture the structure and the rationale of the differences between the correct and the failing executions. Thus, testers spend a relevant amount of time and effort to investigate executions and interpret these differences, reducing effectiveness of anomaly detection techniques. In this paper, we present Automata Violations Analyzer (AVA), a technique to automatically produce candidate interpretations of detected failures from anomalies identified by anomaly detection techniques. Interpretations capture the rationale of the differences between legal and failing executions with user understandable patterns that simplify identification of failure causes. The empirical validation with synthetic cases and third-party systems shows that AVA produces useful interpretations

Log files are commonly inspected by system administrators and developers to detect suspicious behaviors and diagnose failure causes. Since size of log files grows fast, thus making manual analysis impractical, different automatic techniques have been proposed to analyze log files. Unfortunately, accuracy and effectiveness of these techniques are often limited by the unstructured nature of logged messages and the variety of data that can be logged. This paper presents a technique to automatically analyze log files and retrieve important information to identify failure causes. The technique automatically identifies dependencies between events and values in logs corresponding to legal executions, generates models of legal behaviors and compares log files collected during failing executions with the generated models to detect anomalous event sequences that are presented to users. Experimental results show the effectiveness of the technique in supporting developers and testers to identify failure causes. © 2008 IEEE.