Search

To mitigate the cost of manually producing and maintaining models capturing software specifications,specification miningtechniques can be exploited to automatically derive up-to-date models that faithfully represent the behavior of software systems. So far,specification mining solutions focused on extracting information about the functional behavior of the system, especially in the form ofmodels that represent the ordering of the operations. Well-known examples are finite state models capturing the usage protocol ofsoftware interfaces and temporal rules specifying relations among system events.Although the functional behavior of a software system is a primary aspect of concern, there are several other non-functionalcharacteristics that must be typically addressed jointly with the functional behavior of a software system. Efficiency is one of the most relevant characteristics. Indeed, an application that delivers the right functionalities with an inefficient implementation may fail to satisfy the expectations of its users.Interestingly, the timing behavioris strongly dependent on the functional behavior of a software system. For instance, the timing of an operation depends on the functional complexity and size of the computation that is performed. Consequently, models that combine the functional and timing behaviors, as well as their dependencies, are extremely important to precisely reason on the behavior of software systems.In this paper, we address the challenge of generating models that capture both the functional and timing behavior of a software system from execution traces. The result is the Timed k-Tail (TkT) specification mining technique, which can mine finite state models that capture such an interplay: the functional behavior is represented by the possible order of the events accepted by the transitions, while the timing behavior is represented through clocks and clock constraints of different nature associated with transitions.Our empirical evaluation with several libraries and applications shows that TkT can generate accurate models, capable of supporting the identification of timing anomalies due to overloaded environment and performance faults. Furthermore, our study shows that TkT outperforms state-of-the-art techniques in terms of scalability and accuracy of the mined models.

Field monitoring techniques can collect data about the behavior of software applications while running in the field, with real users and real data. Developers can exploit the information extracted from the field to timely improve, tune, and fix their systems, anticipating feedback by users. It is, however, challenging to extract a relevant amount of information from field executions without introducing significant overhead. This paper addresses this challenge by studying how to inexpensively trace data in-memory while postponing save operations to idle time, so that the operations requested by users are exposed to a negligible overhead only. In particular, this paper presents delayed saving, a technique that efficiently traces references to objects, opportunistically saving information only when the monitored application is not serving any user request. Storing references and postponing save operations may introduce inaccuracy in the collected data, that is, a lately saved object might be in a different state compared to the state of the object at the time it was traced. Our evaluation shows that the level of inaccuracy introduced by delayed saving is limited compared to its efficiency and low intrusiveness.

Automatic generators of GUI tests often fail to generate semantically relevant test cases, and thus miss important test scenarios. To address this issue, test adaptation techniques can be used to automatically generate semantically meaningful GUI tests from test cases of applications with similar functionalities. In this paper, we present ADAPTDROID, a technique that approaches the test adaptation problem as a search-problem, and uses evolutionary testing to adapt GUI tests (including oracles) across similar Android apps. In our evaluation with 32 popular Android apps, ADAPTDROID successfully adapted semantically relevant test cases in 11 out of 20 cross-app adaptation scenarios.

The level and quality of automation dramatically affects software testing activities, determines costs and effectiveness of the testing process, and largely impacts on the quality of the final product. While costs and benefits of automating many testing activities in industrial practice (including managing the quality process, executing large test suites, and managing regression test suites) are well understood and documented, the benefits and obstacles of automatically generating system test suites in industrial practice are not well reported yet, despite the recent progresses of automated test case generation tools. Proprietary tools for automatically generating test cases are becoming common practice in large software organizations, and commercial tools are becoming available for some application domains and testing levels. However, generating system test cases in small and medium-size software companies is still largely a manual, inefficient and ad-hoc activity. This paper reports our experience in introducing techniques for automatically generating system test suites in a medium-size company. We describe the technical and organizational obstacles that we faced when introducing automatic test case generation in the development process of the company, and present the solutions that we successfully experienced in that context. In particular, the paper discusses the problems of automating the generation of test cases by referring to a customized ERP application that the medium-size company developed for a third party multinational company, and presents A B T 2 . 0 , the test case generator that we developed by tailoring A B T , a research state-of-the-art GUI test generator, to their industrial environment. This paper presents the new features of A B T 2 . 0 , and discusses how these new features address the issues that we faced.

Systems of systems are highly dynamic software systems that require flexible monitoring solutions to be observed and controlled. Indeed, operators have to frequently adapt the set of collected indicators according to changing circumstances, to visualize the behavior of the monitored systems and timely take actions, if needed. Unfortunately, dashboard systems are still quite cumbersome to configure and adapt to a changing set of indicators that must be visualized. This paper reports our initial effort towards the definition of an automatic dashboard generation process that exploits metamodel layouts to create a full dashboard from a set of indicators selected by operators., Comment: A. Tundo, C. Castelnovo, M. Mobilio, O. Riganelli and L. Mariani. (2020). Declarative Dashboard Generation. In Proc. of the International Workshop on Governing Adaptive and Unplanned Systems of Systems Co-located with the 31st International Symposium on Software Reliability Engineering (ISSRE 2020)

Debugging cyber-physical system (CPS) models is a cumbersome and costly activity. CPS models combine continuous and discrete dynamics—a fault in a physical component manifests itself in a very different way than a fault in a state machine. Furthermore, faults can propagate both in time and space before they can be detected at the observable interface of the model. As a consequence, explaining the reason of an observed failure is challenging and often requires domain-specific knowledge. In this paper, we propose approach, a novel CPSDebug that combines testing, specification mining, and failure analysis, to automatically explain failures in Simulink/Stateflow models. In particular, we address the hybrid nature of CPS models by using different methods to infer properties from continuous and discrete state variables of the model. We evaluate CPSDebug on two case studies, involving two main scenarios and several classes of faults, demonstrating the potential value of our approach.

Mobile operating systems evolve quickly, frequently updating the APIs that app developers use to build their apps. Unfortunately, API updates do not always guarantee backward compatibility, causing apps to not longer work properly or even crash when running with an updated system. This paper presents FILO, a tool that assists Android developers in resolving backward compatibility issues introduced by API upgrades. FILO both suggests the method that needs to be modified in the app in order to adapt the app to an upgraded API, and reports key symptoms observed in the failed execution to facilitate the fixing activity. Results obtained with the analysis of 12 actual upgrade problems and the feedback produced by early tool adopters show that FILO can practically support Android developers.FILO can be downloaded from https://gitlab.com/learnERC/filo, and its video demonstration is available at https://youtu.be/WDvkKj-wnlQ., for associated video, see http://myhost.domain/file.mpg and FILO source code can be downloaded from https://gitlab.com/learnERC/filo . In the International Conference on Automated Software Engineering (2020)

This special issue is dedicated to the presentation of novel results in the scope of program analysis, verification, and testing of software to improve its quality. The papers included in the special issue present approaches that successfully combine model-based test case generation, reasoning about functional equivalence, data mining, classification, and the combination of abstraction with model-checking, to address real software applications in realistic settings.

Collecting traces from software running in the field is both useful and challenging. Traces may indeed help revealing unexpected usage scenarios, detecting and reproducing failures, and building behavioral models that reflect how the software is actually used. On the other hand, recording traces is an intrusive activity that may annoy users, negatively affecting the usability of the applications, if not properly designed. In this paper we address field monitoring by introducing Controlled Burst Recording, a monitoring solution that can collect comprehensive runtime data without compromising the quality of the user experience. The technique encodes the knowledge extracted from the monitored application as a finite state model that both represents the sequences of operations that can be executed by the users and the corresponding internal computations that might be activated by each operation. Our initial assessment with information extracted from ArgoUML shows that Controlled Burst Recording can reconstruct behavioral information more effectively than competing sampling techniques, with a low impact on the system response time., accepted at ICST2020 https://icst2020.info/

Debugging Cyber-Physical System models is often challenging, as it requires identifying a potentially long, complex and heterogenous combination of events that resulted in a violation of the expected behavior of the system. In this paper we present CPSDebug, a tool for supporting designers in the debugging of failures in MATLAB Simulink/Stateflow models. CPSDebug implements a gray-box approach that combines testing, specification mining, and failure analysis to identify the causes of failures and explain their propagation in time and space. The evaluation of the tool, based on multiple usage scenarios and faults and direct feedback from engineers, shows that CPSDebug can effectively aid engineers during debugging tasks.

Collecting data about the sequences of function calls executed by an application while running in the field can be useful to a number of applications, including failure reproduction, profiling, and debugging. Unfortunately, collecting data from the field may introduce annoying slowdowns that negatively affect the quality of the user experience. So far, the impact of monitoring has been mainly studied in terms of the overhead that it may introduce in the monitored applications, rather than considering if the introduced overhead can be really recognized by users. In this paper we take a different perspective studying to what extent collecting data about sequences of function calls may impact the quality of the user experience, producing recognizable effects. Interestingly we found that, depending on the nature of the executed operation and its execution context, users may tolerate a non-trivial overhead. This information can be potentially exploited to collect significant amount of data without annoying users.

Android apps must work correctly even if their execution is interrupted by external events. For instance, an app must work properly even if a phone call is received, or after its layout is redrawn because the smartphone has been rotated. Since these events may require destroying, when the execution is interrupted, and recreating, when the execution is resumed, the foreground activity of the app, the only way to prevent the loss of state information is saving and restoring it. This behavior must be explicitly implemented by app developers, who often miss to implement it properly, releasing apps affected by data loss problems, that is, apps that may lose state information when their execution is interrupted. Although several techniques can be used to automatically generate test cases for Android apps, the obtained test cases seldom include the interactions and the checks necessary to exercise and reveal data loss faults. To address this problem, this paper presents Data Loss Detector (DLD), a test case generation technique that integrates an exploration strategy, data-loss-revealing actions, and two customized oracle strategies for the detection of data loss failures. DLD has been able to reveal 75% of the faults in a benchmark of 54 Android app releases affected by 110 known data loss faults. DLD also revealed unknown data loss problems, outperforming competing approaches., Comment: for associated video presentation, see https://youtu.be/s6XZ7F8L3nY for associated slides, see https://www.slideshare.net/OlivieroRiganelli/oliviero-riganelli-data-loss-detector-automatically-revealing-data-loss-bugs-in-android-apps . In Proc. of the International Symposium on Software Testing and Analysis (ISSTA 2020)

Microservice-based applications consist of multiple services that can evolve independently. When services are modified, they are typically tested before being deployed. However, the test suites that are executed are usually designed without the exact knowledge about how the services will be accessed and used in the field, therefore they may easily miss relevant test scenarios, failing to prevent the deployment of faulty services. To address this problem, we introduce ExVivoMicroTest, an approach that analyzes the execution of deployed services at runtime in the field, in order to generate test cases for future versions of the same services. ExVivoMicroTest exploits cloud technologies, containers in particular, to generate a mocked environment that fully isolates the service under test from the rest of the system. It then reproduces service interactions as previously analyzed, thus testing the new version of the service against usage scenarios that capture the field usages of its earlier versions. We evaluate our approach on an open sourced microservices application and show that ExVivoMicroTest can effectively reveal faults based on automatically collected data.

Many applications are implemented as multi-tier software systems, and are executed on distributed infrastructures, like cloud infrastructures, to benefit from the cost reduction that derives from dynamically allocating resources on-demand. In these systems, failures are becoming the norm rather than the exception, and predicting their occurrence, as well as locating the responsible faults, are essential enablers of preventive and corrective actions that can mitigate the impact of failures, and significantly improve the dependability of the systems. Current failure prediction approaches suffer either from false positives or limited accuracy, and do not produce enough information to effectively locate the responsible faults. In this paper, we present PreMiSE, a lightweight and precise approach to predict failures and locate the corresponding faults in multi-tier distributed systems. PreMiSE blends anomaly-based and signature-based techniques to identify multi-tier failures that impact on performance indicators, with high precision and low false positive rate. The experimental results that we obtained on a Cloud-based IP Multimedia Subsystem indicate that PreMiSE can indeed predict and locate possible failure occurrences with high precision and low overhead., Accepted for publication in Journal of Systems and Software

International audience; This special issue is dedicated to the presentation of novel results in the scope of formal methods for the verification and testing of software. The papers included in the special issue showcase approaches that successfully combine a rigorous and formal background with the capability to address real software applications in realistic settings.

Android applications are executed on smartphones equipped with a variety of resources that must be properly accessed and controlled, otherwise the correctness of the executions and the stability of the entire environment might be negatively affected. For example, apps must properly acquire, use, and release microphones, cameras, and other multimedia devices otherwise the behavior of the apps that use the same resources might be compromised. Unfortunately, several apps do not use resources correctly, for instance due to faults and inaccurate design decisions. By interacting with these apps users may experience unexpected behaviors, which in turn may cause instability and sporadic failures, especially when resources are accessed. In this paper, we present an approach that lets users protect their environment from the apps that use resources improperly by enforcing the correct usage protocol. This is achieved by using software enforcers that can observe executions and change them when necessary. For instance, enforcers can detect that a resource has been acquired but not released, and automatically perform the release operation, thus giving the possibility to use that same resource to the other apps. The main idea is that software libraries, in particular the ones controlling access to resources, can be augmented with enforcers that can be activated and deactivated on demand by users to protect their environment from unwanted app behaviors. We call the software libraries augmented with one or more enforcers proactive libraries because the activation of the enforcer decorates the library with proactive behaviors that can guarantee the correctness of the execution despite the invocation of the operations implemented by the library., accepted for publication in ACM Transactions on Autonomous and Adaptive Systems (TAAS). Special Section on Best Papers from SEAMS 2017 (The conference paper is arXiv:1703.08005)

Dealing with the evolution of operating systems is challenging for developers of mobile apps, who have to deal with frequent upgrades that often include backward incompatible changes of the underlying API framework. As a consequence of framework upgrades, apps may show misbehaviours and unexpected crashes once executed within an evolved environment. Identifying the portion of the app that must be modified to correctly execute on a newly released operating system can be challenging. Although incompatibilities are visibile at the level of the interactions between the app and its execution environment, the actual methods to be changed are often located in classes that do not directly interact with any external element. To facilitate debugging activities for problems introduced by backward incompatible upgrades of the operating system, this paper presents FILO, a technique that can recommend the method that must be changed to implement the fix from the analysis of a single failing execution. FILO can also select key symptomatic anomalous events that can help the developer understanding the reason of the failure and facilitate the implementation of the fix. Our evaluation with multiple known compatibility problems introduced by Android upgrades shows that FILO can effectively and efficiently identify the faulty methods in the apps., accepted for inclusion in Proceedings of the International Symposium on Software Reliability Engineering (ISSRE) 2019

Cloud systems are complex, large, and dynamic systems whose behavior must be continuously analyzed to timely detect misbehaviors and failures. Although there are solutions to flexibly monitor cloud systems, cost-effectively controlling the anomaly detection logic is still a challenge. In particular, cloud operators may need to quickly change the types of detected anomalies and the scope of anomaly detection, for instance based on observations. This kind of intervention still consists of a largely manual and inefficient ad-hoc effort. In this paper, we present Anomaly Detection as-a-Service (ADaaS), which uses the same as-a-service paradigm often exploited in cloud systems to declarative control the anomaly detection logic. Operators can use ADaaS to specify the set of indicators that must be analyzed and the types of anomalies that must be detected, without having to address any operational aspect. Early results with lightweight detectors show that the presented approach is a promising solution to deliver better control of the anomaly detection logic., Paper accepted at the Intl. Workshop on Governing Adaptive and Unplanned Systems of Systems (GAUSS)

The ecosystem in which mobile applications run is highly heterogeneous and configurable. All layers upon which mobile apps are built offer wide possibilities of variations, from the device and the hardware, to the operating system and middleware, up to the user preferences and settings. Testing all possible configurations exhaustively, before releasing the app, is unaffordable. As a consequence, the app may exhibit different, including faulty, behaviours when executed in the field, under specific configurations. In this paper, we describe a framework that can be instantiated to support in-vivo testing of a mobile app. The framework monitors the configuration in the field and triggers in-vivo testing when an untested configuration is recognized. Experimental results show that the overhead introduced by monitoring is unnoticeable to negligible (i.e., 0-6%) depending on the device being used (high- vs. low-end). In-vivo test execution required on average 3s: if performed upon screen lock activation, it introduces just a slight delay before locking the device., Research paper accepted to ICST'20, 10+1 pages

Cloud systems are large scalable distributed systems that must be carefully monitored to timely detect problems and anomalies. While a number of cloud monitoring frameworks are available, only a few solutions address the problem of adaptively and dynamically selecting the indicators that must be collected, based on the actual needs of the operator. Unfortunately, these solutions are either limited to infrastructure-level indicators or technology-specific, for instance, they are designed to work with OpenStack but not with other cloud platforms. This paper presents the VARYS monitoring framework, a technology-agnostic Monitoring-as-a-Service solution that can address KPI monitoring at all levels of the Cloud stack, including the application-level. Operators use VARYS to indicate their monitoring goals declaratively, letting the framework to perform all the operations necessary to achieve a requested monitoring configuration automatically. Interestingly, the VARYS architecture is general and extendable, and can thus be used to support increasingly more platforms and probing technologies.

Debugging Cyber-Physical System (CPS) models can be extremely complex. Indeed, only detection of a failure is insufficient to know how to correct a faulty model. Faults can propagate in time and in space producing observable misbehaviours in locations completely different from the location of the fault. Understanding the reason of an observed failure is typically a challenging and laborious task left to the experience and domain knowledge of the designers. In this paper, we propose CPSDebug, a novel approach that combines testing, specification mining, and failure analysis, to automatically explain failures in Simulink/Stateflow models. We evaluate CPSDebug on two case studies, involving two use scenarios and several classes of faults, demonstrating the potential value of our approach.

The operational environment can be a valuable source of information about the behavior of software applications and their usage context. Although a single instance of an application has limited evidence of the range of the possible behaviors and situations that might be experienced in the field, the collective knowledge composed by the evidence gathered by the many instances of a same application running in several diverse user environments (eg, a browser) might be an invaluable source of information. This information can be exploited by applications able to autonomously analyze how they behave in the field and adjust their behavior accordingly. Augmenting applications with the capability to collaborate and directly share information about their behavior is challenging because it requires the definition of a fully decentralized and dependable networked infrastructure whose nodes are the user machines. The nodes of the infrastructure must be collaborative, to share information, and autonomous, to exploit the available information to change their behavior, for instance, to better accommodate the needs of the users to prevent known problems. This paper describes the initial results that we obtained with the design and the development of an infrastructure that can enable the execution of collaborative scenarios in a fully decentralized way. Our idea is to combine the agent-based paradigm, which is well suited to design collaborative and autonomous nodes, and the peer-to-peer paradigm, which is well suited to design distributed and dynamic network infrastructures. To demonstrate our idea, we augmented the popular JADE agent-based platform with a software layer that supports both the creation of a fully decentralized peer-to-peer network of JADE platforms and the execution of services within that network, thus enabling JADE multiagent systems (MASs) to behave as peer-to-peer networks. The resulting platform can be used to study the design of collaborative applications running in the field.

Summary Android apps share resources, such as sensors, cameras, and Global Positioning System, that are subject to specific usage policies whose correct implementation is left to programmers. Failing to satisfy these policies may cause resource leaks, that is, apps may acquire but never release resources. This might have different kinds of consequences, such as apps that are unable to use resources or resources that are unnecessarily active wasting battery. Researchers have proposed several techniques to detect and fix resource leaks. However, the unavailability of public benchmarks of faulty apps makes comparison between techniques difficult, if not impossible, and forces researchers to build their own data set to verify the effectiveness of their techniques (thus, making their work burdensome). The aim of our work is to define a public benchmark of Android apps affected by resource leaks. The resulting benchmark, called AppLeak, is publicly available on GitLab and includes faulty apps, versions with bug fixes (when available), test cases to automatically reproduce the leaks, and additional information that may help researchers in their tasks. Overall, the benchmark includes a body of 40 faults that can be exploited to evaluate and compare both static and dynamic analysis techniques for resource leak detection.

Cloud systems are complex and large systems where services provided by different operators must coexist and eventually cooperate. In such a complex environment, controlling the health of both the whole environment and the individual services is extremely important to timely and effectively react to misbehaviours, unexpected events, and failures. Although there are solutions to monitor cloud systems at different granularity levels, how to relate the many KPIs that can be collected about the health of the system and how health information can be properly reported to operators are open questions. This paper reports the early results we achieved in the challenge of monitoring the health of cloud systems. In particular we present CloudHealth, a model-based health monitoring approach that can be used by operators to watch specific quality attributes. The CloudHealth Monitoring Model describes how to operationalize high level monitoring goals by dividing them into subgoals, deriving metrics for the subgoals, and using probes to collect the metrics. We use the CloudHealth Monitoring Model to control the probes that must be deployed on the target system, the KPIs that are dynamically collected, and the visualization of the data in dashboards., 8 pages, 2 figures, 1 table

By leveraging large clusters of commodity hardware, the Cloud offers great opportunities to optimize the operative costs of software systems, but impacts significantly on the reliability of software applications. The lack of control of applications over Cloud execution environments largely limits the applicability of state-of-the-art approaches that address reliability issues by relying on heavyweight training with injected faults. In this paper, we propose \emph(LOUD}, a lightweight fault localization approach that relies on positive training only, and can thus operate within the constraints of Cloud systems. \emph{LOUD} relies on machine learning and graph theory. It trains machine learning models with correct executions only, and compensates the inaccuracy that derives from training with positive samples, by elaborating the outcome of machine learning techniques with graph theory algorithms. The experimental results reported in this paper confirm that \emph{LOUD} can localize faults with high precision, by relying only on a lightweight positive training., Comment: 12 pages, 8 figures, paper accepted at ICST 2018

Runtime enforcement can be effectively used to improve the reliability of software applications. However, it often requires the definition of ad hoc policies and enforcement strategies, which might be expensive to identify and implement. This paper discusses how to exploit lifecycle events to obtain useful enforcement strategies that can be easily reused across applications, thus reducing the cost of adoption of the runtime enforcement technology. The paper finally sketches how this idea can be used to define libraries that can automatically overcome problems related to applications misusing them., Comment: International Symposium On Leveraging Applications of Formal Methods, Verification and Validation (ISoLA'18) [Invited Talk Paper]

5G standard emerges at a particular time in technology history when cloud transforms deeply almost all industries and services: it becomes obvious that innovations have to be made cloud-native for being successful. 5G must become the ubiquitous fabric blending universal connectivity (to humans, robots, sensors…) with cloud versatility and scalability. For realizing this vision, another model than IaaS must be adopted, the Platform as a Service (PaaS), which should be built to support telco-grade requirements and combine all sort of third-party applications. These are the core objectives of the Next Generation Platform as a Service (NGPaaS) project, a H2020 5G PPP Phase 2 project. The paper presents the project fundamentals, its architectural proposal and most relevant features.

Testing software applications by interacting with their graphical user interface (GUI) is an expensive and complex process. Current automatic test case generation techniques implement explorative approaches that, although producing useful test cases, have a limited capability of covering semantically relevant interactions, thus frequently missing important testing scenarios. These techniques typically interact with the available widgets following the structure of the GUI, without any guess about the functions that are executed. In this paper we propose Augusto, a test case generation technique that exploits a built-in knowledge of the semantics associated with popular and well-known functionalities, such as CRUD operations, to automatically generate effective test cases with automated functional oracles. Empirical results indicate that Augusto can reveal faults that cannot be revealed with state of the art techniques.

Recently, response-adaptive designs have been proposed in randomized clinical trials to achieve ethical and/or cost advantages by using sequential accrual information collected during the trial to dynamically update the probabilities of treatment assignments. In this context, urn models-where the probability to assign patients to treatments is interpreted as the proportion of balls of different colors available in a virtual urn-have been used as response-adaptive randomization rules. We propose the use of Randomly Reinforced Urn (RRU) models in a simulation study based on a published randomized clinical trial on the efficacy of home enteral nutrition in cancer patients after major gastrointestinal surgery. We compare results with the RRU design with those previously published with the non-adaptive approach. We also provide a code written with the R software to implement the RRU design in practice. In detail, we simulate 10,000 trials based on the RRU model in three set-ups of different total sample sizes. We report information on the number of patients allocated to the inferior treatment and on the empirical power of the t-test for the treatment coefficient in the ANOVA model. We carry out a sensitivity analysis to assess the effect of different urn compositions. For each sample size, in approximately 75% of the simulation runs, the number of patients allocated to the inferior treatment by the RRU design is lower, as compared to the non-adaptive design. The empirical power of the t-test for the treatment effect is similar in the two designs.

International audience; This chapter describes how to use in-the-field runtime techniques to improve the dependability of software systems. In particular, we first present an overall vision of the problem of ensuring highly-dependable behaviours at runtime based on the concept of autonomic monitor, and then we present the two families of relevant approaches for this purpose. First, we present techniques related to runtime enforcement that can prevent the system producing bad behaviours. Second, we describe healing techniques that can detect if the system has produced a bad behaviour and react to the situation accordingly (e.g., moving the system back to a correct state).

Platform-As-A-Service (PaaS) systems offer customers a rich environment in which to build, deploy, and run applications. Today’s PaaS offerings are tailored mainly to the needs of web and mobile applications developers, and involve a fairly rigid stack of components and features. The vision of the H2020 5GPPP Phase 2 Next Generation Platform-as-a-Service (NGPaaS) project is to enable “build-to-order” customized PaaSes, tailored to the needs of a wide range of use cases with telcograde5G characteristics. This paper sets out the salient and innovative features of NGPaaS and explores the impacts on Operational Support Systems and Business Support Systems (OSS/BSS), moving from fixed centralized stacks to a much more flexible and modular distributed architecture.

Policy enforcers are sophisticated runtime components that can prevent failures by enforcing the correct behavior of the software. While a single enforcer can be easily designed focusing only on the behavior of the application that must be monitored, the effect of multiple enforcers that enforce different policies might be hard to predict. So far, mechanisms to resolve interferences between enforcers have been based on priority mechanisms and heuristics. Although these methods provide a mechanism to take decisions when multiple enforcers try to affect the execution at a same time, they do not guarantee the lack of interference on the global behavior of the system. In this paper we present a verification strategy that can be exploited to discover interferences between sets of enforcers and thus safely identify a-priori the enforcers that can co-exist at run-time. In our evaluation, we experimented our verification method with several policy enforcers for Android and discovered some incompatibilities., Comment: Oliviero Riganelli, Daniela Micucci, Leonardo Mariani, and Yli\`es Falcone. Verifying Policy Enforcers. Proceedings of 17th International Conference on Runtime Verification (RV), 2017. (to appear)

This paper presents the result of our experience with the ap- plication of runtime verification, testing and static analysis techniques to several industrial projects. We discuss the eight most relevant challenges that we experienced, and the strategies that we elaborated to face them.

Source Code Management (SCM) systems support software evolution by providing features, such as version control, branching, and conflict detection. Despite the presence of these features, support to parallel software development is often limited. SCM systems can only address a subset of the conflicts that might be introduced by developers when concurrently working on multiple parallel branches. In fact, SCM systems can detect textual conflicts, which are generated by the concurrent modification of the same program locations, but they are unable to detect higher-order conflicts, which are generated by the concurrent modification of different program locations that generate program misbehaviors once merged. Higher-order conflicts are painful to detect and expensive to fix because they might be originated by the interference of apparently unrelated changes. In this paper we present Behavioral Driven Conflict Identification (BDCI), a novel approach to conflict detection. BDCI moves the analysis of conflicts from the source code level to the level of program behavior by generating and comparing behavioral models. The analysis based on behavioral models can reveal interfering changes as soon as they are introduced in the SCM system, even if they do not introduce any textual conflict. To evaluate the effectiveness and the cost of the proposed approach, we developed BDCIf, a specific instance of BDCI dedicated to the detection of higher-order conflicts related to the functional behavior of a program. The evidence collected by analyzing multiple versions of Git and Redis suggests that BDCIf can effectively detect higher-order conflicts and report how changes might interfere.

Despite their growing complexity and increasing size, modern software applications must satisfy strict release requirements that impose short bug fixing and maintenance cycles, putting significant pressure on developers who are responsible for timely producing high-quality software. To reduce developers workload, repairing and healing techniques have been extensively investigated as solutions for efficiently repairing and maintaining software in the last few years. In particular, repairing solutions have been able to automatically produce useful fixes for several classes of bugs that might be present in software programs. A range of algorithms, techniques, and heuristics have been integrated, experimented, and studied, producing a heterogeneous and articulated research framework where automatic repair techniques are proliferating. This paper organizes the knowledge in the area by surveying a body of 108 papers about automatic software repair techniques, illustrating the algorithms and the approaches, comparing them on representative examples, and discussing the open challenges and the empirical evidence reported so far.

Software libraries implement APIs that deliver reusable functionalities. To correctly use these functionalities, software applications must satisfy certain correctness policies, for instance policies about the order some API methods can be invoked and about the values that can be used for the parameters. If these policies are violated, applications may produce misbehaviors and failures at runtime. Although this problem is general, applications that incorrectly use API methods are more frequent in certain contexts. For instance, Android provides a rich and rapidly evolving set of APIs that might be used incorrectly by app developers who often implement and publish faulty apps in the marketplaces. To mitigate this problem, we introduce the novel notion of proactive library, which augments classic libraries with the capability of proactively detecting and healing misuses at run- time. Proactive libraries blend libraries with multiple proactive modules that collect data, check the correctness policies of the libraries, and heal executions as soon as the violation of a correctness policy is detected. The proactive modules can be activated or deactivated at runtime by the users and can be implemented without requiring any change to the original library and any knowledge about the applications that may use the library. We evaluated proactive libraries in the context of the Android ecosystem. Results show that proactive libraries can automati- cally overcome several problems related to bad resource usage at the cost of a small overhead., Comment: O. Riganelli, D. Micucci and L. Mariani, "Policy Enforcement with Proactive Libraries" 2017 IEEE/ACM 12th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS), Buenos Aires, Argentina, 2017, pp. 182-192

Field failures, that is, failures caused by faults that escape the testing phase leading to failures in the field, are unavoidable. Improving verification and validation activities before deployment can identify and timely remove many but not all faults, and users may still experience a number of annoying problems while using their software systems. This paper investigates the nature of field failures, to understand to what extent further improving in-house verification and validation activities can reduce the number of failures in the field, and frames the need of new approaches that operate in the field. We report the results of the analysis of the bug reports of five applications belonging to three different ecosystems, propose a taxonomy of field failures, and discuss the reasons why failures belonging to the identified classes cannot be detected at design time but shall be addressed at runtime. We observe that many faults (70%) are intrinsically hard to detect at design-time.

Field data is an invaluable source of information for testers and developers because it witnesses how software systems operate in real environments, capturing scenarios and configurations relevant to end-users. Unfortunately, collecting traces might be resource-consuming and can significantly affect the user experience, for instance causing annoying slowdowns. Existing monitoring techniques can control the overhead introduced in the applications by reducing the amount of collected data, for instance by collecting each event only with a given probability. However, collecting fewer events limits the amount of information extracted from the field and may fail in providing a comprehensive picture of the behavior of a program. In this paper we present fragmented monitoring, a monitoring technique that addresses the issue of collecting information from the field without annoying users. The key idea of fragmented monitoring is to reduce the overhead by recording partial traces (fragments) instead of full traces, while annotating the beginning and the end of each fragment with state information. These annotations are exploited offline to derive traces that might be likely observed in the field and that could not be collected directly due to the overhead that would be introduced in a program., In Proceedings PrePost 2017, arXiv:1708.06889

In this paper we present VART, a tool for automatically revealing regression faults missed by regression test suites. Interestingly, VART is not limited to faults causing crashing or exceptions, but can reveal faults that cause the violation of application-specific correctness properties. VART achieves this goal by combining static and dynamic program analysis.

Inferring models of program behavior from execution samples can provide useful information about a system, also in the increasingly common case of systems that evolve and adapt in their lifetime, and without requiring large developers' effort. Techniques for learning models of program behavior from execution traces shall address conflicting challenges of recall, specificity and performance: They shall generate models that comprehensively represent the system behavior (recall) while limiting the amount of illegal behaviors that may be erroneously accepted by the model (specificity), and should infer the models within a reasonable time budget to process industrial scale systems (performance). In our early work, we designed GK-Tail, an approach that can infer guarded finite state machines that model the behavior of object-oriented programs in terms of sequences of method calls and constraints on the parameter values. GK-Tail addresses well two of the three main challenges, since it infers guarded finite state machines with a high level of recall and specificity, but presents severe limitations in terms of performance that reduce its scalability. In this paper, we present GK-Tail+, a new approach to infer guarded finite state machines from execution traces of object-oriented programs. GK-Tail+ proposes a new set of inference criteria that represent the core element of the inference process: It largely reduces the inference time of GK-Tail while producing guarded finite state machines with a comparable level of recall and specificity. Thus, GK-Tail+ advances the preliminary results of GK-Tail by addressing all the three main challenges of learning models of program behavior from execution traces.

Many software components are provided with incomplete specifications and little access to the source code. Reusing such gray-box components can result in integration faults that can be difficult to diagnose and locate. In this paper, we present Behavior Capture and Test (BCT), a technique that uses dynamic analysis to automatically identify the causes of failures and locate the related faults. BCT augments dynamic analysis techniques with model-based monitoring. In this way, BCT identifies a structured set of interactions and data values that are likely related to failures (failure causes), and indicates the components and the operations that are likely responsible for failures (fault locations). BCT advances scientific knowledge in several ways. It combines classic dynamic analysis with incremental finite state generation techniques to produce dynamic models that capture complementary aspects of component interactions. It uses an effective technique to filter false positives to reduce the effort of the analysis of the produced data. It defines a strategy to extract information about likely causes of failures by automatically ranking and relating the detected anomalies so that developers can focus their attention on the faults. The effectiveness of BCT depends on the quality of the dynamic models extracted from the program. BCT is particularly effective when the test cases sample the execution space well. In this paper, we present a set of case studies that illustrate the adequacy of BCT to analyze both regression testing failures and rare field failures. The results show that BCT automatically filters out most of the false alarms and provides useful information to understand the causes of failures in 69 percent of the case studies. © 2006 IEEE.

Several techniques and tools have been proposed for the automatic generation of test cases. Usually, these tools are evaluated in terms of fault-revealing or coverage capability, but their impact on the manual debugging activity is not considered. The question is whether automatically generated test cases are equally effective in supporting debugging as manually written tests. We conducted a family of three experiments (five replications) with humans (in total, 55 subjects) to assess whether the features of automatically generated test cases, which make them less readable and understandable (e.g., unclear test scenarios, meaningless identifiers), have an impact on the effectiveness and efficiency of debugging. The first two experiments compare different test case generation tools (Randoop vs. EvoSuite). The third experiment investigates the role of code identifiers in test cases (obfuscated vs. original identifiers), since a major difference between manual and automatically generated test cases is that the latter contain meaningless (obfuscated) identifiers. We show that automatically generated test cases are as useful for debugging as manual test cases. Furthermore, we find that, for less experienced developers, automatic tests are more useful on average due to their lower static and dynamic complexity.

In this chapter we present G-RankTest, a technique for the automatic generation, ranking, and execution of regression test cases for controller applications.

In this chapter we present Verification-Aided Regression Testing, a novel extension of regression testing that is significantly less sensitive to the completeness of the validation test suite due to the use of model checking. We extend the use of test case executions from conventional direct fault discovery to generation of behavioral properties specific to the new version by (i) automatically producing properties that are proved to hold for the base version of a program, (ii) automatically identifying and checking on the upgraded program only the properties that, according to the developers’ intention, must be preserved by the upgrade, and (iii) reporting the faults and the corresponding counterexamples that are not revealed by the regression tests. Our empirical study on both open-source and industrial software systems show that Verification-Aided Regression Testing produces properties that can be extremely beneficial in increasing the effectiveness of regression testing by timely and automatically detecting faults unnoticed by existing test suites.

Research in black-box testing has produced impressive results in the past 40 years, addressing many aspects of the problem that span from integration with the development process, to test case generation and execution. In the past few years, the research in this area has focused mostly on the automation of black-box approaches to improve applicability and scalability. This chapter surveys the recent advances in automatic black-box testing, covering contributions from 2010 to 2014, presenting the main research results and discussing the research trends. © 2015 Elsevier Inc.

The high cost of changes is a primary obstacle to more frequent software upgrades. We suggest addressing this problem by introducing an automated framework and methodology, and a mix of technologies to identify the impact of changes and upgrades (due to error fixing and functionality enhancement) and from component replacement within a single product and a product family. This methodology will improve the reliability of networked software and reduce validation costs by implementing an innovative solution for the automatic detection and localization of program bugs. We note that we do not address the initial analysis and validation, but aim to reduce the costs of testing changes in existing systems by several orders of magnitude by introducing automatic techniques to analyze and test software increments. The goal is to make analyzing and testing networked software after each change practical, which allows developers and testers to diagnose and remove faults as soon as they occur, thus saving testing and development effort while increasing software reliability. The new analysis and testing techniques that we suggest are created by improving state-of-the-art techniques that apply both static and dynamic analysis. Stand-alone solutions for verification of system changes do not scale for complex environments, and any new industrially applicable solution must integrate several technologies into a common framework. Moreover, as we discuss in more detail later, some program constructs pose a challenge for one technology while being easy or at least feasible to solve using another technology or a combination of technologies. We suggest a combination of techniques that creates an integrated paradigm for the analysis of software upgrades. The key approach in our methodology is to automatically verify that safety properties that hold for a given version continue to hold after a change without repeating the entire analysis. This is achieved by means of analysis procedures that focus on the parts of the system that have changed during evolution of a design or are affected by this evolution, and determine whether all behaviors of the original system are preserved in the new version of the design. The analysis is performed by the integration of technologies that apply different types of analysis (static and dynamic) and work at different abstraction levels. These technologies address different classes of problems and share a common solution framework. Clearly, every technology has its limitations—some of the limitations are inherent to the technology itself or to a particular component of the technology; others are challenges that can be dealt with by combining the technology with others. In what follows, we discuss the challenges in more detail.

Component-based software systems raise new problems for the testing community: the reuse of components suggests the possibility of reducing testing costs by reusing information about the quality of the software components. This paper addresses the problem of testing evolving software systems, i.e., systems obtained by modifying and/or substituting some of their components. The paper proposes a technique to automatically identify behavioral differences between different versions of the system, to deduce possible problems from inconsistent behaviors. The approach is based on the automatic distilling of invariants from in-field executions. The computed invariants are used to monitor the behavior of new components, and to reveal unexpected interactions. The event generated while monitoring system executions are presented to software engineers who can infer possible problems of the new versions.

Tool integration is a very difficult challenge. Problems may arise at different abstraction levels and from several sources such as heterogeneity of manipulated data, incompatible interfaces, or uncoordinated services, to name just a few examples. On the other hand, applications based on the coherent composition of activities, components, services, and data from heterogeneous sources are increasingly present in our everyday lives. Consequently, tool integration takes on increasing significance. In this paper we analyze the tool-integration problem at different abstraction levels and discuss different views on a layered software architecture that we have designed specifically for a middleware that supports the execution of distributed applications for the orchestration of human/system activities. We noticed that the agent paradigm provided a suitable technology for abstraction in tool integration. Throughout the paper, the discussion refers to a case study in the bioinformatics domain.

Verification of component-based systems presents new challenges not yet completely addressed by existing testing techniques. This paper proposes a new approach for automatically testing highly reconfigurable component-based systems, i.e., systems that can be obtained by changing some components. The paper presents an industrial case that motivates our research and proposes a testing infrastructure that tracks run-time information for components. The collected information is used for automatic testing new versions of existing components and new configurations of existing systems. © 2003 Published by Elsevier Science B.V.