Search

Virtualization is gaining attraction in the industry as it promises a flexible way to integrate, manage, and re-use heterogeneous software components with mixed-criticality levels, on a shared hardware platform, while obtaining isolation guarantees. This work surveys the state-of-the-practice of real-time virtualization technologies by discussing common issues in the industry. In particular, we analyze how different virtualization approaches and solutions can impact isolation guarantees and testing/certification activities, and how they deal with dependability challenges. The aim is to highlight current industry trends and support industrial practitioners to choose the most suitable solution according to their application domains., Accepted for publication in Elsevier Future Generation Computer Systems

Error propagation analysis is a consolidated practice to gain insights into error modes and effects that pertain to the activation of faults in software systems. A variety of approaches, such as architecture-based, source code instrumentation and variable tracing, have been proposed so far to address software error propagation analysis. Although valuable, existing approaches entail a substantial degree of system internals’ knowledge, visibility and code manipulation that is not well-suited for real-life production environments. This paper proposes an empirical analysis of error propagation. We specifically address the challenges in using fault data and error events in the logs, which are a convenient byproduct of the system’s execution. The approach puts forth the construction of error reporting graphs. We apply the approach to 2,042 failure data points from two real-world critical systems from the Air Traffic Control domain by a top industry provider. The approach contributes to develop a deep understanding on error modes and propagation paths, which can be leveraged by practitioners to make informed decisions on the placement of error detection mechanisms.

In modern smartphones, system performances are tightly related to a variety of underlying subsystems. In particular, internal storage, along the years, has become crucial because it is extensively used to access content relevant to the system and, finally, to the end user. To understand its role in a commercial Android smartphone and to evaluate its effects on the User experience, within the context of a real usage, we analyzed Linux page fault handling, a critical mechanism that puts pressure on storage devices and may cause system inefficiencies. A kernel tracing technique has been conceived for real-time measurement of Android applications and services on commercial smartphones. The experimental results presented in this work are derived from the use of this kernel tracing on a 64-bit Android smartphone, equipped with a ufs storage subsystem. The main subject of the study is major page fault handling, a kernel mechanism behind many end-user actions recognized at industry level as source of possible performance deterioration in a smartphone. The analysis shows that major page fault handling is dominated by read accesses to ufs (between 30% and 40% of the total time), and that the related storage traffic is significantly affected by the ReadAhead mechanism, which is not always efficient.

Cloud computing is a key supporting technology driving the fourth industrial revolution, spanning from the Internet of Things (IoT) and Cloud for Telecoms (C4T) to Industry 4.0 to Smart Cities, and so on. This pervasiveness of cloud technology is due to the ability to easily share and obtain resources on a pay-per-use and elastic provisioning model. Several enterprises use the cloud as a cheap solution to the achieve computational and storage capabilities they need without incurring the costs associated with owning and maintaining data centers. This implies that they rely on the correctness of the services provided by the cloud platforms and that any possible outage could result in considerable loss of reputation and money, which could effectively bring actions against the cloud service provider for violations to the agreed upon Service Level Agreement (SLA).

The industrial Internet of Things (IIoT) is currently foreseen as a foundation to implement the Industry 4.0 vision. However, device heterogeneity and the need of integration and configuration exposes the industrial infrastructure to potential threats, such as black-hole, man-in-the-middle, and malicious configuration attacks. In this article, we investigate how to manage distributed trust information and to enable trusted configuration actions in the IIoT, by opportunistically intermingling blockchain with the software defined networking and container orchestration technologies. In particular, we focus on how the joint and coordinated adoption of such technologies can make technicians’ interventions on industrial equipment both easier and more trusted. To this purpose, we present the design of a software architecture to simplify the management, configuration, and assessment of IIoT systems, and we discuss our experiences with the application of the proposed architecture in a railways use case.

Computer applications, such as servers, databases and middleware, ubiquitously emit execution traces stored in log files. The use of logs for the analysis of application failures is known since the early days of computers. Field data studies have shown that application logs are fraught with uncertainty, i.e., missing or noisy events in the logs. A body of research that has dealt successfully with uncertainty in event logs is process mining from the business process management community, specifically by discovering process models. The literature has shown the value of process mining across several domains, but as yet there is no study that quantifies possible improvements from using process models, and the impact of uncertainty in the context of application failures. This work addresses the use of process mining for detecting failures from application logs. First, process models are discovered from logs; then conformance checking is used to detect deviations from the models. We contribute to knowledge engineering research with a systematic measurement study that quantifies the failure detection capability of conformance checking in spite of missing events, and its accuracy with respect to process models obtained from noisy logs. Analysis is done with a dataset of 55,462 execution traces from three independent real-life applications. We obtain a mixed answer depending on the application under test; our measurements provide insights into the use of process mining for failure analysis.

Critical computer systems strongly rely on event logs to record the occurrence of normative and anomalous events occurring at runtime. In spite of the advances in Security Information and Event Management for handling monitoring data in production, event logs remain quite underutilized with respect to more conventional security data sources. Eliciting actionable knowledge for situational awareness poses many challenges in the case of logs emitted by industrial systems due to the lack of standard practices, formats and threat models. This paper addresses log analysis in a critical industrial system. We conduct our study with a real-life system by a top leading company in the Air Traffic Control domain, which emits massive volumes of unstructured proprietary logs. We propose a filtering method that pinpoints interesting events from logs, i.e., events that should be followed up by analysts. Experiments are done with logs from normative and misuse scenarios; moreover, we compare the outcome of our method with a reference filtering technique based on the conceptual clustering. Results indicate that the proposed method is effective to retain interesting events at remarkable precision and recall and to pinpoint misuse indicators. We overcome several drawbacks of existing filtering techniques, such as the need for labeled logs and domain knowledge, which makes our method easier to use by practitioners.

On-line timing error detection entails gathering and analyzing monitoring data to pinpoint deviations from the expected timing behavior of a given software system. Current solutions for system monitoring and runtime analysis present several practical drawbacks that limit their usability in real industrial systems, such as the need of kernel-level probes or the coarse per-node/per-process monitoring granularity. This paper proposes a novel framework for timing error detection that capitalizes on the systematic interleaving of logging instructions across the functional code in order to overcome above limitations. The paper faces the practical challenges related to the specification and implementation of a log weaving technique, detection algorithms, and a data centralization platform to collect and analyze fine-grained execution traces in distributed systems. We experiment the proposed framework in two real-world critical information systems from the Crisis Management and the Air Traffic Control domains. Results show that our framework achieves 95% timing error coverage and allows reconstructing error trends with high statistical confidence at negligible performance overhead.

The idea of turning data centers executing scientific batch jobs into private clouds is as attractive as troubling. Cloud platforms may help both in limiting power consumption and in implementing fault tolerance strategies. However, there is also the fear that performance may worsen, and that the electricity required for longer job duration and fault tolerance implementation may overcome the saved one. In this paper, we present the consumability analysis for assessing the impact of cloud and fault tolerance tunings on scientific processing systems. The analysis considers performance, consumption, and dependability aspects, jointly. The aim is to pinpoint if, for a given system, there is a setting where consumption and job failure rate decrease, while performance is not affected. Applied to the scientific data center at our University, the analysis allowed us to find the proper selection of virtual machines’ configuration, consolidation strategy, and fault tolerance tuning.

It is nowadays extremely popular to devise applications of top of the blockchain technology, due to the flourishing of several open-source platforms and the progressive success of this technology outside its traditional cryptocurrency domain. Testing these applications is particularly challenging and traditionally limited to the performance and functionality verification. How-ever, the progressive adoption of such a technology in critical domains is calling for dependability guarantees. This paper highlights the challenges related to the dependability assessment of these applications, and sketches a possible research direction.

Wireless sensor networks (WSNs) are increasingly being adopted in critical applications. In these networks undesired events may undermine the reliability level; thus their effects need to be properly assessed from the early stages of the development process onwards to minimize the chances of unexpected problems during use. In this paper we propose two heuristic strategies: what-if analysis and robustness checking. They allow to drive designers towards optimal WSN deployment solutions, from the point of view of the connection and data delivery resiliency, exploiting a formal approach based on the event calculus formal language. The heuristics are backed up by a support tool aimed to simplify their adoption by system designers. The tool allows to specify the target WSN in a user-friendly way and it is able to elaborate the two heuristic strategies by means of the event calculus specifications automatically generated. The WSN reliability is assessed computing a set of specific metrics. The effectiveness of the strategies is shown in the context of three case studies. © 2014 Springer Science+Business Media New York.

Event logs have been widely used over the last three decades to analyze the failure behavior of a variety of systems. Nevertheless, the implementation of the logging mechanism lacks a systematic approach and collected logs are often inaccurate at reporting software failures: This is a threat to the validity of log-based failure analysis. This paper analyzes the limitations of current logging mechanisms and proposes a rule-based approach to make logs effective to analyze software failures. The approach leverages artifacts produced at system design time and puts forth a set of rules to formalize the placement of the logging instructions within the source code. The validity of the approach, with respect to traditional logging mechanisms, is shown by means of around 12,500 software fault injection experiments into real-world systems.

Grid computing is a largely adopted paradigm to federate geographically distributed data centers. Due to their size and complexity, grid systems are often affected by failures that may hinder the correct and timely execution of jobs, thus causing a non-negligible waste of computing resources. Despite the relevance of the problem, state-of-the-art management solutions for grid systems usually neglect the identification and handling of failures at runtime. Among the primary goals to be considered, we claim the need for novel approaches capable to achieve the objectives of scalable integration with efficient monitoring solutions and of fitting large and geographically distributed systems, where dynamic and configurable tradeoffs between overhead and targeted granularity are necessary. This paper proposes GAMESH, a Grid Architecture for scalable Monitoring and Enhanced dependable job ScHeduling. GAMESH is conceived as a completely distributed and highly efficient management infrastructure, concentrating on two crucial aspects for large-scale and multi-domain grid environments: (i) the scalable dissemination of monitoring data and (ii) the troubleshooting of job execution failures. GAMESH has been implemented and tested in a real deployment encompassing geographically distributed data centers across Europe. Experimental results show that GAMESH (i) enables the collection of measurements of both computing resources and conditions of task scheduling at geographically sparse sites, while imposing a limited overhead on the entire infrastructure, and (ii) provides a failure-aware scheduler able to improve the overall system performance, even in the presence of failures, by coordinating local job schedulers at multiple domains.

Virtualized multicore systems represent an emerging computing paradigm in the critical systems industry. Virtualization-based solutions leverage the different cores of the processor to run operating systems and applications within separate partitions, to support the development of parallel mixed-criticality systems, and to improve fault-tolerance by protecting and isolating the operating environments. The critical systems industry is subjected to international standards, which recommend fault injection as a mean to contribute with evidence to safety cases. This paper proposes a framework to inject hardware faults in virtualized multicore systems. Our proposal capitalizes on the error reporting architecture of modern processors and allows injecting faults both at hypervisor- and guest-OS-level. We implement the framework in the context of the widely used Intel Core i7 processor and Xen hypervisor. We demonstrate the use of the framework by means of about 60,000 injection experiments in a Linux-based virtualized multicore system installation.

This short paper introduces the design of AndroDepLog, a logging tool for enabling the live collection offailure and usage data. Collected data are useful to assessthe dependability behavior and software aging trends of anAndroid device during its exercise. The paper will discusspreliminary results concerning the application of the tool realAndroid devices, in the framework of a research collaborationbetween Critiware S.r.l. and an Android vendor.

The management of Grid systems commonly lacks information for identifying the failures that may hinder the timely completion of jobs, and cause the wasting of computing resources. Monitoring can certainly help, but novel approaches need to be conceived for such large and geographically distributed systems. We propose a Grid Architecture for scalable Monitoring and Enhanced dependable job ScHeduling (GAMESH). GAMESH is a completely distributed and highly efficient management infrastructure for the dissemination of monitoring data and troubleshooting of job execution failures in large-scale and multi-domain Grid environments. Challenged in a real deployment and compared to other Grid management systems, GAMESH demonstrates to (i) ensure measurements of both computing resources and conditions of task scheduling at geographically sparse sites, while inducing a low overhead on the entire infrastructure, and (ii) enable failure-aware scheduling and improve overall system performance, even in the presence of failures, by coordinating local job schedulers at multiple domains.

Wireless Sensor Networks (WSNs) are widely recognized as a promising solution to build next-generation monitoring systems. Their industrial uptake is however still compromised by the low level of trust on their performance and dependability. Whereas analytical models represent a valid mean to assess nonfunctional properties via simulation, their wide use is still limited by the complexity and dynamicity of WSNs, which lead to unaffordable modeling costs. To reduce this gap between research achievements and industrial development, this paper presents a framework for the assessment of WSNs based on the automated generation of analytical models. The framework hides modeling details, and it allows designers to focus on simulation results to drive their design choices. Models are generated starting from a high-level specification of the system and by a preliminary characterization of its fault-free behavior, using behavioral simulators. The benefits of the framework are shown in the context of two case studies, based on the wireless monitoring of civil structures.

Summary Software reliability growth models support the prediction/assessment of product quality, release time, and testing/debugging cost. Several software reliability growth model extensions take into account the bug correction process. However, their estimates may be significantly inaccurate when debugging fails to fully fit modelling assumptions. This paper proposes debugging-workflow-aware software reliability growth method (DWA-SRGM), a method for reliability growth analysis leveraging the debugging data usually managed by companies in bug tracking systems. On the basis of a characterization of the debugging workflow within the software project under consideration (in terms of bug features and treatment phases), DWA-SRGM pinpoints the factors impacting the estimates and to spot bottlenecks, thus supporting process improvement decisions. Two industrial case studies are presented, a customer relationship management system and an enterprise resource planning system, whose defects span a period of about 17 and 13 months, respectively. DWA-SRGM revealed effective to obtain more realistic estimates and to capitalize on the awareness of critical factors for improving debugging.

This paper discusses our preliminary analysis of event logging practices adopted in a large-scale industrial development process at Selex ES, a top-leading Finmeccanica company in electronic and information technologies for defense systems, aerospace, and land security. The analysis aims to support log reengineering activities that are currently conducted at SELEX ES. At time being, some of the issues described in the paper have been fixed by system developers. Analysis encompasses total around 50+ millions lines of log produced by an Air Traffic Control (ATC) system. Analysis reveals that event logging is not strictly regulated by company-wide practices, which results into heterogeneous logs across different development teams. We introduce our ongoing effort at developing an automatic support to browse collected logs along with a uniform logging policy supplementing the reengineering process.

This paper presents the main objectives and preliminary results of the NAPOLI FUTURA project, which aims to define novel approaches for protecting critical infrastructures from cyber attacks. The paper focuses on the architectural design of the NAPOLI FUTURA platform. The platform leverages cutting-edge big data analytics solutions to detect security attacks and to support live migration of services in the context of Critical Information Infrastructures (CIIs).

The analysis of monitoring data is extremely valuable for critical computer systems. It allows to gain insights into the failure behavior of a given system under real workload conditions, which is crucial to assure service continuity and downtime reduction. This paper proposes an experimental evaluation of different direct monitoring techniques, namely event logs, assertions, and source code instrumentation, that are widely used in the context of critical industrial systems. We inject 12,733 software faults in a real-world air traffic control (ATC) middleware system with the aim of analyzing the ability of mentioned techniques to produce information in case of failures. Experimental results indicate that each technique is able to cover a limited number of failure manifestations. Moreover, we observe that the quality of collected data to support failure diagnosis tasks strongly varies across the techniques considered in this study.

Reliability is one of the most relevant software quality attributes. The literature offers a variety of mathematical models - namely, software reliability growth models (SRGMs) - to estimate the reliability of a software product at a given time, as well as to predict the reliability that will be achieved as testing activities progress. One of the typical assumptions of SRGMs is the immediate debugging of detected faults. In reality, the impact of the debugging process cannot be neglected at all. This paper reports the results of a real-world case-study in which we analyze the debugging process of a Customer Relationship Management (CRM) system, and study its impact on SRGM-based reliability estimation and prediction.

As Wireless Sensor Networks (WSNs) are starting to be adopted in critical scenarios, it becomes important to study the behavior of WSN software in response to errors induced by hardware faults. To this aim, in this paper we present the results of an extensive fault-injection campaign, conducted on three different WSN operating systems (OSs). Results show that, depending on the concurrency model and on the memory management, the OS reacts to injected faults differently, indicating that fault containment strategies and hang-checking assertions should be implemented to avoid spreading and activations of errors. Analysis also allowed us to define a detailed dependability model of the WSN software, to be used to simulate the expected failure behavior of a given OS when solicited by given low-level hardware faults.

The design and realization of health monitoring systems has attracted the interest of large communities both from industry and academia. Remote and continuous monitoring of patient’s vital signs is the target of an emerging business market that aims both to improve the quality of life of patients and to reduce costs of national healthcare services. Such applications, however, are particularly critical from the point of view of dependability. This presents the design of a set of services for the assurance of high degrees of dependability to generic mobile health monitoring systems. The design is based on the results of a detailed failure modes and effects analysis (FMEA), conducted to identify the typical dependability threats of health monitoring systems. The FMEA allowed the authors to conceive a set of configurable monitoring services, enriching the system with the ability to detect failures at runtime, and enabling the realization of dependable services for future mobile health monitoring systems.

Multicore processors are becoming more and more attractive in embedded and safety-critical domains because they allow increasing the performance by ensuring reduced power consumption. However, moving to multicore systems raises novel dependability challenges: the number of cores, concurrency issues, shared resources and interconnections among cores make it hard to develop and validate software deployed on the top of multicore processors. This paper discusses a preliminary fault injection framework, which aims to investigate dependability properties of multicore-based systems. The proposed framework leverages the error reporting architecture provided by modern processors and has been instantiated in the context of the Intel Core i7 processor. Fault injection campaigns have been conducted under the Linux OS to show the benefits of the framework.

The increasing complexity of smart phones introduces new dependability threats. Often, novel applications and features are delivered with scarce testing, due to the impressive market pressure. Despite these concerns, there is still little understanding on the dependability behavior of today smart phones. This paper proposes the design of a logging platform for the Android OS. The logging platform enables the collection of failure data, useful to assess the dependability of smart phones at runtime. Preliminary experimental results on real-world Android devices show the feasibility of the approach and encourage further research activities.

Many critical services are nowadays provided by large and complex software systems. However, the increasing complexity introduces several sources of non-determinism, which may lead to hang failures: the system appears to be running, but part of its services is perceived as unresponsive. Online monitoring is the only way to detect and to promptly react to such failures. However, when dealing with off-the-shelf-based systems, online detection can be tricky since instrumentation and log data collection may not be feasible in practice. In this paper, a detection framework to cope with software hangs is proposed. The framework enables the non-intrusive monitoring of complex systems, based on multiple sources of data gathered at the operating system (OS) level. Collected data are then combined to reveal hang failures. The framework is evaluated through a fault injection campaign on two complex systems from the air traffic management (ATM) domain. Results show that the combination of several monitors at the OS level is effective to detect hang failures in terms of coverage and false positives and with a negligible impact on performance.

Direct analysis of failures from the field of application is an effective practice to understand the actual dependability behavior of an operational system. However, despite its wide use over the last four decades on a large variety of systems, field data analysis has rarely been applied to wireless networks. Users accessing the Internet ubiquitously through these networks are increasing, and they expect the same dependability level they already experience on wired networks. But how can we analyze the dependability level of a wireless network? The article defines a novel combined approach to model and analyze the dependability of short range wireless technologies exploiting field data. Through the experience gained from extensive failure analysis of Bluetooth networks, the paper shows how field failure data can play a key role to fill the gap on understanding the dependability behavior of wireless networks.

As the incidence of faults in real Wireless Sensor Networks (WSNs) increases, fault injection is starting to be adopted to verify and validate their design choices. Following this recent trend, this paper presents a tool, named AVR-INJECT, designed to automate the fault injection, and analysis of results, on WSN nodes. The tool emulates the injection of hardware faults, such as bit flips, acting via software at the assembly level. This allows to attain simplicity, while preserving the low level of abstraction needed to inject such faults. The potential of the tool is shown by using it to perform a large number of fault injection experiments, which allow to study the reaction to faults of real WSN software.

On-line failure detection is an essential means to control and assess the dependability of complex and critical software systems. In such context, effective detection strategies are required, in order to minimize the possibility of catastrophic consequences. This objective is however difficult to achieve in complex systems, especially due to the several sources of non-determinism (e.g., multi-threading and distributed interaction) which may lead to software hangs, i.e., the system is active but no longer capable of delivering its services. The paper proposes a detection approach to uncover application hangs. It exploits multiple indirect data gathered at the operating system level to monitor the system and to trigger alarms if the observed behavior deviates from the expected one. By means of fault injection experiments conducted on a research prototype, it is shown how the combination of several operating system monitors actually leads to an high quality of detection, at an acceptable overhead.

This paper proposes a flexible framework for dependability modeling and assessing of Wireless Sensor Networks (WSNs). The framework takes into account network related aspects (topology, routing, network traffic) as well as hardware/software characteristics of nodes (type of sensors, running applications, power consumption). It is composed of two basic elements: i) a parametric Stochastic Activity Networks (SAN) failure model, reproducing WSN failure behavior as inferred from a detailed Failure Mode Effect Analysis (FMEA), and ii) an external library reproducing network behavior on behalf of the SAN model. This library specializes the SAN model by feeding it with quantitative parameters obtained by simulation or by experimental campaigns; it is also in charge of updating the network state in response to failure events during the simulation (e.g., routing tree updated due to node failures). The framework is thus suited to evaluate the dependability of several WSNs, with different topologies, routing algorithms, hardware/software platforms, without requiring any changes to its structure. The use of the external library makes the model simpler, decoupling the network behavior from the failure behavior. Simulation experiments are discussed that provide a quantitative evaluation of WSN dependability for a sample scenario: results show how the proposed framework supports WSN developers to find proper cost-reliability trade-offs for the system being deployed.

The increasing success of mobile-enabled embedded devices is stressing the need for software architectures facing mobility-related issues. This paper proposes a simple yet effective mobility management scheme to ease the development of mobile ubiquitous applications. The scheme seamlessly handles handoff events and provides ubiquitous applications with both location-awareness and mobility prediction support. An implementation prototype has been developed on real-world Bluetooth enabled devices. Experimental results are then obtained from the prototype, showing the effectiveness of the proposed scheme.