Search

Here, performance assessment of complex systems is ideally accomplished through system-level testing, but because they are expensive, such tests are seldom performed. On the other hand, for economic reasons, data from tests on individual components that are parts of complex systems are more readily available. The lack of system-level data leads to a need to build computational models of systems and use them for performance prediction in lieu of experiments. Because their complexity, models are sometimes built in a hierarchical manner, starting with simple components, progressing to collections of components, and finally, to the full system. Quantification of uncertainty in the predicted response of a system model is required in order to establish confidence in the representation of actual system behavior. This paper proposes a framework for the complex, but very practical problem of quantification of uncertainty in system-level model predictions. It is based on Bayes networks and uses the available data at multiple levels of complexity (i.e., components, subsystem, etc.). Because epistemic sources of uncertainty were shown to be secondary, in this application, aleatoric only uncertainty is included in the present uncertainty quantification. An example showing application of the techniques to uncertainty quantification of measures of response of amore » real, complex aerospace system is included.« less

Performance assessment of complex systems is ideally done through full system-level testing which is seldom available for high consequence systems. Further, a reality of engineering practice is that some features of system behavior are not known from experimental data, but from expert assessment, only. On the other hand, individual component data, which are part of the full system are more readily available. The lack of system level data and the complexity of the system lead to a need to build computational models of a system in a hierarchical or building block approach (from simple components to the full system). The models are then used for performance prediction in lieu of experiments, to estimate the confidence in the performance of these systems. Central to this are the need to quantify the uncertainties present in the system and to compare the system response to an expected performance measure. This is the basic idea behind Quantification of Margins and Uncertainties (QMU). QMU is applied in decision making—there are many uncertainties caused by inherent variability (aleatoric) in materials, configurations, environments, etc., and lack of information (epistemic) in models for deterministic and random variables that influence system behavior and performance. This paper proposes a methodology to quantify margins and uncertainty in the presence of both aleatoric and epistemic uncertainty. It presents a framework based on Bayes networks to use available data at multiple levels of complexity (i.e. components, subsystem, etc.) and demonstrates a method to incorporate epistemic uncertainty given in terms of intervals on a model parameter.

There is an increasing reliance in the engineering community on the use of mathematical models to characterize physical system behavior. This is happening even though mathematical models rarely simulate real system behavior perfectly. Due to this reliance, we require objective, well-founded mathematical techniques for model validation. This paper develops a formal approach to the validation of mathematical models of structural dynamics systems. It uses a probabilistic/statistical approach to the characterization of an important measure of behavior of dynamic systems subjected to random excitations, and seeks to validate a mathematical model in a statistical sense. An example is presented.

Model verification and validation (V&V) is an enabling methodology for the development of numerical models that can be used to make predictions with quantified accuracy and confidence. Model V&V procedures are needed to reduce the time, cost and risk associated with component and full-scale testing of engineered systems. Quantifying the confidence and predictive accuracy of model calculations provides the decision-maker with the information necessary for making high-consequence decisions. One of the key steps in a model validation project is to compare model predictions and experimental data given uncertainties in both. This step is critical because it defines how the model will be challenged against the physical data relative to the intended use of the model. This paper briefly reviews the main concepts involved in model V&V, introduces a simple measure termed the z-metric, and compares the z-metric with another measure called the area metric.

A systematic approach to defining margin in a manner that incorporates statistical information and accommodates data uncertainty, but does not require assumptions about specific forms of the tails of distributions is developed. This approach extends to calculations underlying validation assessment and quantitatively conservative predictions.

The objective of this paper is to present a framework for modeling and simulation of parameter vector random processes. While the approach described in the paper can be applied generally, we are interested in problems where there is a limited set of random process data (e.g., FRFs or IRFs) for which the dominant sources of randomness (i.e., the dominant random parameters) are not known a-priori. From this limited set of random process data we aim to create a framework for estimating the underlying distribution to enable simulation and analysis of the family of systems from which the original data set came. The approach recommended for modeling random parameter vectors for physical systems is the Karhunen-Loeve expansion. Our innovation is to use additional knowledge (constraints) about the system that spawned the random process in a screening that classifies the synthetic realizations as plausible or implausible and yields a more representative data set.

Metrics for determining the stability- and performance-robustness of a given controller to variations in open-loop dynamics are examined. Of particular interest are the probability distributions that these metrics have given some amount of uncertainty in the plant. The analysis herein characterizes these distributions using frequency response data collected experimentally for the open-loop plant. In doing so, it avoids using a model for the plant, thereby reducing errors and uncertainty inherent to the modeling of a plant. For a proposed controller, a Monte-Carlo method is used to evaluate the closed-loop performance and stability metrics for a large ensemble of test data and thus create an estimate of their distributions. Since ensembles of test data are typically quite small, a Karhunen-Loeve expansion based method is used to synthetically expand the set of FRFs.

Modeling and simulation to aid in decision making, such as qualification of high consequence systems, is becoming more prevalent. With this increased use of models, there arises a need to formally assess the accuracy of the model underlying these decisions. This process is referred to as model validation. Furthermore an understanding of the uncertainty in the model responses is necessary if the model is used to predict the response of internal system components for establishing their performance. These responses, presumably, cannot be directly measured and/or multiple pieces of hardware are not available to quantify their inherent variability and thus the value of the model is heightened. Additionally, requirements on component assessment that require quantification of margins, in the presence of uncertainty, provide yet another motivation to use modeling and simulation. In this paper, we describe a model validation process to assess the accuracy of a system level model and perform an assessment of the model relative to existing data.

Verification and validation (V&V) are processes designed to assure, with quantifiable confidence, that the computational models we use, solve the problems they are meant to solve, and that the models adequately represent the structures they are meant to model. The American Society of Mechanical Engineers (ASME) Standards Committee on Verification and Validation in Computational Solid Mechanics (PTC 60/V&V 10) has approved a Guide for performing V&V [1]. The Guide is brief, and does not provide specific methods for the performance of model V&V, therefore, subcommittees have been formed to develop documents specifying methods for accomplishing the various activities of V&V. One of these subcommittees was formed to develop the “Uncertainty Quantification Supplement to the ASME V&V 10-2006 Guide.” The talks in this special session describe some of the activities associated with uncertainty quantification in V&V. This paper presents an example used to illuminate some of the tools and measures developed in the other papers in this sequence. The example describes experimental results and model predictions.

We present a proportional hazards model (PHM) that establishes a framework suitable for performing reliability estimates and risk prognostics on complex multi-component systems which are transferred at arbitrary times among a discrete set of non-stationary stochastic environments. Such a scenario is not at all uncommon for portable and mobile systems. It is assumed that survival data, possibly interval censored, is available at several “typical” environments. This collection of empirical survival data forms the foundation upon which the basic effects of selected covariates are incorporated via the proportional hazards model. Proportional hazards models are well known in medical statistics, and can provide a variety of data-driven risk models which effectively capture the effects of the covariates. The paper describes three modifications we have found most suitable for this class of systems: development of suitable survival estimators that function well under realistic censoring scenarios, our modifications to the PHM which accommodate time-varying stochastic covariates, and implementation of said model in a non-linear network context which is itself model-free. Our baseline hazard is a parameterized reliability model developed from the empirical reliability estimates. Development of the risk score for arbitrary covariates arising from movement among different random environments is through interaction of the non-linear network and training data obtained from a Markov chain simulation based on stochastic environmental responses generated from Karhunen-Loeve models.Copyright © 2005 by ASME

We frequently develop mathematical models of system behavior and sometimes use test data to help identify the parameters of the mathematical model. However, no general purpose technique exists for formally, statistically judging the quality of a model. The paper suggests a formal statistical procedure for the validation of mathematical models of systems when data taken during operation of the system are available. The statistical validation procedure is based on the bootstrap, and it seeks to build a framework where a statistical test of hypothesis can be run to determine whether or not a mathematical model is an acceptable model of a system with regard to user specified measures of system behavior. A numerical example is presented to demonstrate the application of the technique.

Dynamic thermography is a promising technology for inspecting metallic and composite structures used in high-consequence industries. However, the reliability and inspection sensitivity of this technology has historically been limited by the need for extensive operator experience and the use of human judgment and visual acuity to detect flaws in the large volume of infrared image data collected. To overcome these limitations new automated data analysis algorithms and software is needed. The primary objectives of this research effort were to develop a data processing methodology that is tied to the underlying physics, which reduces or removes the data interpretation requirements, and which eliminates the need to look at significant numbers of data frames to determine if a flaw is present. Considering the strengths and weakness of previous research efforts, this research elected to couple both the temporal and spatial attributes of the surface temperature. Of the possible algorithms investigated, the best performing was a radiance weighted root mean square Laplacian metric that included a multiplicative surface effect correction factor and a novel spatio-temporal parametric model for data smoothing. This metric demonstrated the potential for detecting flaws smaller than 0.075 inch in inspection areas on the order of one square foot. Included in this report is the development of a thermal imaging model, a weighted least squares thermal data smoothing algorithm, simulation and experimental flaw detection results, and an overview of the ATAC (Automated Thermal Analysis Code) software that was developed to analyze thermal inspection data.