Search

Objectives: Decision makers adopt health technologies based on health economic models that are subject to uncertainty. In an ideal world, these models parameterize all uncertainties and reflect them in the cost-effectiveness probability and risk associated with the adoption. In practice, uncertainty assessment is often incomplete, potentially leading to suboptimal reimbursement recommendations and risk management. This study examines the feasibility of comprehensive uncertainty assessment in health economic models.Methods: A state transition model on peripheral arterial disease treatment was used as a case study. Uncertainties were identified and added to the probabilistic sensitivity analysis if possible. Parameter distributions were obtained by expert elicitation, and structural uncertainties were either parameterized or explored in scenario analyses, which were model averaged.Results: A truly comprehensive uncertainty assessment, parameterizing all uncertainty, could not be achieved. Expert elicitation informed 8 effectiveness, utility, and cost parameters. Uncertainties were parameterized or explored in scenario analyses and with model averaging. Barriers included time and resource constraints, also of clinical experts, and lacking guidance regarding some aspects of expert elicitation, evidence aggregation, and handling of structural uncertainty. The team's multidisciplinary expertise and existing literature and tools were facilitators.Conclusions: While comprehensive uncertainty assessment may not be attainable, improvements in uncertainty assessment in general are no doubt desirable. This requires the development of detailed guidance and hands-on tutorials for methods of uncertainty assessment, in particular aspects of expert elicitation, evidence aggregation, and handling of structural uncertainty. The issue of benefits of uncertainty assessment versus time and resources needed remains unclear.

Karst aquifers are characterized by high-conductivity conduits embedded in a low-conductivity fractured matrix, resulting in extreme heterogeneity and variable groundwater flow behavior. The conduit network controls groundwater flow, but is often unmapped, making it difficult to apply numerical models to predict system behavior. This paper presents a multi-model ensemble method to represent structural and conceptual uncertainty inherent in simulation of systems with limited spatial information, and to guide data collection. The study tests the new method by applying it to a well-mapped, geologically complex long-term study site: the Gottesacker alpine karst system (Austria/Germany). The ensemble generation process, linking existing tools, consists of three steps: creating 3D geologic models using GemPy (a Python package), generating multiple conduit networks constrained by the geology using the Stochastic Karst Simulator (a MATLAB script), and, finally, running multiple flow simulations through each network using the Storm Water Management Model (C-based software) to reject nonbehavioral models based on the fit of the simulated spring discharge to the observed discharge. This approach captures a diversity of plausible system configurations and behaviors using minimal initial data. The ensemble can then be used to explore the importance of hydraulic flow parameters, and to guide additional data collection. For the ensemble generated in this study, the network structure was more determinant of flow behavior than the hydraulic parameters, but multiple different structures yielded similar fits to the observed flow behavior. This suggests that while modeling multiple network structures is important, additional types of data are needed to discriminate between networks., National Science Foundation Graduate Research Fellowship Program

In this introductory paper to the special issue on crop model prediction uncertainty, we present and compare the methodological choices in the studies included in this issue, and highlight some remaining challenges. As a common framework for all studies, we define prediction uncertainty as the distribution of prediction error, which can be written as the sum of a bias plus a predictor uncertainty term that represents the random variation due to uncertainty in model structure, model parameters or model inputs. Several themes recur in many of the studies: Use of multi-model ensembles (MMEs) to quantify model structural uncertainty; Emphasis on uncertainty in those inputs related to prediction of regional results or climate change impact assessment; Simultaneous consideration of multiple sources of uncertainty; Emphasis on exploring the variability of uncertainty over space or time; Use of sensitivity analysis techniques to disaggregate the separate contributions to prediction uncertainty. Relatively new approaches include the estimation of both the bias and predictor uncertainty terms of prediction error, the construction of MMEs specifically designed to explore the uncertainty in model structure, the use of emulators for sensitivity analysis and the exploration of ways to reduce prediction uncertainty other than through model improvement. Major remaining challenges are standardization of approaches to quantifying uncertainty in model structure, parameters and inputs, going beyond studies of specific sources of uncertainty to estimation of overall prediction uncertainty, comparing and combining validation and uncertainty studies, and evaluation of uncertainty estimates. Looking forward, we suggest that assessment of prediction uncertainty should be a standard part of any modelling project. The studies here will contribute toward that goal.

Integrated assessment (IA) modelling can be an effective tool to gain insight into the dynamics of coupled earth system (land use, climate etc.) and socio-economic components. Quantifying and communicating uncertainties is a challenge of any scientific assessment, but is here magnified by the complex and boundary-crossing nature of IA models. Understanding the dynamics of coupled earth and socio-economic systems require data and methods from multiple disciplines, each with its own perspective on epistemological uncertainties (parametric and structural uncertainties), and its own protocols for assessing uncertainty. During the Paris Agreement, the lack of uncertainty analyses (UA) in IAs was risen ( Rogelj et al. 2017 ) and calls for close collaboration of scientists coming from different fields. In this study, we review how uncertainties are tackled in a range of science disciplines that are related to global change including climate, hydrology, energy and land use, and which contribute to IA modelling. We conducted a meta-analysis to identify the contributing disciplines, and review which type of uncertainties are assessed. We then describe sources of uncertainty (e.g. parameter values, model structure), and present opportunities for improved assessment and communication of uncertainties in IA modelling. We show in our meta-analysis that parametric uncertainty is the uncertainty analysis that has been applied the most, while structural uncertainty is less commonly applied, with the exception of the energy scientific discipline. We finish our study with key recommendations to improve uncertainty analysis such as including risk analysis. By embracing uncertainties, resilient and effective solutions for climate change mitigation and adaptation could be better communicated, identified and implemented.

Background The conditional reimbursement policy for expensive medicines in The Netherlands requires data collection on actual use and cost-effectiveness after the initial decision to reimburse a drug. This introduces new sources of uncertainty (less important in a randomized controlled trial than in daily practice), which may affect priorities for further research. Objectives This article focuses on determining the impact of including these uncertainties at the time a decision is made, and whether more complex models are always needed to address prioritization of additional research. Methods We constructed a typical decision model for chronic progressive diseases with four health states and parameters related to transition and exacerbation probabilities, costs, and utilities. Different scenarios are built on the basis of three additional uncertainties: persistence, compliance, and broadening of indication. Persistence refers to treatment duration. Compliance describes the fraction of treatment benefit obtained because of not taking the medication as prescribed. Broadening of indication reflects a shift in the severity distribution at treatment start. These uncertainties were parameterized in the model and included in the value-of-information analysis. Results The most important parameters were transition probabilities. Broadening of indication had little impact on the overall uncertainty. Compliance and persistence were important when establishing priorities for further research. Major differences with respect to the reference scenario were due to the parameterization of compliance in the decision model. Conclusions The usual practice of modeling only randomized controlled trial data at the time the decision on conditional reimbursement is made can lead to wrong decisions. Additional uncertainties arising from outcomes studies should be anticipated at an early stage and included in the model because this can have a strong impact on the prioritization of further research.

The aim of this chapter is to provide guidance regarding the various types and sources of uncertainty that influence the outcome of a multi-criteria decision analysis (MCDA) model. For each MCDA step, i.e., structuring, scoring, weighting, and aggregating, we will describe sources of uncertainty and point to methods to deal with these uncertainties. Also the use of sensitivity analyses and the relevance of qualifying and quantifying uncertainty in MCDA will be discussed. The consideration of uncertainty is a difficult but important balancing act between capturing the complex uncertainties of the decision and keeping the MCDA comprehensible for decision makers.

Background Identifying a proper model structure, using methods that address both structural and parameter uncertainty, is a crucial problem within the systems approach to biology. And yet, it has a marginal presence in the recent literature. While many existing approaches integrate methods for simulation and parameter estimation of a single model to address parameter uncertainty, only few of them address structural uncertainty at the same time. The methods for handling structure uncertainty often oversimplify the problem by allowing the human modeler to explicitly enumerate a relatively small number of alternative model structures. On the other hand, process-based modeling methods provide flexible modular formalisms for specifying large classes of plausible model structures, but their scope is limited to deterministic models. Here, we aim at extending the scope of process-based modeling methods to inductively learn stochastic models from knowledge and data. Results We combine the flexibility of process-based modeling in terms of addressing structural uncertainty with the benefits of stochastic modeling. The proposed method combines search trough the space of plausible model structures, the parsimony principle and parameter estimation to identify a model with optimal structure and parameters. We illustrate the utility of the proposed method on four stochastic modeling tasks in two domains: gene regulatory networks and epidemiology. Within the first domain, using synthetically generated data, the method successfully recovers the structure and parameters of known regulatory networks from simulations. In the epidemiology domain, the method successfully reconstructs previously established models of epidemic outbreaks from real, sparse and noisy measurement data. Conclusions The method represents a unified approach to modeling dynamical systems that allows for flexible formalization of the space of candidate model structures, deterministic and stochastic interpretation of model dynamics, and automated induction of model structure and parameters from data. The method is able to reconstruct models of dynamical systems from synthetic and real data.

Model uncertainty and limited data are fundamental challenges to robust management of human intervention in a natural system. These challenges are acutely highlighted by concerns that many ecological systems may contain tipping points, such as Allee population sizes. Before a collapse, we do not know where the tipping points lie, if they exist at all. Hence, we know neither a complete model of the system dynamics nor do we have access to data in some large region of state-space where such a tipping point might exist. We illustrate how a Bayesian Non-Parametric (BNP) approach using a Gaussian Process (GP) prior provides a flexible representation of this inherent uncertainty. We embed GPs in a Stochastic Dynamic Programming (SDP) framework in order to make robust management predictions with both model uncertainty and limited data. We use simulations to evaluate this approach as compared with the standard approach of using model selection to choose from a set of candidate models. We find that model selection erroneously favors models without tipping points -- leading to harvest policies that guarantee extinction. The GPDP performs nearly as well as the true model and significantly outperforms standard approaches. We illustrate this using examples of simulated single-species dynamics, where the standard model selection approach should be most effective, and find that it still fails to account for uncertainty appropriately and leads to population crashes, while management based on the GPDP does not, since it does not underestimate the uncertainty outside of the observed data., 2015 Proceedings of the Royal Society B: Biological Sciences

The problem of placing sensors at appropriate locations plays a central role in dynamic assessment of mechanical systems. All sensors have to be placed on the structure such that required system features can be obtained during tests in a robust way, that means with the maximum information quality and the smaller sensitivities to parameters uncertainties. A robust approach is proposed in this paper, were is considered the determination of optimal sensor locations on a uncertain linear mechanical systems. The criteria here proposed is obtained by using a modified version of the classical effective independence method.

Comprehensive problem framing that includes different perspectives is essential for holistic understanding of complex problems and as the first step in building models. We involved five stakeholders to frame the management problem of the Central Baltic herring fishery. By using the Bayesian belief networks (BBNs) approach, the views of the stakeholders were built into graphical influence diagrams representing variables and their dependencies. The views of the scientists involved concentrated on biological concerns, whereas the fisher, the manager, and the representative of an environmental nongovernmental organization included markets and fishing industry influences. Management measures were considered to have a relatively small impact on the development of the herring stock; their impact on socioeconomic objectives was greater. Overall, the framings by these stakeholders propose a focus on socioeconomic issues in research and management and explicitly define management objectives, not only in biological but also in social and economic terms. We find the approach an illustrative tool to structure complex issues systematically. Such a tool can be used as a forum for discussion and for decision support that explicitly includes the views of different stakeholder groups. It enables the examination of social and biological factors in one framework and facilitates bridging the gap between social and natural sciences. A benefit of the BBN approach is that the graphical model structures can be transformed into a quantitative form by inserting probabilistic information.

Background Standard approaches to estimation of Markov models with data from randomized controlled trials tend either to make a judgment about which transition(s) treatments act on, or they assume that treatment has a separate effect on every transition. An alternative is to fit a series of models that assume that treatment acts on specific transitions. Investigators can then choose among alternative models using goodness-of-fit statistics. However, structural uncertainty about any chosen parameterization will remain and this may have implications for the resulting decision and the need for further research. Methods We describe a Bayesian approach to model estimation, and model selection. Structural uncertainty about which parameterization to use is accounted for using model averaging and we developed a formula for calculating the expected value of perfect information (EVPI) in averaged models. Marginal posterior distributions are generated for each of the cost-effectiveness parameters using Markov Chain Monte Carlo simulation in WinBUGS, or Monte-Carlo simulation in Excel (Microsoft Corp., Redmond, WA). We illustrate the approach with an example of treatments for asthma using aggregate-level data from a connected network of four treatments compared in three pair-wise randomized controlled trials. Results The standard errors of incremental net benefit using structured models is reduced by up to eight- or ninefold compared to the unstructured models, and the expected loss attaching to decision uncertainty by factors of several hundreds. Model averaging had considerable influence on the EVPI. Conclusions Alternative structural assumptions can alter the treatment decision and have an overwhelming effect on model uncertainty and expected value of information. Structural uncertainty can be accounted for by model averaging, and the EVPI can be calculated for averaged models.

International audience; This paper presents an approach for reducing uncertainty related to the process of land-cover change (LCC) prediction. LCC prediction models have, almost, two sources of uncertainty which are the uncertainty related to model parameters and the uncertainty related to model structure. These uncertainties have a big impact on decisions of the prediction model. To deal with these problems, the proposed approach is divided into three main steps: (1) an uncertainty propagation step based on possibility theory is used as a tool to evaluate the performance of the model; (2) a sensitivity analysis step based on Hartley-like measure is then used to find the most important sources of uncertainty; and (3) a knowledge base based on machine learning algorithm is built to identify the reduction factors of all uncertainty sources of parameters and to reshape their values to reduce in a significant way the uncertainty about future changes of land cover. In this study, the present and future growths of two case studies were anticipated using multi-temporal Spot-4 and Landsat satellite images. These data are used for the preparation of prediction map of year 2025. The results show that our approach based on possibility theory has a potential for reducing uncertainty in LCC prediction modeling.

A history matching workflow, as typically used for updating of petrophysical reservoir model properties, is modified to include structural parameters including the top reservoir and several fault properties: position, slope, throw and transmissibility. A simple 2D synthetic oil reservoir produced by waterflooding is used to demonstrate issues related to observability and identifiability of structural uncertainties. A series of history matching experiments are performed that demonstrate the potential for estimation of uncertain structural parameter values. The experiments are performed with ensemble-based assisted history matching schemes that allow for the proper characterization of prior uncertainties, and in principle enable the updating of both model parameters and the dynamic states such as pressures and saturations. The results show that the approach works well when individual structural parameters are estimated and suggest further scope for building a fully integrated workflow that includes the improved construction of geo-models into the history matching loop.