Search

While sensitivity analysis improves the transparency and reliability of mathematical models, its uptake by modelers is still scarce. This is partially explained by its technical requirements, which may be hard to understand and implement by the non-specialist. Here we propose a sensitivity analysis approach based on the concept of discrepancy that is as easy to understand as the visual inspection of input-output scatterplots. Firstly, we show that some discrepancy measures are able to rank the most influential parameters of a model almost as accurately as the variance-based total sensitivity index. We then introduce an ersatz-discrepancy whose performance as a sensitivity measure matches that of the best-performing discrepancy algorithms, is simple to implement, easier to interpret and orders of magnitude faster.

Here we briefly reflect on the philosophical foundations that ground the quest towards ever-detailed models and identify four practical dangers derived from this pursuit: explosion of the model's uncertainty space, model black-boxing, computational exhaustion and model attachment. We argue that the growth of a mathematical model should be carefully and continuously pondered lest models become extraneous constructs chasing the Cartesian dream., Comment: This is a draft of a chapter that has been accepted for publication by Oxford University Press in the forthcoming book "Views on Mathematical Modelling", edited by Andrea Saltelli and Monica Di Fiore and due for publication in 2023

Variance-based sensitivity indices have established themselves as a reference among practitioners of sensitivity analysis of model outputs. A variance-based sensitivity analysis typically produces the first-order sensitivity indices $S_j$ and the so-called total-effect sensitivity indices $T_j$ for the uncertain factors of the mathematical model under analysis. The cost of the analysis depends upon the number of model evaluations needed to obtain stable and accurate values of the estimates. While efficient estimation procedures are available for $S_j$, this availability is less the case for $T_j$. When estimating these indices, one can either use a sample-based approach whose computational cost depends on the number of factors or use approaches based on meta modelling/emulators. The present work focuses on sample-based estimation procedures for $T_j$ and tests different avenues to achieve an algorithmic improvement over the existing best practices. To improve the exploration of the space of the input factors (design) and the formula to compute the indices (estimator), we propose strategies based on the concepts of economy and explorativity. We then discuss how several existing estimators perform along these characteristics. We conclude that: a) sample-based approaches based on the use of multiple matrices to enhance the economy are outperformed by designs using fewer matrices but with better explorativity; b) among the latter, asymmetric designs perform the best and outperform symmetric designs having corrective terms for spurious correlations; c) improving on the existing best practices is fraught with difficulties; and d) ameliorating the results comes at the cost of introducing extra design parameters., Comment: arXiv admin note: text overlap with arXiv:1703.05799

This paper explores how the modelling of energy systems may lead to undue closure of alternatives by generating an excess of certainty around some of the possible policy options. We exemplify the problem with two cases: first, the International Institute for Applied Systems Analysis (IIASA) global modelling in the 1980s; and second, the modelling activity undertaken in support of the construction of a radioactive waste repository at Yucca Mountain (Nevada, USA). We discuss different methodologies for quality assessment that may help remedy this issue, which include NUSAP (Numeral Unit Spread Assessment Pedigree), diagnostic diagrams, and sensitivity auditing. We demonstrate the potential of these reflexive modelling practices in energy policy making with four additional cases: (i) stakeholders evaluation of the assessment of the external costs of a potential large-scale nuclear accident in Belgium in the context of the ExternE (External Costs of Energy) project; (ii) the case of the ESME (Energy System Modelling Environment) for the creation of UK energy policy; (iii) the NETs (Negative Emission Technologies) uptake in Integrated Assessment Models (IAMs); and (iv) the Ecological Footprint (EF) indicator. We encourage modellers to widely adopt these approaches to achieve more robust and inclusive modelling activities in the field of energy modelling., Comment: 39 pages, 2 tables, 3 figures

The R package "sensobol" provides several functions to conduct variance-based uncertainty and sensitivity analysis, from the estimation of sensitivity indices to the visual representation of the results. It implements several state-of-the-art first and total-order estimators and allows the computation of up to third-order effects, as well as of the approximation error, in a swift and user-friendly way. Its flexibility makes it also appropriate for models with either a scalar or a multivariate output. We illustrate its functionality by conducting a variance-based sensitivity analysis of three classic models: the Sobol' (1998) G function, the logistic population growth model of Verhulst (1845), and the spruce budworm and forest model of Ludwig, Jones and Holling (1976)., Comment: The paper has been accepted for publication in the Journal of Statistical Software (expected for 2022)

The Variogram Analysis of Response Surfaces (VARS) has been proposed by Razavi and Gupta as a new comprehensive framework in sensitivity analysis. According to these authors, VARS provides a more intuitive notion of sensitivity and it is much more computationally efficient than Sobol' indices. Here we review these arguments and critically compare the performance of VARS-TO, for total-order index, against the total-order Jansen estimator. We argue that, unlike classic variance-based methods, VARS lacks a clear definition of what an "important" factor is, and show that the alleged computational superiority of VARS does not withstand scrutiny. We conclude that while VARS enriches the spectrum of existing methods for sensitivity analysis, especially for a diagnostic use of mathematical models, it complements rather than substitutes classic estimators used in variance-based sensitivity analysis., Comment: Currently under review in Environmental Modelling & Software

Sensitivity analysis helps identify which model inputs convey the most uncertainty to the model output. One of the most authoritative measures in global sensitivity analysis is the Sobol' total-order index, which can be computed with several different estimators. Although previous comparisons exist, it is hard to know which estimator performs best since the results are contingent on the benchmark setting defined by the analyst (the sampling method, the distribution of the model inputs, the number of model runs, the test function or model and its dimensionality, the weight of higher order effects or the performance measure selected). Here we compare several total-order estimators in an eight-dimension hypercube where these benchmark parameters are treated as random parameters. This arrangement significantly relaxes the dependency of the results on the benchmark design. We observe that the most accurate estimators are Razavi and Gupta's, Jansen's or Janon/Monod's for factor prioritization, and Jansen's, Janon/Monod's or Azzini and Rosati's for approaching the "true" total-order indices. The rest lag considerably behind. Our work helps analysts navigate the myriad of total-order formulae by reducing the uncertainty in the selection of the most appropriate estimator., Comment: Previous versions of this manuscript were titled "The Battle of Total-Order Sensitivity Indices". The version with the title "A comprehensive comparison of total-order estimators for global sensitivity analysis" has been accepted for publication in the International Journal for Uncertainty Quantification. Please kindly cite the journal version

De Graaf et al. (2019) suggest that groundwater pumping will bring 42--79\% of worldwide watersheds close to environmental exhaustion by 2050. We are skeptical of these figures due to several non-unique assumptions behind the calculation of irrigation water demands and the perfunctory exploration of the model's uncertainty space. Their sensitivity analysis reveals a widespread lack of elementary concepts of design of experiments among modellers, and can not be taken as a proof that their conclusions are robust., Comment: Comment on the paper by De Graaf et al. 2019. Environmental flow limits to global groundwater pumping. Nature 574 (7776), 90-94

The PAWN index is gaining traction among the modelling community as a sensitivity measure. However, the robustness to its design parameters has not yet been scrutinized: the size ($N$) and sampling ($\varepsilon$) of the model output, the number of conditioning intervals ($n$) or the summary statistic ($\theta$). Here we fill this gap by running a sensitivity analysis of a PAWN-based sensitivity analysis. We compare the results with the design uncertainties of the Sobol' total-order index ($S_{Ti}^*$). Unlike in $S_{Ti}^*$, the design uncertainties in PAWN create non-negligible chances of producing biased results when ranking or screening inputs. The dependence of PAWN upon ($N,n,\varepsilon, \theta$) is difficult to tame, as these parameters interact with one another. Even in an ideal setting in which the optimum choice for ($N,n,\varepsilon, \theta$) is known in advance, PAWN might not allow to distinguish an influential, non-additive model input from a truly non-influential model input., Comment: The paper has been accepted for publication in Environmental Modeling and Software (use the journal version to cite the paper)

Variance-based sensitivity indices have established themselves as a reference among practitioners of sensitivity analysis of model output. It is not unusual to consider a variance-based sensitivity analysis as informative if it produces at least the first order sensitivity indices S_j and the so-called total-effect sensitivity indices T_j for all the uncertain factors of the mathematical model under analysis. Computational economy is critical in sensitivity analysis. It depends mostly upon the number of model evaluations needed to obtain stable values of the estimates. While efficient estimation procedures independent from the number of factors under analysis are available for the first order indices, this is less the case for the total sensitivity indices. When estimating T_j, one can either use a sample-based approach, whose computational cost depends fromon the number of factors, or approaches based on meta-modelling/emulators, e.g. based on Gaussian processes. The present work focuses on sample-based estimation procedures for T_j and tries different avenues to achieve an algorithmic improvement over the designs proposed in the existing best practices. We conclude that some proposed sample-based improvements found in the literature do not work as claimed, and that improving on the existing best practice is indeed fraught with difficulties. We motivate our conclusions introducing the concepts of explorativity and efficiency of the design., Comment: 26 pages