Search

Input model bias is the bias found in the output performance measures of a simulation model caused by estimating the input distributions/processes used to drive it. When the simulation response is a nonlinear function of its inputs, as is usually the case when simulating complex systems, input modelling bias is amongst the errors that arise. In this paper, we introduce a method that recalibrates the input parameters of parametric input models to reduce the bias in the simulation output. The proposed method is based on sequential quadratic programming with a closed form analytical solution at each step. An algorithm with guidance on how to practically implement the method is presented. The method is shown to be successful in reducing input modelling bias and the total mean squared error caused by input modelling error. Summary of Contribution: This paper furthers the understanding and treatment of input modelling error in computer simulation. We provide a novel method for reducing input model bias by recalibrating the input parameters used to drive a simulation model. A sequential quadratic programming approach with an explicit solution is provided to recalibrate the input parameters. The method is therefore computationally inexpensive. An algorithm outlining our proposed procedure is provided within the paper. An evaluation of the method shows the method successfully reduces input model bias and may also reduce the mean squared error caused by input modelling in the output of a simulation model. [ABSTRACT FROM AUTHOR]

Kriging or Gaussian process models are popular metamodels (surrogate models or emulators) of simulation models; these metamodels give predictors for input combinations that are not simulated. To validate these metamodels for computationally expensive simulation models, the analysts often apply computationally efficient cross-validation. In this paper, we derive new statistical tests for so-called leave-one-out cross-validation. Graphically, we present these tests as scatterplots augmented with confidence intervals that use the estimated variances of the Kriging predictors. To estimate the true variances of these predictors, we might use bootstrapping. Like other statistical tests, our tests—with or without bootstrapping—have type I and type II error probabilities; to estimate these probabilities, we use Monte Carlo experiments. We also use such experiments to investigate statistical convergence. To illustrate the application of our tests, we use (i) an example with two inputs and (ii) the popular borehole example with eight inputs. Summary of Contribution: Simulation models are very popular in operations research (OR) and are also known as computer simulations or computer experiments. A popular topic is design and analysis of computer experiments. This paper focuses on Kriging methods and cross-validation methods applied to simulation models; these methods and models are often applied in OR. More specifically, the paper provides the following; (1) the basic variant of a new statistical test for leave-one–out cross-validation; (2) a bootstrap method for the estimation of the true variance of the Kriging predictor; and (3) Monte Carlo experiments for the evaluation of the consistency of the Kriging predictor, the convergence of the Studentized prediction error to the standard normal variable, and the convergence of the expected experimentwise type I error rate to the prespecified nominal value. The new statistical test is illustrated through examples, including the popular borehole model. [ABSTRACT FROM AUTHOR]