Search

The quality of morphodynamic predictions is generally expressed by an overall grid-point based skill score, which measures the relative accuracy of a morphological prediction over a prediction of zero morphological change, using the mean-squared error (MSE) as the accuracy measure. Through a generic ranking for morphodynamic model predictions, this MSE based skill score (MSESS) aims at making model performance comparable across different prediction situations (geographical locations, forcing conditions, time periods, internal dynamics). The implicit assumptions underlying this approach are that the MSE is an appropriate measure of correspondence for morphological predictions and that the accuracy of the initial bed as the reference correctly reflects the inherent difficulty or ease of prediction situations. This paper presents a thorough analysis of the perception of model skill through the MSE skill score. Using synthetic examples, an example from literature and a long-yearly Delft3D model simulation, we demonstrate that unexpected skill may be reported due to a violation of either of the above assumptions. It is shown that the accuracy of the reference fails to reflect the relative difficulty of prediction situations with a different morphological development prior to the evaluation time (for instance trend, cyclic/seasonal, episodic, speed of the development). We further demonstrate that the MSESS tends to favor model results that underestimate the variance of cumulative bed changes, a feature inherited from the MSE. As a consequence of these limitations, the MSESS may report a relative ranking of predictions not matching the intuitive judgment of experts. Guidelines are suggested for how to adjust calibration and validation procedures to be more in line with a morphologist's expert judgment. [ABSTRACT FROM AUTHOR]

Abstract: Evaluating the performance of numerical models of coastal morphology against observations is an essential part of establishing their credibility. In the past, this has usually been done by comparing predicted with observed behavior, and the modeler making a subjective judgement of the goodness of fit. By contrast, meteorological models have been tested for many years against observations using objective, scientifically rigorous methods. Drawing on the analogy between distributions of atmospheric pressure (and similar scalars) and coastal bathymetry, the meteorological statistical methods are here adapted to coastal applications. A set of criteria is proposed that identify the desirable attributes of statistical measures of the performance of coastal morphological models. Various statistical parameters used in the meteorological field are described, dealing both with cases in which the model outputs comprise a small number of categories (e.g., advance/equilibrium/retreat of shoreline), and those in which they are continuous (e.g., bathymetry). Examples of the application of these methods are given, in terms of both hypothetical illustrations, and real field data and model predictions. Following meteorological practice, it is shown that measuring the skill of a model (i.e., its performance relative to a simple baseline predictor) is a more critical test than measuring its absolute accuracy. The attributes of the different performance measures are compared with the proposed desirable criteria, and those that match them best are selected. These are the LEPSOB test for categorical data, and the Brier Skill Score (BSS) for continuous data. Routine use of these measures of performance would aid inter-comparability of models, and be a step toward strengthening user confidence in the predictions of coastal numerical models. [Copyright &y& Elsevier]