Search

A sensitivity analysis methodology recently developed by the authors is applied to COAMPS and WRF. The method involves varying model parameters according to Latin Hypercube Sampling, and developing multivariate multiple regression models that map the model parameters to forecasts over a spatial domain. The regression coefficients and p values testing whether the coefficients are zero serve as measures of sensitivity of forecasts with respect to model parameters. Nine model parameters are selected from COAMPS and WRF, and their impact is examined on nine forecast quantities (water vapor, convective and gridscale precipitation, and air temperature and wind speed at three altitudes). Although the conclusions depend on the model parameters and specific forecast quantities, it is shown that sensitivity to model parameters is often accompanied by nontrivial spatial structure, which itself depends on the underlying forecast model (i.e., COAMPS vs WRF). One specific difference between these models is in their sensitivity with respect to a parameter that controls temperature increments in the Kain–Fritsch trigger function; whereas this parameter has a distinct spatial structure in COAMPS, that structure is completely absent in WRF. The differences between COAMPS and WRF also extend to the quality of the statistical models used to assess sensitivity; specifically, the differences are largest over the waters off the southeastern coast of the United States. The implication of these findings is twofold: not only is the spatial structure of sensitivities different between COAMPS and WRF, the underlying relationship between the model parameters and the forecasts is also different between the two models.

A methodology is proposed for examining the effect of model parameters (assumed to be continuous) on the spatial structure of forecasts. The methodology involves several statistical methods of sampling and inference to assure the sensitivity results are statistically sound. Specifically, Latin hypercube sampling is employed to vary the model parameters, and multivariate multiple regression is used to account for spatial correlations in assessing the sensitivities. The end product is a geographic “map” of p values for each model parameter, allowing one to display and examine the spatial structure of the sensitivity. As an illustration, the effect of 11 model parameters in a mesoscale model on forecasts of convective and grid-scale precipitation, surface air temperature, and water vapor is studied. A number of spatial patterns in sensitivity are found. For example, a parameter that controls the fraction of available convective clouds and precipitation fed back to the grid scale influences precipitation forecasts mostly over the southeastern region of the domain; another parameter that modifies the surface fluxes distinguishes between precipitation forecasts over land and over water. The sensitivity of surface air temperature and water vapor forecasts also has distinct spatial patterns, with the specific pattern depending on the model parameter. Among the 11 parameters examined, there is one (an autoconversion factor in the microphysics) that appears to have no influence in any region and on any of the forecast quantities.

The United States has had three operational numerical weather prediction centers since the Joint Numerical Weather Prediction Unit was closed in 1958. This led to separate paths for U.S. numerical weather prediction, research, technology, and operations, resulting in multiple community calls for better coordination. Since 2006, the three operational organizations—the U.S. Air Force, the U.S. Navy, and the National Weather Service—and, more recently, the Department of Energy, the National Aeronautics and Space Administration, the National Science Foundation, and the National Oceanic and Atmospheric Administration/Office of Oceanic and Atmospheric Research, have been working to increase coordination. This increasingly successful effort has resulted in the establishment of a National Earth System Prediction Capability (National ESPC) office with responsibility to further interagency coordination and collaboration. It has also resulted in sharing of data through an operational global ensemble, common software standards, and model components among the agencies. This article discusses the drivers, the progress, and the future of interagency collaboration.

Knowledge of the relationship between model parameters and forecast quantities is useful because it can aid in setting the values of the former for the purpose of having a desired effect on the latter. Here it is proposed that a well-established multivariate statistical method known as canonical correlation analysis can be formulated to gauge the strength of that relationship. The method is applied to several model parameters in the Coupled Ocean–Atmosphere Mesoscale Prediction System (COAMPS) for the purpose of “controlling” three forecast quantities: 1) convective precipitation, 2) stable precipitation, and 3) snow. It is shown that the model parameters employed here can be set to affect the sum, and the difference between convective and stable precipitation, while keeping snow mostly constant; a different combination of model parameters is shown to mostly affect the difference between stable precipitation and snow, with minimal effect on convective precipitation. In short, the proposed method cannot only capture the complex relationship between model parameters and forecast quantities, it can also be utilized to optimally control certain combinations of the latter.

Numerical weather prediction models have a number of parameters whose values are either estimated from empirical data or theoretical calculations. These values are usually then optimized according to some criterion (e.g., minimizing a cost function) in order to obtain superior prediction. To that end, it is useful to know which parameters have an effect on a given forecast quantity, and which do not. Here the authors demonstrate a variance-based sensitivity analysis involving 11 parameters in the Coupled Ocean–Atmosphere Mesoscale Prediction System (COAMPS). Several forecast quantities are examined: 24-h accumulated 1) convective precipitation, 2) stable precipitation, 3) total precipitation, and 4) snow. The analysis is based on 36 days of 24-h forecasts between 1 January and 4 July 2009. Regarding convective precipitation, not surprisingly, the most influential parameter is found to be the fraction of available precipitation in the Kain–Fritsch cumulus parameterization fed back to the grid scale. Stable and total precipitation are most affected by a linear factor that multiplies the surface fluxes; and the parameter that most affects accumulated snow is the microphysics slope intercept parameter for snow. Furthermore, all of the interactions between the parameters are found to be either exceedingly small or have too much variability (across days and/or parameter values) to be of primary concern.

No abstract available.

Three spatial verification techniques are applied to three datasets. The datasets consist of a mixture of real and artificial forecasts, and corresponding observations, designed to aid in better understanding the effects of global (i.e., across the entire field) displacement and intensity errors. The three verification techniques, each based on well-known statistical methods, have little in common and, so, present different facets of forecast quality. It is shown that a verification method based on cluster analysis can identify “objects” in a forecast and an observation field, thereby allowing for object-oriented verification in the sense that it considers displacement, missed forecasts, and false alarms. A second method compares the observed and forecast fields, not in terms of the objects within them, but in terms of the covariance structure of the fields, as summarized by their variogram. The last method addresses the agreement between the two fields by inferring the function that maps one to the other. The map—generally called optical flow—provides a (visual) summary of the “difference” between the two fields. A further summary measure of that map is found to yield useful information on the distortion error in the forecasts.

The verification of a gridded forecast field, for example, one produced by numerical weather prediction (NWP) models, cannot be performed on a gridpoint-by-gridpoint basis; that type of approach would ignore the spatial structures present in both forecast and observation fields, leading to misinformative or noninformative verification results. A variety of methods have been proposed to acknowledge the spatial structure of the fields. Here, a method is examined that compares the two fields in terms of their variograms. Two types of variograms are examined: one examines correlation on different spatial scales and is a measure of texture; the other type of variogram is additionally sensitive to the size and location of objects in a field and can assess size and location errors. Using these variograms, the forecasts of three NWP model formulations are compared with observations/analysis, on a dataset consisting of 30 days in spring 2005. It is found that within statistical uncertainty the three formulations are comparable with one another in terms of forecasting the spatial structure of observed reflectivity fields. None, however, produce the observed structure across all scales, and all tend to overforecast the spatial extent and also forecast a smoother precipitation (reflectivity) field. A finer comparison suggests that the University of Oklahoma 2-km resolution Advanced Research Weather Research and Forecasting (WRF-ARW) model and the National Center for Atmospheric Research (NCAR) 4-km resolution WRF-ARW slightly outperform the 4.5-km WRF-Nonhydrostatic Mesoscale Model (NMM), developed by the National Oceanic and Atmospheric Administration/National Centers for Environmental Prediction (NOAA/NCEP), in terms of producing forecasts whose spatial structures are closer to that of the observed field.

Recently, an object-oriented verification scheme was developed for assessing errors in forecasts of spatial fields. The main goal of the scheme was to allow the automatic and objective evaluation of a large number of forecasts. However, processing speed was an obstacle. Here, it is shown that the methodology can be revised to increase efficiency, allowing for the evaluation of 32 days of reflectivity forecasts from three different mesoscale numerical weather prediction model formulations. It is demonstrated that the methodology can address not only spatial errors, but also intensity and timing errors. The results of the verification are compared with those performed by a human expert. For the case when the analysis involves only spatial information (and not intensity), although there exist variations from day to day, it is found that the three model formulations perform comparably, over the 32 days examined and across a wide range of spatial scales. However, the higher-resolution model formulation appears to have a slight edge over the other two; the statistical significance of that conclusion is weak but nontrivial. When intensity is included in the analysis, it is found that these conclusions are generally unaffected. As for timing errors, although for specific dates a model may have different timing errors on different spatial scales, over the 32-day period the three models are mostly “on time.” Moreover, although the method is nonsubjective, its results are shown to be consistent with an expert’s analysis of the 32 forecasts. This conclusion is tentative because of the focused nature of the data, spanning only one season in one year. But the proposed methodology now allows for the verification of many more forecasts.

In a recent paper, a statistical method referred to as cluster analysis was employed to identify clusters in forecast and observed fields. Further criteria were also proposed for matching the identified clusters in one field with those in the other. As such, the proposed methodology was designed to perform an automated form of what has been called object-oriented verification. Herein, a variation of that methodology is proposed that effectively avoids (or simplifies) the criteria for matching the objects. The basic idea is to perform cluster analysis on the combined set of observations and forecasts, rather than on the individual fields separately. This method will be referred to as combinative cluster analysis (CCA). CCA naturally lends itself to the computation of false alarms, hits, and misses, and therefore, to the critical success index (CSI). A desirable feature of the previous method—the ability to assess performance on different spatial scales—is maintained. The method is demonstrated on reflectivity data and corresponding forecasts for three dates using three mesoscale numerical weather prediction model formulations—the NCEP/NWS Nonhydrostatic Mesoscale Model (NMM) at 4-km resolution (nmm4), the University of Oklahoma’s Center for Analysis and Prediction of Storms (CAPS) Weather Research and Forecasting Model (WRF) at 2-km resolution (arw2), and the NCAR WRF at 4-km resolution (arw4). In the small demonstration sample herein, model forecast quality is efficiently differentiated when performance is assessed in terms of the CSI. In this sample, arw2 appears to outperform the other two model formulations across all scales when the cluster analysis is performed in the space of spatial coordinates and reflectivity. However, when the analysis is performed only on spatial data (i.e., when only the spatial placement of the reflectivity is assessed), the difference is not significant. This result has been verified both visually and using a standard gridpoint verification, and seems to provide a reasonable assessment of model performance. This demonstration of CCA indicates promise in quickly evaluating mesoscale model performance while avoiding the subjectivity and labor intensiveness of human evaluation or the pitfalls of non-object-oriented automated verification.

A statistical method referred to as cluster analysis is employed to identify features in forecast and observation fields. These features qualify as natural candidates for events or objects in terms of which verification can be performed. The methodology is introduced and illustrated on synthetic and real quantitative precipitation data. First, it is shown that the method correctly identifies clusters that are in agreement with what most experts might interpret as features or objects in the field. Then, it is shown that the verification of the forecasts can be performed within an event-based framework, with the events identified as the clusters. The number of clusters in a field is interpreted as a measure of scale, and the final “product” of the methodology is an “error surface” representing the error in the forecasts as a function of the number of clusters in the forecast and observation fields. This allows for the examination of forecast error as a function of scale.

Statistical postprocessing methods have been successful in correcting many defects inherent in numerical weather prediction model forecasts. Among them, model output statistics (MOS) and perfect prog have been most common, each with its own strengths and weaknesses. Here, an alternative method (called RAN) is examined that combines the two, while at the same time utilizes the information in reanalysis data. The three methods are examined from a purely formal/mathematical point of view. The results suggest that whereas MOS is expected to outperform perfect prog and RAN in terms of mean squared error, bias, and error variance, the RAN approach is expected to yield more certain and bias-free forecasts. It is suggested therefore that a real-time RAN-based postprocessor be developed for further testing.

Conventional measures of model fit for indexed data (e.g., time series or spatial data) summarize errors in y, for instance by integrating (or summing) the squared difference between predicted and measured values over a range of x. We propose an approach which recognizes that errors can occur in the x-direction as well. Instead of just measuring the difference between the predictions and observations at each site (or time), we first "deform" the predictions, stretching or compressing along the x-direction or directions, so as to improve the agreement between the observations and the deformed predictions. Error is then summarized by (a) the amount of deformation in x, and (b) the remaining difference in y between the data and the deformed predictions (i.e., the residual error in y after the deformation). A parameter, lambda, controls the tradeoff between (a) and (b), so that as lambda-->infinity no deformation is allowed, whereas for lambda=0 the deformation minimizes the errors in y. In some applications, the deformation itself is of interest because it characterizes the (temporal or spatial) structure of the errors. The optimal deformation can be computed by solving a system of nonlinear partial differential equations, or, for a unidimensional index, by using a dynamic programming algorithm. We illustrate the procedure with examples from nonlinear time series and fluid dynamics.

The Earth System Prediction Suite (ESPS) is a collection of flagship U.S. weather and climate models and model components that are being instrumented to conform to interoperability conventions, documented to follow metadata standards, and made available either under open-source terms or to credentialed users. The ESPS represents a culmination of efforts to create a common Earth system model architecture, and the advent of increasingly coordinated model development activities in the United States. ESPS component interfaces are based on the Earth System Modeling Framework (ESMF), community-developed software for building and coupling models, and the National Unified Operational Prediction Capability (NUOPC) Layer, a set of ESMF-based component templates and interoperability conventions. This shared infrastructure simplifies the process of model coupling by guaranteeing that components conform to a set of technical and semantic behaviors. The ESPS encourages distributed, multiagency development of coupled modeling systems; controlled experimentation and testing; and exploration of novel model configurations, such as those motivated by research involving managed and interactive ensembles. ESPS codes include the Navy Global Environmental Model (NAVGEM), the Hybrid Coordinate Ocean Model (HYCOM), and the Coupled Ocean–Atmosphere Mesoscale Prediction System (COAMPS); the NOAA Environmental Modeling System (NEMS) and the Modular Ocean Model (MOM); the Community Earth System Model (CESM); and the NASA ModelE climate model and the Goddard Earth Observing System Model, version 5 (GEOS-5), atmospheric general circulation model.

The Earth System Prediction Capability (ESPC) inter-agency program was established in 2010 as a coordinating effort to improve collaboration across the federally sponsored environmental research and operational prediction communities for the development and implementation of improved national physical earth system prediction. Towards these goals, a set of five demonstration projects are under development and researchers are invited to participate in the definition and execution of the projects. The goal of the demonstrations is to provide unifying themes and common diagnostics to develop a common modeling environment, establish community data sets & test cases, assess predictability at sub-seasonal to inter-annual timescales, and begin to develop guidelines for the future transition to operational forecasts.

All numerical models (e.g., Numerical Weather Prediction models) have certain parameters within model algorithms which effect forecasts to a different degree, depending on the forecast quantity. The specific values of these model parameters are determined either theoretically using fundamental physics laws but incorporating necessary approximations to reduce computational cost, or empirically using observations from field experiments where observational error introduces uncertainty. In either case, the exact value of the parameter is often unknown a priori, and so their values are usually set to improve forecast quality through some form of forecast verification. Such an approach to model tuning, however, requires knowledge of the observations to which the forecasts must be compared, and therefore, a multitude of highly detailed experimental cases in order to fully resolve parameter values, a data set very difficult to obtain. A knowledge of the relationship between model parameters and forecast quantities, without reference to observations, can not only aid in such an observation-based approach to model tuning, it can also aid in tuning the model parameters according to other criteria that may not be based on observations directly, e.g., a desire to affect the forecasts according to some long-term experience of a forecaster. The main goal of our work has been to develop a framework for representing the complex relationship between model parameters and forecast quantities, without any reference to observations.

During this final year of the Pacific Northwest Regional Collaboratory we focused significantly on continuing the relationship between technical teams and government end-users. The main theme of the year was integration. This took the form of data integration via our web portal and integration of our technologies with the end users. The PNWRC's technical portfolio is based on EOS strategies, and focuses on 'applications of national priority: water management, invasive species, coastal management and ecological forecasting.' The products of our technical approaches have been well received by the community of focused end-users. The objective this year was to broaden that community and develop external support to continue and operationalize product development.

The APL atmospheric sciences group is working to improve forecaster performance at Navy operational weather forecast detachments afloat and ashore. This work encompasses broad research and technology development in areas of visualization, human factors, human-machine interaction, and model and forecast verification with an emphasis on mesoscale ensembles and visualization of uncertainty. The verification effort's long-term goal is to develop an automated, objective verification technique for assessment of very high-resolution mesoscale predictions which accurately accounts for spatially or temporally misplaced features, false alarms and misses (Brown 2002).

Earth System Prediction Capability Workshop Washington, D. C., 21–23 March 2012 A meeting to advance a federal partnership toward operational prediction of the physical environment at subseasonal to decadal time scales was held in Washington, D. C. Scientists, headquarters representatives, and program managers from the Department of Energy, NASA, the National Oceanic and Atmospheric Administration (NOAA), the National Science Foundation, the U.S. Air Force, and the U.S. Navy met to discuss pressing agency requirements for extended-range environmental prediction to inform economic, energy, agricultural, national security, and infrastructure decisions. After significant review and discussion, participants agreed that the highest potential for progress was at the interseasonal to interannual (ISI) time scales (Advancing the Science of Climate Change (2010), Board on Atmospheric Sciences and Climate (BASC), http://www.nap.edu/openbook.php?record_id=12782). They agreed to pursue a joint effort, identifying five areas for near-term demonstrations of predictability and establishing volunteer coordinators to organize the demonstration efforts. The demonstrations will establish operational extended-range predictive skill, inform further research, enhance interagency collaboration, and push forward environmental prediction technical and computational capabilities.