Search

Background: Over the past few decades, numerous forecasting methods have been proposed in the field of epidemic forecasting. Such methods can be classified into different categories such as deterministic vs. probabilistic, comparative methods vs. generative methods, and so on. In some of the more popular comparative methods, researchers compare observed epidemiological data from the early stages of an outbreak with the output of proposed models to forecast the future trend and prevalence of the pandemic. A significant problem in this area is the lack of standard well-defined evaluation measures to select the best algorithm among different ones, as well as for selecting the best possible configuration for a particular algorithm.Results: In this paper we present an evaluation framework which allows for combining different features, error measures, and ranking schema to evaluate forecasts. We describe the various epidemic features (Epi-features) included to characterize the output of forecasting methods and provide suitable error measures that could be used to evaluate the accuracy of the methods with respect to these Epi-features. We focus on long-term predictions rather than short-term forecasting and demonstrate the utility of the framework by evaluating six forecasting methods for predicting influenza in the United States. Our results demonstrate that different error measures lead to different rankings even for a single Epi-feature. Further, our experimental analyses show that no single method dominates the rest in predicting all Epi-features when evaluated across error measures. As an alternative, we provide various Consensus Ranking schema that summarize individual rankings, thus accounting for different error measures. Since each Epi-feature presents a different aspect of the epidemic, multiple methods need to be combined to provide a comprehensive forecast. Thus we call for a more nuanced approach while evaluating epidemic forecasts and we believe that a comprehensive evaluation framework, as presented in this paper, will add value to the computational epidemiology community. [ABSTRACT FROM AUTHOR]

This paper evaluates, for the first time, flash-flood guidance (FFG) values and recently developed gridded FFG (GFFG) used by the National Weather Service (NWS) to monitor and predict imminent flash flooding, which is the leading storm-related cause of death in the United States. It is envisioned that results from this study will be used 1) to establish benchmark performance of existing operational flash-flood prediction tools and 2) to provide information to NWS forecasters that reveals how the existing tools can be readily optimized. Sources used to evaluate the products include official reports of flash floods from the NWS Storm Data database, discharge measurements on small basins available from the U.S. Geological Survey, and witness reports of flash flooding collected during the Severe Hazards Analysis and Verification Experiment. Results indicated that the operational guidance values, with no calibration, were marginally skillful, with the highest critical success index of 0.20 occurring with 3-h GFFG. The false-alarm rates fell and the skill improved to 0.34 when the rainfall was first spatially averaged within basins and then reached 50%% of FFG for 1-h accumulation and exceeded 3-h FFG. Although the skill of the GFFG values was generally lower than that of their FFG counterparts, GFFG was capable of detecting the spatial variability of reported flash flooding better than FFG was for a case study in an urban setting. [ABSTRACT FROM AUTHOR]

This paper describes methods for automatically analyzing formal, state-based requirements specifications for some aspects of completeness and consistency. The approach uses a low-level functional formalism, simplifying the analysis process. State-space explosion problems are eliminated by applying the analysis at a high level of abstraction; i.e., instead of generating a reachability graph for analysis, the analysis is performed directly on the model. The method scales up to large systems by decomposing the specification into smaller, analyzable parts and then using functional composition rules to ensure that verified properties hold for the entire specification. The analysis algorithms and tools have been validated on TCAS II, a complex, airborne, collision-avoidance system required on all commercial aircraft with more than 30 passengers that fly in U.S. airspace. [ABSTRACT FROM AUTHOR]