Search

Recent studies have shown that reducing the precision of floating‐point calculations in an atmospheric model can improve the model's computational performance without affecting model fidelity, but code changes are needed to accommodate lower precision or to prevent undue round‐off error. For complex and computationally expensive systems like the Energy Exascale Earth System Model, a method is needed to objectively and efficiently assess the quality of the lower‐precision simulations and to quickly identify problematic code pieces. This paper demonstrates that the accuracy of the computed solution can be evaluated through a simple and quantitative error metric based on time step convergence. The proposed test method can unambiguously detect the accumulation of precision error and objectively assess its impact on solution accuracy. Furthermore, since fast physical processes are known to affect key features of the multiyear mean climate in atmospheric models, we show that the convergence test applied to short simulations can provide useful information about the impact of reduced precision on a model's long‐term behavior. In contrast, the traditional method of climate model evaluation requires multiple years of simulations and involves inspecting many physical quantities for statistical significance in the presence of natural variability. The simplicity, computational efficiency, and objectivity of the proposed method makes it very attractive for high‐resolution model development. The method is also expected to be applicable to other models that numerically solve time evolution equations.

Convergence testing is a common practice in the development of dynamical cores of atmospheric models but is not as often exercised for the parameterization of subgrid physics. An earlier study revealed that the stratiform cloud parameterizations in several predecessors of the Energy Exascale Earth System Model (E3SM) showed strong time step sensitivity and slower‐than‐expected convergence when the model's time step was systematically refined. In this work, a simplified atmosphere model is configured that consists of the spectral‐element dynamical core of the E3SM atmosphere model coupled with a large‐scale condensation parameterization based on commonly used assumptions. This simplified model also resembles E3SM and its predecessors in the numerical implementation of process coupling and shows poor time step convergence in short ensemble tests. We present a formal error analysis to reveal the expected time step convergence rate and the conditions for obtaining such convergence. Numerical experiments are conducted to investigate the root causes of convergence problems. We show that revisions in the process coupling and closure assumption help to improve convergence in short simulations using the simplified model; the same revisions applied to a full atmosphere model lead to significant changes in the simulated long‐term climate. This work demonstrates that causes of convergence issues in atmospheric simulations can be understood by combining analyses from physical and mathematical perspectives. Addressing convergence issues can help to obtain a discrete model that is more consistent with the intended representation of the physical phenomena.

Global atmospheric models seek to capture physical phenomena across a wide range of time and length scales. For this to be a feasible task, the physical processes with time or length scales below that of a computational time step or grid cell size are simplified as one or more parameterizations. Inadvertent oversimplification can violate constraints or destroy relationships in the original physical system and consequently lead to unexpected and physically invalid behavior. An example of such a problem has been investigated in the work of Wan et al. (2020, https://doi.org/10.1029/2019MS001982). This work addresses the issues at a more fundamental level by revisiting the parameterization derivation. A derivation of an unaveraged condensation rate in the unaveraged equations, sometimes referred to as subgrid equations, provides a clear description and more accurate quantification of the condensation/evaporation processes associated with cloud growth/decay, while avoiding simplifications used in earlier studies. A subgrid reconstruction (SGR) methodology is used to connect the unaveraged condensation rate with the grid cell averaged equations solved by the global model. Analyses of the SGR method and the numerical results provide insights into root causes of inconsistent discrete formulations and nonphysical behavior. It is also shown that the SGR methodology provides a flexible framework for addressing such inconsistencies. This work serves as a demonstration that when nonphysical behavior in a parameterization of subgrid variability is avoided through rigorous mathematical derivation, the resulting formulation can exhibit both better numerical convergence properties and significant impact on long‐term climate.

Transient simulation of power systems can be challenging. As models become more complex, applications require acceleration or faster than real time operations to assist system operators, and new methods of solving the problem are required. One potential approach includes time-parallel multigrid reduction methods that distribute the simulation time domain onto multiple compute units and solve the system equations with a parallel, iterative nonlinear multigrid scheme. Achieving significant speedup via these methods for real world power system problems requires examination of how the different features and characteristics of power system simulations interact with the parallel-in-time calculations and how they can be handled by such tools. One such characteristic is the use of unscheduled events that change the system at solution-dependent times. Examples of such events include equipment limits, governor deadbands, voltage regulator adjustments, or protection system actions. In sequential simulations, such events are detected by using root-finding methods or directly in control system models. In time-parallel methods these roots must be handled in parallel. In this paper we propose a method for addressing unscheduled events in time-parallel calculations for power system transient simulation.

As part of the Exascale Computing Project (ECP), a recent focus of development efforts for the SUite of Nonlinear and DIfferential/ALgebraic equation Solvers (SUNDIALS) has been to enable GPU-accelerated time integration in scientific applications at extreme scales. This effort has resulted in several new GPU-enabled implementations of core SUNDIALS data structures, support for programming paradigms which are aware of the heterogeneous architectures, and the introduction of utilities to provide new points of flexibility. In this paper, we discuss our considerations, both internal and external, when designing these new features and present the features themselves. We also present performance results for several of the features on the Summit supercomputer and early access hardware for the Frontier supercomputer, which demonstrate negligible performance overhead resulting from the additional infrastructure and significant speedups when using both NVIDIA and AMD GPUs.

The efficient simulation of non-hydrostatic atmospheric dynamics requires time integration methods capable of overcoming the explicit stability constraints on time step size arising from acoustic waves. In this work, we investigate various implicit–explicit (IMEX) additive Runge–Kutta (ARK) methods for evolving acoustic waves implicitly to enable larger time step sizes in a global non-hydrostatic atmospheric model. The IMEX formulations considered include horizontally explicit – vertically implicit (HEVI) approaches as well as splittings that treat some horizontal dynamics implicitly. In each case, the impact of solving nonlinear systems in each implicit ARK stage in a linearly implicit fashion is also explored.The accuracy and efficiency of the IMEX splittings, ARK methods, and solver options are evaluated on a gravity wave and baroclinic wave test case. HEVI splittings that treat some vertical dynamics explicitly do not show a benefit in solution quality or run time over the most implicit HEVI formulation. While splittings that implicitly evolve some horizontal dynamics increase the maximum stable step size of a method, the gains are insufficient to overcome the additional cost of solving a globally coupled system. Solving implicit stage systems in a linearly implicit manner limits the solver cost but this is offset by a reduction in step size to achieve the desired accuracy for some methods. Overall, the third-order ARS343 and ARK324 methods performed the best, followed by the second-order ARS232 and ARK232 methods.

Explicit Runge–Kutta methods and implicit multistep methods utilizing a Newton–Krylov nonlinear solver are evaluated for a range of configurations of the shallow-water dynamical core of the spectral element community atmosphere model to evaluate their computational performance. These configurations are designed to explore the attributes of each method under different but relevant model usage scenarios including varied spectral order within an element, static regional refinement, and scaling to the largest problem sizes. This analysis is performed within the shallow-water dynamical core option of a full climate model code base to enable a wealth of simulations for study, with the aim of informing solver development within the more complete hydrostatic dynamical core used for climate research. The limitations and benefits to using explicit versus implicit methods, with different parameters and settings, are discussed in light of the trade-offs with Message Passing Interface (MPI) communication and memory and their inherent efficiency bottlenecks. Given the performance behavior across the configurations analyzed here, the recommendation for future work using the implicit solvers is conditional based on scale separation and the stiffness of the problem. For the regionally refined configurations, the implicit method has about the same efficiency as the explicit method, without considering efficiency gains from a preconditioner. The potential for improvement using a preconditioner is greatest for higher spectral order configurations, where more work is shifted to the linear solver. Initial simulations with OpenACC directives to utilize a Graphics Processing Unit (GPU) when performing function evaluations show improvements locally, and that overall gains are possible with adjustments to data exchanges.

Productivity from day one on supercomputers that leverage new technologies requires significant preparation. An institution that procures a novel system architecture often lacks sufficient institutional knowledge and skills to prepare for it. Thus, the "Center of Excellence" (CoE) concept has emerged to prepare for systems such as Summit and Sierra, currently the top two systems in the Top 500. This paper documents CoE experiences that prepared a workload of diverse applications and math libraries for a heterogeneous system. We describe our approach to this preparation, including our management and execution strategies, and detail our experiences with and reasons for using different programming approaches. Our early science and performance results show that the project enabled significant early seismic science with up to a l4X throughput increase over Cori. In addition to our successes, we discuss our challenges and failures so others may benefit from our experience.

In this report we document performance test results on a SUNDIALS-based multiphysics demonstration application. We aim to assess the large-scale parallel performance of new capabilities that have been added to the SUNDIALS suite of time integrators and nonlinear solvers in recent years under funding from both the Exascale Computing Project (ECP) and the Scientific Discovery through Advanced Scientific (SciDAC) program, specifically: (a) SUNDIALS' new MPIManyVector module, that allows extreme flexibility in how a solution "vector" is staged on computational resources, (b) ARKode's new multirate integration module, MRIStep, allowing high-order accurate calculations that subcycle "fast" processes within "slow" ones, (c) SUNDIALS' new flexible linear solver interfaces, that allow streamlined specification of problem-specific linear solvers, and (d) SUNDIALS' new N_Vector additions of "fused" vector operations (to increase arithmetic intensity) and separation of reduction operations into "local" and "global" versions (to reduce latency by combining multiple reductions into a single MPI_Allreduce call). We anticipate that subsequent reports will extend this work to investigate a variety of other new features, including SUNDIALS' generic SUNNonlinearSolver interface and accelerator-enabled N_Vector modules, and upcoming MRIStep extensions to support custom "fast" integrators (that leverage problem structure) and IMEX integration of the "slow" time scale (to add diffusion).

Surface and subsurface flow constitute a naturally linked hydrologic continuum that has not traditionally been simulated in an integrated fashion. Recognizing the interactions between these systems has encouraged the development of integrated hydrologic models (IHMs) capable of treating surface and subsurface systems as a single integrated resource. IHMs is dynamically evolving with improvement in technology and the extent of their current capabilities are often only known to the developers and not general users. This article provides an overview of the core functionality, capability, applications, and ongoing development of one open-source IHM, ParFlow. ParFlow is a parallel, integrated, hydrologic model that simulates surface and subsurface flows. ParFlow solves Richards’ equation for three-dimensional variably saturated groundwater flow and the two-dimensional kinematic wave approximation of the shallow water equations for overland flow. The model employs a conservative centered finite difference scheme and a conservative finite volume method for subsurface flow and transport, respectively. ParFlow uses multigrid preconditioned Krylov and Newton-Krylov methods to solve the linear and nonlinear systems within each time step of the flow simulations. The code has demonstrated very efficient parallel solution capabilities. ParFlow has been coupled to geochemical reaction, land surface (e.g. Common Land Model), and atmospheric models to study the interactions among the subsurface, land surface, and the atmosphere systems across different spatial scales. This overview focuses on the current capabilities of the code, the core simulation engine, and the primary couplings of the subsurface model to other codes, taking a high-level perspective.