Search

Numerical simulation, and in particular simulation of the earth system, relies on contributions from diverse communities, from those who develop models to those involved in devising, executing, and analysing numerical experiments. Often these people work in different institutions and may be working with significant separation in time (particularly analysts, who may be working on data produced years earlier), and they typically communicate via published information (whether journal papers, technical notes, or websites). The complexity of the models, experiments, and methodologies, along with the diversity (and sometimes inexact nature) of information sources, can easily lead to misinterpretation of what was actually intended or done. In this paper we introduce a taxonomy of terms for more clearly defining numerical experiments, put it in the context of previous work on experimental ontologies, and describe how we have used it to document the experiments of the sixth phase for the Coupled Model Intercomparison Project (CMIP6). We describe how, through iteration with a range of CMIP6 stakeholders, we rationalized multiple sources of information and improved the clarity of experimental definitions. We demonstrate how this process has added value to CMIP6 itself by (a) helping those devising experiments to be clear about their goals and their implementation, (b) making it easier for those executing experiments to know what is intended, (c) exposing interrelationships between experiments, and (d) making it clearer for third parties (data users) to understand the CMIP6 experiments. We conclude with some lessons learnt and how these may be applied to future CMIP phases as well as other modelling campaigns. [ABSTRACT FROM AUTHOR]

Earth system modelling relies on contributions from groups who develop models and from those involved in devising, executing, and exploiting numerical experiments. Often these people work in different institutions, and they may communicate primarily via published information (whether journal papers, technical notes, or websites). The complexity of the models, experiments, and methodologies, along with the diversity (and sometimes inexact nature) of information sources can easily lead to misinterpretation of what was actually intended or done. In this paper we introduce a taxonomy of terms for more clearly defining numerical experiments, put it in the context of previous work on experimental ontologies, and describe how we have used it to document the CMIP6 experiments. We describe how this process involved iteration with a range of CMIP6 stakeholders to rationalise multiple sources of information and add clarity to experimental definitions. We demonstrate how this process has added value to CMIP6 itself by a) helping those devising experiments to be clear about their goals and expected methodology, b) making it easier for those executing experiments to know what was intended, c) exposing inter-relationships between experiments, and d) making it clearer for third parties (data users) to understand the CMIP6 experiments. We conclude with some lessons learned, and how these may be applied for any modelling campaign as well as future CMIP phases. [ABSTRACT FROM AUTHOR]

The ability of data assimilation systems to infer unobserved variables has brought major benefits to atmospheric and oceanographic sciences. Information is transferred from observations to unobserved variables in two ways: through the temporal evolution of the predictive equations (either a forecast model or its adjoint) or through an error covariance matrix (or a parametrized approximation to the error covariance). Here, it is found that high frequency information tends to flow through the former route, low frequency through the latter. It is also noted that using the Kalman Filter analysis to estimate the correlation between the observed and unobserved variables can lead to a biased result because of an error correlation: this error correlation is absent when the Kalman Smoother is used. [ABSTRACT FROM AUTHOR]

In recent years a number of operational meteorological centres have completed multidecadal reanalyses of their observation records using a version of their operational analysis systems. These operational systems aim to approximate the best possible analysis of the atmospheric state at a given time using all observations made prior to that time, and require major resources to produce. Re-analyses are made with the same real-time systems because they can be done as marginal activities on the back of operational efforts. In this paper, we examine some of the salient differences between the use of optimal real-time analyses and optimal retrospective analyses in the context of a simple linear system. In this case, the optimal real-time analysis could be obtained by the Kalman filter. When observations are available both before and after the analysis time the additional information can, in principle, be exploited to improve on the Kalman filter analysis. For linear systems the optimal retrospective analysis is given by the Kalman smoother, which combines a forward and backward Kalman filter. Results comparing these methods are presented which demonstrate the importance of using all the available data for a retrospective analysis. While using all future data is not yet tractable for retrospective meteorological analyses, such techniques are of use for more limited re-analysis. [ABSTRACT FROM AUTHOR]

Observations of winds in mesospheric airglow layers have been made at Mt. John (44°S,170°E), New Zealand for some years. We present a modelling study of airglow emissions which shows that the properties of wind detection based on airglow emission means that high-frequency gravity waves are effectively filtered from the wind spectrum observed. This filtering means that any waves with periods of the order of hours should be detectable in the record (as they will not be hidden in the noise of the higher-frequency waves ubiquitous at these heights). One example of such a wave is shown. As part of the analysis, we show that because the airglow layers differ in width, some waves might be observed in only one airglow layer, even when present in both. [Copyright &y& Elsevier]

The Coupled Model Intercomparison Project (CMIP) is one of the biggest international efforts aimed at better understanding the past, present, and future of climate changes in a multi-model context. A total of 21 model intercomparison projects (MIPs) were endorsed in its sixth phase (CMIP6), which included 190 different experiments that were used to simulate 40 000 years and produced around 40 PB of data in total. This paper presents the main findings obtained from the CPMIP (the Computational Performance Model Intercomparison Project), a collection of a common set of metrics, specifically designed for assessing climate model performance. These metrics were exclusively collected from the production runs of experiments used in CMIP6 and primarily from institutions within the IS-ENES3 consortium. The document presents the full set of CPMIP metrics per institution and experiment, including a detailed analysis and discussion of each of the measurements. During the analysis, we found a positive correlation between the core hours needed, the complexity of the models, and the resolution used. Likewise, we show that between 5 %–15 % of the execution cost is spent in the coupling between independent components, and it only gets worse by increasing the number of resources. From the data, it is clear that queue times have a great impact on the actual speed achieved and have a huge variability across different institutions, ranging from none to up to 78 % execution overhead. Furthermore, our evaluation shows that the estimated carbon footprint of running such big simulations within the IS-ENES3 consortium is 1692 t of CO 2 equivalent. As a result of the collection, we contribute to the creation of a comprehensive database for future community reference, establishing a benchmark for evaluation and facilitating the multi-model, multi-platform comparisons crucial for understanding climate modelling performance. Given the diverse range of applications, configurations, and hardware utilised, further work is required for the standardisation and formulation of general rules. The paper concludes with recommendations for future exercises aimed at addressing the encountered challenges which will facilitate more collections of a similar nature. [ABSTRACT FROM AUTHOR]

For effective comparison of Earth system models in large multi-model ensembles one must first have an understanding of the model design, accurate knowledge of the requirements of experiments and insight into how forcing constraints are implemented.Earth system modelling involves many roles, from those involving model construction, to those involving devising, executing, and exploiting numerical experiments. In most cases these roles involve multiple people, and in many cases these people are in different institutions and may only interact via published information (whether journal papers, technical notes, or websites). These multiple people, sources and methodologies lead to scope for misinterpretation of intent and misunderstanding of what was actually done. We present the ES-DOC documentation methodology and information sharing ecosystem which allows the Earth system modelling community to accurately document and share information about both the design of their models and also about how their models are configured to conform to common forcing protocols.We describe how ES-DOC concepts have been applied to the design phase of CMIP6, improving not only the documentation of CMIP6, but also improving the experiments themselves. As a consequence we believe it will be easier for both the MIP designers and participants to be confident that they have requested, understood and/or executed what was required. [ABSTRACT FROM AUTHOR]

The Centre for Environmental Data Analysis (CEDA) hosts two key services that support environmental science; the CEDA archive and JASMIN. In particular, JASMIN is a state-of-the-art, unique data intensive supercomputer allowing scientists to 'bring their processing to the data'. It provides over 44 Petabytes of storage and 11,000 cores, which sit alongside the curated atmospheric and earth observation CEDA data archive. This is fast becoming essential as environmental data, such as from satellites or climate models, continue to grow too large for local processing. There are over 2000 JASMIN users, from over 160 science projects. We are frequently required to provide evidence of JASMIN's impact in order to receive additional and/or continued funding. Our funders required evidence of impact during 2016, so impact stories were collected from a select number of known users. The process for collecting these was reactive and labour intensive. In 2018, a user survey (n = 520) and six focus groups (n = 26) were undertaken as part of a MSc project; aiming to investigate how impact stories could be collected from CEDA users and propose suggestions as to how processes could be implemented in the future. Key findings and recommendations from this project will be discussed, alongside how we have begun to implement a regular collection process for evidencing impact information. [ABSTRACT FROM AUTHOR]

The Coupled Model Intercomparison Project (CMIP) is one of the biggest international efforts to better understand past, present and future climate changes in a multi-model context. A total of 21 Model Intercomparison Projects (MIPs) were endorsed in its 6th phase (CMIP6), which included 190 different experiments that were used to simulate 40000 years and produced around 40 PB of data in total. This paper shows the main results obtained from the collection of performance metrics done for CMIP6 (CPMIP). The document provides the list of partners involved, the CPMIP metrics per institution/model, and the approach used for the collection and the coordination behind this process. Furthermore, a section has been included to analyze the results and prove the usefulness of the metrics for the community. Moreover, we describe the main difficulties faced during the collection and propose recommendations for future exercises. [ABSTRACT FROM AUTHOR]

The land surface, hydrological, and groundwater modelling communities all have expertise in simulating the hydrological processes at play in the terrestrial component of the Earth system. However, these communities, and the wider Earth system modelling community, have largely remained distinct with limited collaboration between disciplines, hindering progress in the representation of hydrological processes in the land component of Earth system models (ESMs). In order to address key societal questions regarding the future availability of water resources and the intensity of extreme events such as floods and droughts in a changing climate, these communities must come together and build on the strengths of one another to produce next-generation land system models that are able to adequately simulate the terrestrial water cycle under change. The development of a common modelling infrastructure can contribute to stimulating cross-fertilisation by structuring and standardising the interactions. This paper presents such an infrastructure, a land system framework, which targets an intermediate level of complexity and constrains interfaces between components (and communities) and, in doing so, aims to facilitate an easier pipeline between the development of (sub-)community models and their integration, both for standalone use and for use in ESMs. This paper first outlines the conceptual design and technical capabilities of the framework; thereafter, its usage and useful characteristics are demonstrated through case studies. The main innovations presented here are (1) the interfacing constraints themselves; (2) the implementation in Python (the Unified Framework for Hydrology, unifhy); and (3) the demonstration of standalone use cases using the framework. The existing framework does not yet meet all our goals, in particular, of directly supporting integration into larger ESMs, so we conclude with the remaining limitations of the current framework and necessary future developments. [ABSTRACT FROM AUTHOR]

Land surface, hydrological, and groundwater modelling communities all have expertise in simulating the hydrological processes at play in the land system, but these communities have largely remained distinct with limited collaboration between disciplines. In order to address key societal questions regarding the future availability of water resources and the intensity of extreme events such as floods and droughts in a changing climate, these communities must build on the strengths of one another. The development of a common modelling infrastructure, a framework, can contribute to stimulating cross-fertilisation between them. By allowing (parts of) their existing models to be coupled together, improved land system models can be built to better understand and simulate the terrestrial hydrological cycle. This paper presents a Python implementation of such a framework named the Unified Framework for Hydrology (unifhy). The framework aims to provide the technical infrastructure required to couple models, taking into account the specific needs of a land system model. Its conceptual design and technical capabilities are outlined first, before its usage and useful characteristics are demonstrated through case studies. The limitations of the current framework and necessary future developments are finally presented as a road map for later versions and/or other implementations of the framework. [ABSTRACT FROM AUTHOR]

The results of climate models are of increasing and widespread importance. No longer is climate model output of sole interest to climate scientists and researchers in the climate change impacts and adaptation fields. Now nonspecialists such as government officials, policy makers, and the general public all have an increasing need to access climate model output and understand its implications. For this host of users, accurate and complete metadata (i.e., information about how and why the data were produced) is required to document the climate modeling results. Here we describe a pilot community initiative to collect and make available documentation of climate models and their simulations. In an initial application, a metadata repository is being established to provide information of this kind for a major internationally coordinated modeling activity known as CMIP5 (Coupled Model Intercomparison Project, Phase 5). It is expected that for a wide range of stakeholders, this and similar community-managed metadata repositories will spur development of analysis tools that facilitate discovery and exploitation of Earth system simulations. INSETS: THE METAFOR PROJECT;USING THE CMIP5 MODEL AND SIMULATION DESCRIPTION PORTALS. [ABSTRACT FROM AUTHOR]

We describe the approach taken to develop the United Kingdom's first community Earth system model, UKESM1. This is a joint effort involving the Met Office and the Natural Environment Research Council (NERC), representing the U.K. academic community. We document our model development procedure and the subsequent U.K. submission to CMIP6, based on a traceable hierarchy of coupled physical and Earth system models. UKESM1 builds on the well‐established, world‐leading HadGEM models of the physical climate system and incorporates cutting‐edge new representations of aerosols, atmospheric chemistry, terrestrial carbon, and nitrogen cycles and an advanced model of ocean biogeochemistry. A high‐level metric of overall performance shows that both models, HadGEM3‐GC3.1 and UKESM1, perform better than most other CMIP6 models so far submitted for a broad range of variables. We point to much more extensive evaluation performed in other papers in this special issue. The merits of not using any forced climate change simulations within our model development process are discussed. First results from HadGEM3‐GC3.1 and UKESM1 include the emergent climate sensitivity (5.5 and 5.4 K, respectively) which is high relative to the current range of CMIP5 models. The role of cloud microphysics and cloud‐aerosol interactions in driving the climate sensitivity, and the systematic approach taken to understand this role, is highlighted in other papers in this special issue. We place our findings within the broader modeling landscape indicating how our understanding of key processes driving higher sensitivity in the two U.K. models seems to align with results from a number of other CMIP6 models. Plain Language Summary: The United Kingdom has taken a community approach to model development and delivery of simulations to the sixth phase of the Coupled Model Intercomparison Project (CMIP6). The papers in this special issue characterize and evaluate the U.K. models and highlight emerging properties of the models, such as the climate sensitivity. Understanding of the reasons for an increase in sensitivity in these new models is highlighted here, and similarities to our findings from other modeling centers are discussed. Key Points: The United Kingdom has developed its first community Earth system model and delivered a traceable hierarchy of models to CMIP6We applied a process‐based evaluation strategy in model development but chose not to use historic trends or measures of climate responseThe U.K. models exhibit higher climate sensitivity than seen in CMIP5 arising in part from more positive cloud feedbacks [ABSTRACT FROM AUTHOR]

The data request of the Coupled Model Intercomparison Project Phase 6 (CMIP6) defines all the quantities from CMIP6 simulations that should be archived. This includes both quantities of general interest needed from most of the CMIP6-endorsed model intercomparison projects (MIPs) and quantities that are more specialized and only of interest to a single endorsed MIP. The complexity of the data request has increased from the early days of model intercomparisons, as has the data volume. In contrast with CMIP5, CMIP6 requires distinct sets of highly tailored variables to be saved from each of the more than 200 experiments. This places new demands on the data request information base and leads to a new requirement for development of software that facilitates automated interrogation of the request and retrieval of its technical specifications. The building blocks and structure of the CMIP6 Data Request (DREQ), which have been constructed to meet these challenges, are described in this paper. [ABSTRACT FROM AUTHOR]

The data request of the Coupled Model Intercomparison Project Phase 6 (CMIP6) defines all the quantities from CMIP6 simulations that should be archived. This includes both quantities of general interest needed from most of the CMIP6-endorsed Model Intercomparison Projects (MIPs) and quantities that are more specialised and only of interest to a single endorsed MIP. The complexity of the data request has increased from the early days of model intercomparisons, as has the data volume. In contrast with CMIP5, CMIP6 requires distinct sets of highly tailored variables to be saved from each of the more than 200 experiments. This places new demands on the data request information base and results in a new requirement for development of software that facilitates automated interrogation of the request and retrieval of its technical specifications. The building blocks and structure of the CMIP6 Data Request (DREQ) which has been constructed to meet these challenges is described in this paper. [ABSTRACT FROM AUTHOR]

The World Climate Research Programme (WCRP)'s Working Group on Climate Modelling (WGCM) Infrastructure Panel (WIP) was formed in 2014 in response to the explosive growth in size and complexity of Coupled Model Intercomparison Projects (CMIPs) between CMIP3 (2005-2006) and CMIP5 (2011-2012). This article presents the WIP recommendations for the global data infrastructure needed to support CMIP design, future growth, and evolution. Developed in close coordination with those who build and run the existing infrastructure (the Earth System Grid Federation; ESGF), the recommendations are based on several principles beginning with the need to separate requirements, implementation, and operations. Other important principles include the consideration of the diversity of community needs around data - a data ecosystem - the importance of provenance, the need for automation, and the obligation to measure costs and benefits. This paper concentrates on requirements, recognizing the diversity of communities involved (modelers, analysts, software developers, and downstream users). Such requirements include the need for scientific reproducibility and accountability alongside the need to record and track data usage. One key element is to generate a dataset-centric rather than system-centric focus, with an aim to making the infrastructure less prone to systemic failure. With these overarching principles and requirements, the WIP has produced a set of position papers, which are summarized in the latter pages of this document. They provide specifications for managing and delivering model output, including strategies for replication and versioning, licensing, data quality assurance, citation, long-term archiving, and dataset tracking. They also describe a new and more formal approach for specifying what data, and associated metadata, should be saved, which enables future data volumes to be estimated, particularly for well-defined projects such as CMIP6. The paper concludes with a future facing consideration of the global data infrastructure evolution that follows from the blurring of boundaries between climate and weather, and the changing nature of published scientific results in the digital age. [ABSTRACT FROM AUTHOR]

The World Climate Research Programme (WCRP)'s Working Group on Climate Modeling (WGCM) Infrastructure Panel (WIP) was formed in 2014 in response to the explosive growth in size and complexity of Coupled Model Intercomparison Projects (CMIPs) between CMIP3 (2005-06) and CMIP5 (2011-12). This article presents the WIP recommendations for the global data infrastructure needed to support CMIP design, future growth and evolution. Developed in close coordination with those who build and run the existing infrastructure (the Earth System Grid Federation), the recommendations are based on several principles beginning with the need to separate requirements, implementation, and operations. Other important principles include the consideration of data as a commodity in an ecosystem of users, the importance of provenance, the need for automation, and the obligation to measure costs and benefits. This paper concentrates on requirements, recognising the diversity of communities involved (modelers, analysts, software developers, and downstream users). Such requirements include the need for scientific reproducibility and accountability alongside the need to record and track data usage for the purpose of assigning credit. One key element is to generate a dataset-centric rather than system-centric focus, with an aim to making the infrastructure less prone to systemic failure. With these overarching principles and requirements, the WIP has produced a set of position papers, which are summarized here. They provide specifications for managing and delivering model output, including strategies for replication and versioning, licensing, data quality assurance, citation, long-term archival, and dataset tracking. They also describe a new and more formal approach for specifying what data, and associated metadata, should be saved, which enables future data volumes to be estimated. The paper concludes with a future-facing consideration of the global data infrastructure evolution that follows from the blurring of boundaries between climate and weather, and the changing nature of published scientific results in the digital age. [ABSTRACT FROM AUTHOR]

The CF (Climate and Forecast) metadata conventions are designed to promote the creation, processing, and sharing of climate and forecasting data using Network Common Data Form (netCDF) files and libraries. The CF conventions provide a description of the physical meaning of data and of their spatial and temporal properties, but they depend on the netCDF file encoding which can currently only be fully understood and interpreted by someone familiar with the rules and relationships specified in the conventions documentation. To aid in development of CF-compliant software and to capture with a minimal set of elements all of the information contained in the CF conventions, we propose a formal data model for CF which is independent of netCDF and describes all possible CF-compliant data. Because such data will often be analysed and visualised using software based on other data models, we compare our CF data model with the ISO 19123 coverage model, the Open Geospatial Consortium CF netCDF standard, and the Unidata Common Data Model. To demonstrate that this CF data model can in fact be implemented, we present cf-python, a Python software library that conforms to the model and can manipulate any CF-compliant dataset. [ABSTRACT FROM AUTHOR]

The CF (Climate and Forecast) metadata conventions are designed to promote the creation, processing and sharing of climate and forecasting data using Network Common Data Form (netCDF) files and libraries. The CF conventions provide a description of the physical meaning of data and of their spatial and temporal properties, but they depend on the netCDF file encoding which can currently only be fully understood and interpreted by someone familiar with the rules and relationships specified in the conventions documentation. To aid in development of CF-compliant software and to capture with a minimal set of elements all of the information contained in the CF conventions, we propose a formal data model for CF which is independent of netCDF and describes all possible CF-compliant data. Because such data will often be analysed and visualised using software based on other data models, we compare the CF data model with the ISO 19123 coverage model, the Open Geospatial Consortium CF netCDF standard and the Unidata Common Data Model. To demonstrate that the CF data model can in fact be implemented, we present cf-python, a Python software library that conforms to the model and can manipulate any CF-compliant dataset. [ABSTRACT FROM AUTHOR]

A climate model represents a multitude of processes on a variety of timescales and space scales: a canonical example of multi-physics multi-scale modeling. The underlying climate system is physically characterized by sensitive dependence on initial conditions, and natural stochastic variability, so very long integrations are needed to extract signals of climate change. Algorithms generally possess weak scaling and can be I/O and/or memory-bound. Such weak-scaling, I/O, and memory-bound multi-physics codes present particular challenges to computational performance. Traditional metrics of computational efficiency such as performance counters and scaling curves do not tell us enough about real sustained performance from climate models on different machines. They also do not provide a satisfactory basis for comparative information across models. codes present particular challenges to computational performance. We introduce a set of metrics that can be used for the study of computational performance of climate (and Earth system) models. These measures do not require specialized software or specific hardware counters, and should be accessible to anyone. They are independent of platform and underlying parallel programming models. We show how these metrics can be used to measure actually attained performance of Earth system models on different machines, and identify the most fruitful areas of research and development for performance engineering. codes present particular challenges to computational performance. We present results for these measures for a diverse suite of models from several modeling centers, and propose to use these measures as a basis for a CPMIP, a computational performance model intercomparison project (MIP). [ABSTRACT FROM AUTHOR]

The Coupled Model Intercomparison Project (CMIP) has successfully provided the climate community with a rich collection of simulation output from Earth system models (ESMs) that can be used to understand past climate changes and make projections and uncertainty estimates of the future. Confidence in ESMs can be gained because the models are based on physical principles and reproduce many important aspects of observed climate. More research is required to identify the processes that are most responsible for systematic biases and the magnitude and uncertainty of future projections so that more relevant performance tests can be developed. At the same time, there are many aspects of ESM evaluation that are well established and considered an essential part of systematic evaluation but have been implemented ad hoc with little community coordination. Given the diversity and complexity of ESM analysis, we argue that the CMIP community has reached a critical juncture at which many baseline aspects of model evaluation need to be performed much more efficiently and consistently. Here, we provide a perspective and viewpoint on how a more systematic, open, and rapid performance assessment of the large and diverse number of models that will participate in current and future phases of CMIP can be achieved, and announce our intention to implement such a system for CMIP6. Accomplishing this could also free up valuable resources as many scientists are frequently "re-inventing the wheel" by re-writing analysis routines for well-established analysis methods. A more systematic approach for the community would be to develop and apply evaluation tools that are based on the latest scientific knowledge and observational reference, are well suited for routine use, and provide a wide range of diagnostics and performance metrics that comprehensively characterize model behaviour as soon as the output is published to the Earth System Grid Federation (ESGF). The CMIP infrastructure enforces data standards and conventions for model output and documentation accessible via the ESGF, additionally publishing observations (obs4MIPs) and reanalyses (ana4MIPs) for model intercomparison projects using the same data structure and organization as the ESM output. This largely facilitates routine evaluation of the ESMs, but to be able to process the data automatically alongside the ESGF, the infrastructure needs to be extended with processing capabilities at the ESGF data nodes where the evaluation tools can be executed on a routine basis. Efforts are already underway to develop community-based evaluation tools, and we encourage experts to provide additional diagnostic codes that would enhance this capability for CMIP. At the same time, we encourage the community to contribute observations and reanalyses for model evaluation to the obs4MIPs and ana4MIPs archives. The intention is to produce through the ESGF a widely accepted quasi-operational evaluation framework for CMIP6 that would routinely execute a series of standardized evaluation tasks. Over time, as this capability matures, we expect to produce an increasingly systematic characterization of models which, compared with early phases of CMIP, will more quickly and openly identify the strengths and weaknesses of the simulations. This will also reveal whether long-standing model errors remain evident in newer models and will assist modelling groups in improving their models. This framework will be designed to readily incorporate updates, including new observations and additional diagnostics and metrics as they become available from the research community. [ABSTRACT FROM AUTHOR]

The Community Intercomparison Suite (CIS) is an easy-to-use command-line tool which has been developed to allow the straightforward intercomparison of remote sensing, in situ and model data. While there are a number of tools available for working with climate model data, the large diversity of sources (and formats) of remote sensing and in situ measurements necessitated a novel software solution. Developed by a professional software company, CIS supports a large number of gridded and ungridded data sources "out-of-the-box", including climate model output in NetCDF or the UK Met Office pp file format, CloudSat, CALIOP (Cloud-Aerosol Lidar with Orthogonal Polarization), MODIS (MODerate resolution Imaging Spectroradiometer), Cloud and Aerosol CCI (Climate Change Initiative) level 2 satellite data and a number of in situ aircraft and ground station data sets. The open-source architecture also supports user-defined plugins to allow many other sources to be easily added. Many of the key operations required when comparing heterogenous data sets are provided by CIS, including subsetting, aggregating, collocating and plotting the data. Output data are written to CF-compliant NetCDF files to ensure interoperability with other tools and systems. The latest documentation, including a user manual and installation instructions, can be found on our website (http://cistools.net). Here, we describe the need which this tool fulfils, followed by descriptions of its main functionality (as at version 1.4.0) and plugin architecture which make it unique in the field. [ABSTRACT FROM AUTHOR]

The Community Intercomparison Suite (CIS) is an easy-to-use command-line tool which has been developed to allow the straightforward intercomparison of remote sensing, in-situ and model data. While there are a number of tools available for working with climate model data, the large diversity of sources (and formats) of remote sensing and in-situ measurements necessitated a novel software solution. Developed by a professional software company, CIS supports a large number of gridded and ungridded data sources "out-of-the-box", including climate model output in NetCDF or the UK Met Office pp file format, CALIOP (Cloud-Aerosol Lidar with Orthogonal Polarization), MODIS (MODerate resolution Imaging Spectroradiometer), Cloud and Aerosol CCI (Climate Change Initiative) level 2 satellite data, and a number of in-situ aircraft and ground station datasets. The open-source architecture also supports user defined "plugins" to allow many other sources to be easily added. Many of the key operations required when comparing heterogenous datasets are provided by CIS, including subsetting, aggregating, collocating and plotting the data. Output data is written to CF-compliant NetCDF files to ensure interoperability with other tools and systems. The latest documentation, including a user manual and installation instructions can be found on our website (http://cistools.net). Here we describe the need which this tool fulfils, followed by descriptions of its main functionality (as at version 1.3.2) and plugin architecture which make it unique in the field. [ABSTRACT FROM AUTHOR]

The article discusses the highlights of the Infrastructure Project European Network for Earth System Modelling (IS-ENES) Workshop on High-Performance Computing for Climate Models that was held in December 2011 in Lecce, Italy. Topics discussed include a review of the context of European Union (EU) exascale projects and the major advances taking in the U.S. and Europe. The data challenge for climate models is also explored.

IS-ENES is the distributed e-infrastructure of the European Network for Earth System Modelling (ENES).A complex and heterogeneous infrastructure, IS-ENES encompasses models and tools development, documentation, deployment and sharing (including HPC aspects), data storage, processing and distribution as well as the interface with climate impact science and climate services, therefore serving climate science and the users of climate data. This direct impact has a path to many socio-economic sectors (energy, agriculture, transport, water management, health, safety...), as well as national and international policy makers. IS-ENES' third phase (H2020 funded I3 European project H2020 2019-20122) IS-ENES3 starts as the European climate modelling community faces the challenges of contributing to the next IPCC assessment report through the 6th phase of the Coupled Model Intercomparison Project (CMIP). IS-ENES3 addresses these challenges by developing services to deal with unprecedented data volumes and model complexity. The infrastructure delivers the European part of the Earth System Grid Federation and a central point of entry to services providing access to new data, software, models and tools.IS-ENES3 will further stimulate collaboration, disseminate software and data, and integrate and expand Europe wide its user base. Moreover it will find innovative ways of working with the Copernicus programme, already relying on IS-ENES ESGF services, and with other parts of the European data infrastructure, and with the high performance computing and data analytics industries.We propose to employ a blend of indicators to show the advancement of key aspects/activities toward the achievement of the project's manifold short and long-term impacts. Besides relying on indicators used so far to measure software uptake and use and data processing and download, including the reliability of help and documentation services, we plan in IS-ENES3 to add ways to measure contributions to joint development activities (e.g. for new model modules), map applications for Trans National Access and the landscape of national and international science projects relying on the infrastructure. Targeting infrastructure openness and outreach toward adjacent communities, we deem important to map participation in interoperability initiatives, as well as involvement per community and career stage in IS-ENES outreach and training initiatives. We also aim at discussing ways on how to measure tangible outcomes resulting from engagement with Copernicus, EOSC, other European infrastructures, as well as with the industry. We offer a retrospective analysis of the methodology used in the previous phases of IS-ENES to measure impacts and an outlook of what IS-ENES3 proposes to increase and optimize them. In the crucial moment of preparing sustainability of a mature, heterogeneous, acknowledged yet expanding infrastructure, estimates of the extent of impacts (including financial ones when applicable) and return of target communities might ultimately feed back into infrastructure architecture decisions. [ABSTRACT FROM AUTHOR]

We describe the development of a resilient, distributed data infrastructure created to serve global climate projections data to the Copernicus Climate Change Service (C3S) Climate Data Store (CDS). The CDS provides a single public point of access for freely available climate-related observations, re-analyses and model data. This is achieved using a distributed architecture, with individual participating data providers hosting and serving datathrough to the CDS via an agreed set of web service interfaces. The CP4CDS project was established to provide access to a quality controlled subset of CMIP5 model data and is led by CEDA working together with partners DKRZ and IPSL.From the outset, an architecture was proposed leveraging the existing Earth System Grid Federation (ESGF), a globally distributed software infrastructure operated to provide access to CMIP data and other earth sciences data. However, the project requirements specified a high level of resilience for the data services (98% uptime) which no single site could meet as part of its standard SLA. A novel architecture was developed to meet this requirement based on a distributed deployment model and taking advantage of public cloud capabilities. Each of the three partner site provides an identical mirror of the data services. These are combined using load-balancing to deliver a single point of access thus providing an aggregate uptime which meets requirements. Amazon Web Services' Route 53 was selected in order to provide DNS-based load balancing. Cloud computing technologies have also had an important role in the hosting of the services themselves with CEDA's services hosted on-premise on the JASMIN data analysis facility's community cloud. In addition, container and container orchestration technologies have have many attractive features for building a highly available system. Over the last year, the ESGF collaboration have worked together to port the ESGF software stack to Docker and Kubernetes. This has facilitated a deployment of part of the system to Google Cloud.Public cloud hosting provides the means to meet the requirements for availability and uptime with a single deployment and potentially dispenses with the need for a distributed architecture. However, this must be weighed up against the high relative cost for the hosting of large volumes of model data (100s of TBs) when compared with on-premise hosting. In the short to medium term, a likely scenario in the wider ESGF community is the continued use of on-premise hosting for large datasets together with increased adoption of public cloud for the hosting of lighter weight services such a data search where there are much smaller storage requirements.The project has successfully delivered an operational system then, based on a hybrid model of a distributed data infrastructure together with public cloud. The same approach will be applied for a project to deliver regional model data CORDEX4CDS led by IPSL and will also influence the evolution of the architecture for ESGF. [ABSTRACT FROM AUTHOR]

In order to advance our ability to predict the future availability of water resources and the risk of water related disasters under a changing climate, it is necessary to bring together the land surface and hydrological modelling communities. This presentation will describe Hydro-JULES, a new research programme funded by the UK's Natural Environment Research Council. The Hydro-JULES programme will build a three-dimensional, open source, community model of the terrestrial water cycle to support and enable collaborative work across the research and academic communities in hydrology and land-surface science. An advanced terrestrial hydrological model will be generated that couples to the Joint UK Land Environment Simulator (JULES) and related models, including the UK Met Office Unified Model. This new five-year programme, is supported by NERC National Capability funding and will be delivered by the Centre for Ecology and Hydrology (CEH) in partnership with the British Geological Survey (BGS) and National Centre for Atmospheric Science (NCAS). Hydro-JULES aims to address critical research questions in the fields of hydrology, land-atmosphere feedbacks, carbon and nutrient cycles, data science and integration with novel instrumentation and Earth observation technologies; quantify the risks of hydro-climatic extremes (e.g., floods and drought) in a changing environment to support long-range planning and policy decisions; improve hydrological forecasting using new sensors and modelling technology. The Hydro-JULES project covers topics in land-surface science and hydrology including: quantification of hydro-meteorological risks, using high-resolution climate predictions for hydrological applications, calculation the impacts of environmental change on evaporation, transpiration, and soil moisture, modelling flood inundation over large areas, representing anthropogenic interventions in the water cycle, and application of new techniques including Earth observation and data assimilation. The presentation will explore the scientific drivers for integrated land surface and hydrological modelling and outline the main elements of the structure of the work programme. We discuss the development of interfaces between components of the terrestrial water cycle to support integrated modelling of the water cycle and associated Earth system feedbacks, and describe mechanisms for community engagement and collaboration. [ABSTRACT FROM AUTHOR]