Search

In the context of the EOSC-Pillar project, Work Package 6 (EOSC in action: Use cases and community-driven pilots) delivers use cases to analyse different tools and services used for the "FAIRification". Among them, Use Case 1 (Defining procedures and services to enforce data provenance for thematic communities and beyond) aims at elaborating cross-domain, FAIR-oriented procedures and recommendations to enforce data provenance in two scientific domains: Materials Science/Nanoscience and Climate Science. Learn more about use case 1

Due to the increasing complexity of data analysis workflows, provenance management is a key component to support the re-use of scientific data. Alongside findability, accessibility and interoperability, the reusability of data constitutes a pillar of the FAIR principles and in this regard rich and reliable provenance is essential to ensure that. In the context of the EOSC-Pillar project, Work Package 6 (EOSC in action: Use cases and community-driven pilots) delivers use cases to analyze different tools and services used for the "FAIRification". Among them, our use case (UC1: Defining procedures and services to enforce data provenance for thematic communities and beyond) aims at elaborating cross-domain, FAIR-oriented procedures and recommendations to enforce data provenance by taking into account needs coming from two scientific domains: Materials Science/Nanoscience and Climate Science. In such a context, (W3C) PROV standard has been considered a recommended family of specifications for tracking processes and responsibilities as well as describing provenance structures within complex experiments and large workflows on scientific data. Starting from the W3C PROV data model and ontology, the PROV Core concepts have been applied in the form of adaptations and extensions on top of the existing scientific data services, to enhance provenance capabilities, thus addressing FAIR-oriented full provenance management.

This document provides the description of the final releases and achieved integration for the thematic services part of EOSC-hub WP7 activities. For each thematic service, it also describes the impact, lesson learnt, exploitation of the services and how it will be sustained after the end of EOSC-hub.

This report provides a summary of the development activities carried out for the first software release of the ESDM-PAV runtime in the context of WP5 in the ESiWACE2 project.

Review of outcomes of the DMP and document lessons for future EU and international big data projects.

Talk at the 6th ENES Workshop on High Performance Computing for Climate and Weather, which was held on 25-26 and 28-29 May, 2020.

The term Big Data identifies large data sets that cannot be managed with traditional processing software. Volume, Variety and Velocity are properties used to classify Big Data. Velocity, the most relevant property, refers to high speed of data acquisition and processing. Management of Big Data, and particularly Velocity, has become an open issue for scientific communities. The EC project INDIGO­Datacloud was conceived to tackle some of the Big Data challenges that scientific communities are facing. In the context of INDIGO­Datacloud we defined a case study in which one of the goals was the improvement of processing performance on seismological data. Starting with a practical case study, we identified, as possible software solution, Ophidia, a big data analytics framework. The first step to exploit the framework functionalities required the import of the seismological data in Ophidia. For this purpose, we designed and implemented a software module that enables the import of seismological data in SAC format. As a second step, we tested the capabilities of Ophidia through a performance test to evaluate the framework potential.

The ESDM middleware has been designed to allow interactions with different kinds of backend, by means of data storing and retrieval. Specifically, the ESDM middleware allows interactions with various backends, exploiting different specific plugins to enable connection and I/O operations on different storage types. These plugins are embedded into a pre‐existing and generic core module able to support and expose all the interfaces needed to define I/O operations on the different backends; each specific plugin implements the interfaces to allow data management, storage, access and retrieval. The design targets, but is not limited to, earth system data, addressing the challenges faced by domain scientist and data centre operators for usability and performance. The architecture builds on top of NetCDF and HDF5 but utilises scientific metadata to harness a data structure centric perspective while retaining well‐established end‐user interfaces. We demonstrate prototype capability by conducting a preliminary performance assessment of the middleware on different systems to illustrate the importance of adaptive data placement. The milestone provides technical details of the deployments on the clusters at STFC, DKRZ, CMCC, and on a test system at Seagate, meeting the formal project requirement for deployment on these platforms. This deployment could be used to support future demonstrator applications, albeit not all features are completely implemented yet., ESiWACE has received funding from the European Union's Horizon 2020 Research and Innovation Programme under Grant Agreement No 675191

About this document Work package in charge: WP3 Usability Actual delivery date for this deliverable: M35 Dissemination level: PU (for public use) Lead authors: Met Office (Met O): Matt Shin, Mick Carter, Dave Matthews Fondazione Centro Euro-Mediterraneo sui Cambiamenti Climatici (CMCC): Sandro Fiore, Alessandro D’Anca, Maria Mirto, Cosimo Palazzo, Giovanni Aloisio Other contributing authors: Max-Planck-Institut für Meteorologie (MPI-M): Reinhard Budich Barcelona Supercomputing Center (BSC): Domingo Manubens, Kim Serradell Reviewer: German Climate Computing Center (DKRZ): Philipp Neumann This document describes the work done in the context of the ESiWACE project on the topic Earth System Modelling scheduling (Task3.3) whose ultimate goal is to build a supported, user-driven community around the Cylc meta-scheduler for complex climate and weather suites on HPC systems and support advanced analysis experiments in the areas of data analytics and multi-site, multi-model experiments. This deliverable provides an update on this topic with respect to the Deliverable D3.9 - ESiWACE Scheduler development and support activities. The main focus of this effort relates to three aspects: Cylc developments (Task 3.3.2), Cylc Support Services (Task 3.3.3), and Scheduling Systems Development (Task 3.3.4). In particular, (i) the Cylc developments have followed the priorities agreed at a workshop on workflows organised by the IS-ENES2 project (Lisbon, September 2016) and include optimisations which supports the aims of large ensembles on Exascale platforms as anticipated in the long term, arising from workflow management and production-ready versions of the ESiWACE demonstrators, (ii) the Cylc support has addressed needs from the community regarding Cylc deployment and configuration at major data centres in the community, and, finally, (iii) the scheduling systems development has addressed major points on this topic, like architectural aspects and integrated scenarios, about large-scale analytics workflows for multi-model/multi-site data analysis. Here there are opportunities to build links to ESDM middleware from work-package 4. Finally, from a networking perspective, the 3rd Workshop on Workflows (funded by ESiWACE) has been organised over the last months and will be held in Brussels in September with 18 speakers presenting talks on several workflow topics, more than 40 registrants, 3 roundtable discussions and 1 training session., ESiWACE has received funding from the European Union's Horizon 2020 Research and Innovation Programme under Grant Agreement No 675191

Poster on CMCC contribution on Scalability, Exploitability and Usability in the context of theESiWACE project presented at the CMCC Annual Meeting (June 5-6, 2018, Ugento (IT)).

Presentation given at PASC18 Conference which mentions Ophidia as an ESDM big data analytics demonstrator in HPC environment.

Presentation of the paper "Towards an Open (Data) Science Analytics-Hub for Reproducible Multi-Model Climate Analysis at Scale" at the IEEE BigData2018 Conference (Seattle, WA, USA)

Open Science is key to future scientific research and promotes a deep transformation in the whole scientific research process encouraging the adoption of transparent and collaborative scientific approaches aimed at knowledge sharing. Open Science is increasingly gaining attention in the current and future research agenda worldwide. To effectively address Open Science goals, besides Open Access to results and data, it is also paramount to provide tools or environments to support the whole research process, in particular the design, execution and sharing of transparent and reproducible experiments, including data provenance (or lineage) tracking. This work introduces the Climate Analytics-Hub, a new component on top of the Earth System Grid Federation (ESGF), which joins big data approaches and parallel computing paradigms to provide an Open Science environment for reproducible multi-model climate change data analytics experiments at scale. An operational implementation has been set up at the SuperComputing Centre of the Euro-Mediterranean Center on Climate Change, with the main goal of becoming a reference Open Science hub in the climate community regarding the multi-model analysis based on the Coupled Model Intercomparison Project (CMIP). This paper reports about some ESiWACE WP3 activities described in the deliverable D3.10 "ESiWACE Scheduler development and support activities"