Report on the implementation of the ESDM PAV analytical kernels for post-processing, analysis and visualisation (T5.3) - Deliverable D5.2

The Earth System Data Middleware (ESDM) framework was developed in ESiWACE and can now serve as a platform to execute processing, analysis and visualisation (PAV) workflows in a way that is using meta data of the underlying technology and site specific storage systems on one hand and the workflows themselves on the other hand to automatically optimise data access for performance. This has the potential to remove bottlenecks in the workflows for very large data sets as they are to be expected for the upcoming Exascale systems. Analytical kernels have been developed further, rewritten and tested for the post-processing and analysis systems CDO and Ophidia to show the interplay with ESDM. For visualisation, there now is an ESDM-enabled netCDF library to accommodate direct input from ESDM into the visualisation tool ParaView. These developments and preliminary tests were successful despite the fact that ESDM currently is not used in a widely spread manor for Earth System Model (ESM) outputs. This seems to be due to the fact that so far its potential has not been shown to be useful enough to replace off-the-shelf, proven, mature systems providing the same functionalities. This document provides an overview of the analytical kernels developed for CDO and Ophidia making use of accelerated architectures (i.e., GPUs) and of streaming I/O APIs (for in-flight analytics).
ESiWACE2 stands for Centre of Excellence in Simulation of Weather and Climate in Europe Phase 2. ESiWACE2 is funded by the European Union's Horizon 2020 research and innovation programme (H2020-INFRAEDI-2018-1 call) under grant agreement 823988.