Search

The EC H2020 C-SCALE (Copernicus - eoSC AnaLytics Engine, https://c-scale.eu) project implements a European open source Big (Copernicus) Data Analytics platform by federating the best-of-breed tools, competences and services by collaboratively building on the experience and competences of pan-European e-Infrastructures and existing project initiatives.The vision of the project is to empower European researchers, institutions and initiatives to easily discover, access, process, analyse and share Copernicus data, tools, resources and services through the EOSC Portal. To this end, C-SCALE delivers a federated compute and data infrastructure offering Copernicus and Earth Observation (EO) data, including a seamless user experience where the complexity of resource provisioning and orchestration is abstracted away from the end-user. The service offer of C-SCALE includes four main services:The Federated Earth System Simulation and Data Processing Platform (FedEarthData) service brings together the providers of data and processing capacity, so that EO products held in distributed archives across the federation can be easily discovered and seamlessly accessed and processed on batch as well as on interactive analytic platforms deployed on distributed computing resources anywhere across the federation. The Metadata Query Service (MQS) makes Copernicus data distributed across partners within the federation discoverable and searchable. It is a STAC-compliant API that redistributes incoming queries among the federated sites and provides a consolidated response containing the list of aggregated results. The MQS exposes all STAC collections available within the federation on a single endpoint and provides a search interface that accepts the core parameters of the STAC API Item Search specification. The openEO platform service provides intuitive programming libraries alongside with a large EO data repository to simplify processing and data management. This large-scale data access and computation is performed on multiple infrastructures allowing use cases from exploratory research to large-scale production of EO-derived maps and information in an accelerated way. The Workflow Solutions are easily deployable workflows supporting monitoring, modelling and forecasting of the Earth system. They provide adaptable templates and examples, in the form of Jupyter Notebooks, of Copernicus and EO data and analysis workflows enabling users to more easily arrange a processing pipeline to create results on the C-SCALE federation. In addition to these services, the project delivers the C-SCALE community, for the engagement with existing and new stakeholders, including both researchers and service providers in Earth Observation, documentation and training material.By the aforementioned services, the project aims to scale up the EOSC Portal through a continuously growing catalogue of services and resources supporting the whole research life cycle and enabling more scientific communities to access state-of-the-art services for their research activities. In addition, C-SCALE facilitates synergies between pan-European e-infrastructures operators, leading to harmonised services, improved use of resources and economies of scale.

Through the provision of massive streams of high-resolution Earth Observation (EO) data, the EU Copernicus programme has established itself globally as the predominant spatial data provider. These data are widely used by research communities to monitor and address global challenges, such as environmental monitoring and climate change, supporting European policy initiatives, such as the Green Deal and others. To date, there is no single European data sharing and processing infrastructure that serves all datasets of interest, and Europe is falling behind international developments in Big Data analytics and computing.The C-SCALE (Copernicus - eoSC AnaLytics Engine, https://c-scale.eu) project federates European EO infrastructure services, such as ESA’s Sentinel Collaborative Ground Segment, the Copernicus DIASes (Data and Information Access Services under the EC), independent nationally-funded EO service providers, and European Open Science Cloud (EOSC) e-infrastructure providers. It capitalises on EOSC's capacity and capabilities to support Copernicus research and operations with large and easily accessible European computing environments. The project will implement and publish the C-SCALE Federation in the EOSC Portal as a suite of complementary services that can be easily exploited. It will consist of a Data Federation, a service providing access to a large EO data archive, a Compute Federation, and analytics tools.The C-SCALE Data Federation aims at making EO data providers under EOSC findable, their metadata databases searchable, and their product storage accessible. While a centralised, monolithic, complete Copernicus data archive may not be feasible, some organisations maintain various archives for limited areas of their interest. C-SCALE, therefore, integrates these heterogeneous resources into a “system of systems” that will offer the users an interface that, in most cases, provides similar functionality and quality of service as a centralised, monolithic data archive would. The federation is built on existing technologies, avoiding redundancy and replication of functions and not disrupting existing usage patterns at participating sites, instead only adding a simple layer for improved discovery and seamless access.At the same time, the C-SCALE Compute Federation provides access to a wide range of computing providers (IaaS VMs, container orchestration platforms, HPC and HTC systems) to enable the analysis of Copernicus and EO data under EOSC. The design of the federation allows users to deploy their applications using federated authentication mechanisms, find their software under a common catalogue, and have access to data using C-SCALE Data Federation tools. The federation relies on existing tools and services already compliant with EOSC, thus facilitating the integration into the larger EOSC ecosystem.By making such a scalable Big Copernicus Data Analytics federated services available through EOSC and its Portal and linking the problems and results with experience from other research disciplines, C-SCALE helps to support the EO sector in its development. By abstracting the set-up of computing and storage resources from the end-users, it enables the deployment of custom workflows to generate meaningful results quickly and easily. Furthermore, the project will deliver a blueprint, setting up an interaction model between service providers to facilitate interoperability between commercial and public cloud infrastructures.

EGI-ACE delivers ‘EOSC Compute Platform’ services and ‘Data Space services’ in EOSC. This deliverable reports on the integration and alignment activities between EGI-ACE and EOSC. The document describes the approaches that were taken by the project for the service portfolio integration; technical interoperability; service management system; FAIRness assessment of the data spaces. Based on the findings the project will continue to strengthen the links with EOSC in 2022. An updated version of this deliverable will be produced as D2.8 in December 2022.

This deliverable introduces the second version of the Technical Roadmap for the EOSC-hub services. It describes the capabilities, the features and the plan for the technical evolution of the services within the EOSC hub service portfolios, with a focus on the activities needed to improve the interoperability among the services and to enable service composition in the wider EOSC environment. Plans depicted in this deliverable will be implemented in EOSC-hub follow-up projects, notably those funded under the INFRAEOSC-03 (EOSC Future) and INFRAOESC-07 calls (EGI-ACE, DICE, etc.).

The Technical Coordination Work Package (WP10) in the EOSC-hub project plays an important role aimed at supporting both participants (e.g. Thematic Services, Competence Centres and Business pilots) and external user communities (not directly involved in the project but engaged through other channels, e.g. the EOSC Portal) in the process to integrate their services into the Hub. In the first report WP10 has defined and is currently operating procedures to elicit, assess and track the technical requirements of such communities and, consequently, provide adequate assistance to the user communities to adopt the service of the Hub. In order to better structure and consolidate this support WP10 extended such activities by launching the Early Adopter Programme for research communities interested in exploring the latest state-of-art technologies and services offered by the European Open Science Cloud. In this respect, details are provided about the outcome of the above mentioned technical support activities, together with the related requirements and gap analysis extracted from the needs of the different communities involved.

Monitoring allows to quickly detect, correlate, and analyse data for a fast reaction to anomalous behaviour that may affect end-users and ultimately the productivity of the organization. The key functional requirements of a monitoring system are monitoring of services, reporting availability and reliability, visualization of the services status, provide dashboard interfaces and sending real-time alerts. This document describes the high-level service architecture for an EOSC Monitoring service and presents the main integration and usage use cases for monitoring in EOSC. It proposes interfaces as guidelines to be followed to achieve the interoperability between monitoring systems in EOSC for three envisaged use cases: (1) combine Results of one or more infrastructures in EOSC in a unified UI, (2) add a Service Provider/Infrastructure to EOSC Monitoring and (3) Third-party services exploiting EOSC Monitoring data.

This document describes the EOSC-hub contribution to the definition of the EOSC Technical Architecture, which is currently being developed by the EOSC architecture Working Group. It is based on the concepts of service interoperability and end-to-end composition of services and foresees the definition of a reference architecture in which EOSC building blocks and the main functions, interfaces, APIs and standards are identified. This architecture is expected to facilitate access to services, lower the barriers to integrate and composes services and promote the usage of services between adjacent communities. As a basis for the proposed architecture, service categories have been introduced, mapping their functions, relationships and organisation to the kinds of services required for the federating core of EOSC and the external EOSC service portfolio. The concept of the end-to-end composition of services has been presented, highlighting the most common integration scenarios. Leveraging the defined service categories and on the concepts of service interoperability and composition, a reference EOSC Technical Architecture has been defined identifying a hierarchical structure where the first level relies on service categories (Federation & Access enabling,, Common and Thematic), the second level introduces functional categories, that groups technical functions to facilitate their identification, and the third is made of the technical functions that has been called building blocks. EOSC-hub is working on defining the building blocks of the architecture for each service type and specified a common approach to complete this task. It foresees the identification of the main building blocks/technical functions in each service category and, for each of those, the definition of a technical specification that includes a high-level architecture, suggested EOSC standards and APIs and interoperability guidelines. As a consequence, interoperability between services compliant with the EOSC specifications will be easier to be achieved.

Nowadays, research practice in all scientific disciplines is increasingly, and in many cases exclusively, data driven. Knowledge of how to use tools to manipulate research data, and the availability of e-Infrastructures to support them for data storage, processing, analysis and preservation, is fundamental. In parallel, new types of communities are forming around interests in digital tools, computing facilities and data repositories. By making infrastructure services, community engagement and training inseparable, existing communities can be empowered by new ways of doing research, and new communities can be created around tools and data. Europe is ideally positioned to become a world leader as provider of research data for the benefit of research communities and the wider economy and society. Europe would benefit from an integrated infrastructure where data and computing services for big data can be easily shared and reused. This is particularly challenging in EO given the volumes and variety of the data that make scalable access difficult, if not impossible, to individual researchers and small groups (i.e. to the so-called long tail of science). To overcome this limitation, as part of the European Commission Digital Single Market strategy, the European Open Science Cloud (EOSC) initiative was launched in April 2016, with the final aim to realise the European Research Area (ERA) and raise research to the next level. It promotes not only scientific excellence and data reuse, but also job growth and increased competitiveness in Europe, and results in Europe-wide cost efficiencies in scientific infrastructure through the promotion of interoperability on an unprecedented scale. This chapter analyses existing barriers to achieve this aim and proposes the Open Science Commons as the fundamental principles to create an EOSC able to offer an integrated infrastructure for the depositing, sharing and reuse of big data, including Earth Observation (EO) data, leveraging and enhancing the current e-Infrastructure landscape, through standardization, interoperability, policy and governance. Finally, it is shown how an EOSC built on e-Infrastructures can improve the discovery, retrieval and processing capabilities of EO data, offering virtualised access to geographically distributed data and the computing necessary to manipulate and manage large volumes. Well-established e-Infrastructure services could provide a set of reusable components to accelerate the development of exploitation platforms for satellite data solving common problems, such as user authentication and authorisation, monitoring or accounting.

The motivation of this work fits with the general vision to enable e-health for European citizens, irrespective of their social and financial status and their place of residence. Services to be provided include access to a high-quality early diagnostic and prognostic service for the Alzheimer Disease and other forms of dementia, based both on the European Research and Education Networks and the European Grid Infrastructure. The present paper reports on the architecture and services of a Science Gateway developed in the context of the DECIDE project, which aims to support the medical community in its daily duties of patients’ examination and diagnosis. The implementation of the Science Gateway is described with particular focus on the standard technologies adopted to ease the access by non IT-.expert users. The work leverages on an authentication and authorization infrastructure based on Identity Federations and robot certificates and on the adoption of the SAGA standard for middleware-independent Grid interaction. The architecture and the functionalities of the digital repository for medical image storage and analysis are also presented.

Grids currently in production can be broadly classified as either service Grids, composed of dedicated resources, or opportunistic Grids that harvest the computing power of non-dedicated resources when they are idle. While a service Grid provides high and well defined levels of quality of service, an opportunistic Grid provides only a best-effort service. Nevertheless, since opportunistic Grids do not require resources to be fully dedicated to the Grid, they have the potential to assemble a much larger number of resources. Moreover, these Grids cater very well to the execution of the so-called embarrassingly parallel applications, a type of application that is frequently found in practice, and that comprises the largest portion of the typical workload processed in production Grid systems. The EELA-2 e-infrastructure is comprised of a service Grid and an opportunistic Grid that federates computing resources from scientific institutions in both Europe and Latin America. Due to the complementary characteristics of these two types of Grids, a lot of attention has recently been placed in how to interoperate them. In this paper we focus on the less studied problem of assessing the feasibility of such interoperation. We analyse different prioritisation policies that define when the resources of one Grid can be used to run jobs originating from the other. Our results show that in the absence of a suitable prioritisation policy, the benefits that the users of one Grid may have, frequently come with an important negative impact on the users of the other Grid. We also show that a simple reciprocation mechanism is capable of arbitrating the interoperation in such a way that, whenever possible, users profit from the interoperation and, in no case, this benefit leads to a noticeable reduction on the quality of service that the users would experience were the Grids not to interoperate. We conclude discussing how we have implemented, in the context of the EELA-2 project, this prioritisation mechanism, allowing the effective interoperation of a service Grid based on the gLite middleware with an opportunistic Grid that uses the OurGrid middleware.

One of the benefits of OCCI stems from simplifying the life of developers aiming to integrate multiple cloud managers. It provides them with a single protocol to abstract the differences between cloud service implementations used on sites run by different providers. This comes particularly handy in federated clouds, such as the EGI Federated Cloud Platform, which bring together providers who run different cloud management platforms on their sites: most notably OpenNebula, OpenStack, or Synnefo. Thanks to the wealth of approaches and tools now available to developers of virtual resource management solutions, different paths may be chosen, ranging from a small-scale use of an existing command line client or single-user graphical interface, to libraries ready for integration with large workload management frameworks and job submission portals relied on by large science communities across Europe. From lone wolves in the long-tail of science to virtual organizations counting thousands of users, OCCI simplifies their life through standardization, unification, and simplification. Hence cloud applications based on OCCI can focus on user specifications, saving cost and reaching a robust development life-cycle. To demonstrate this, the paper shows several EGI Federated Cloud experiences, demonstrating the possible approaches and design principles.

This paper details the state of the art, the design, development and deployment of the EGI Federated Cloud platform, an e-infrastructure offering scalable and flexible models of utilization to the European research community. While continuing support for the traditional High Throughput Computing model, the EGI Cloud Platform extends its reach to other models of utilization such as long-lived services and on demand computation. Following a two-year period of development, the EGI Federated Cloud platform was officially launched in May 2014 offering resources provided by trusted academic and research organisations from within the user communities and consistently with their standard funding regime. Since then, the use cases supported have significantly increased both in total number and diversity of model of service required, validating both the choice of enforcing cloud technology agnosticism and of supporting service mobility and portability by means of open standards. These design choices have also allowed for the inclusion of commercial cloud providers into an infrastructure previously supported only by academic institutions. This contributes to a wider goal of funding agencies to create economic and social impact from supported research activities.

The EGI Federated Cloud is a standards-based, open cloud system as well as its enabling technologies that federates institutional clouds to offer a scalable computing platform for data and/or compute driven applications and services. The EGI Federated Cloud is based on open standards and open source Cloud Management Frameworks and offers to its users IaaS, PaaS and SaaS capabilities and interfaces tuned towards the needs of users in research and education. The federation enables scientific data, workloads, simulations and services to span across multiple administrative locations, allowing researchers and educators to access and exploit the distributed resources as an integrated system. The EGI Federated Cloud collaboration established a user support model and a training infrastructure to raise visibility of this service within European scientific communities with the overarching goal to increase adoption and, ultimately increase the usage of e-infrastructures for the benefit of the whole European Research Area. The paper describes this scalable user support and training infrastructure models. The training infrastructure is built on top of the production sites to reduce costs and increase its sustainability. Appropriate design solutions were implemented to reduce the security risks due to the cohabitation of production and training resources on the same sites. The EGI Federated Cloud educational program foresees different kind of training events from basic tutorials to spread the knowledge of this new infrastructure to events devoted to specific scientific disciplines teaching how to use tools already integrated in the infrastructure with the assistance of experts identified in the EGI community. The main success metric of this educational program is the number of researchers willing to try the Federated Cloud, which are steered into the EGI world by the EGI Federated Cloud Support Team through a formal process that brings them from the initial tests to fully exploit the production resources. © 2015 IEEE.

Chipster is an easy to use data analysis platform for bioinformatics. It provides an uniform graphical interface for over 360 commonly used bioinformatics tools including several R/Bioconductor-based tools and standalone programs (i.e. BWA, TopHat). Chipster is based on a client-server system where the user runs locally a Chipster-client that submits analysis tasks to a Chipster server. Even though Chipster is an open source tool, there is no public Chipster server that would be open for everybody. Due to that, a researcher needs to have an access to some of the existing Chipster servers to be able to use this platform. Alternatively, a researcher can set up his own Chipster server. In this paper, we describe how a Chipster server can be launched EGI Federated Could environment, that provides resources for all European researchers. With the instructions provided here, any European researcher can launch and manage his own Chipster server, suited for needs of a small research group or a bioinformatics course. The setup described here is based on a collaboration of several European instances. Chipster is developed by CSC – IT Center for Science Ltd. in Finland. European Grid Infrastructure (EGI) has fitted Chipster to cloud environment and provides the cloud computing resources. Finally, Rutherford Appleton Laboratory hosts the CVMFS server that provides the scientific tools and data sets for the Chipster servers running in EGI federated cloud.

This paper identifies the high value to researchers in many disciplines of having web-based graphical editors for scientific workflows and draws attention to two technological transitions: good quality editors can now run in a browser and workflow enactment systems are emerging that manage multiple workflow languages and support multi-lingual workflows. We contend that this provides a unique opportunity to introduce multi-lingual graphical workflow editors which in turn would yield substantial benefits: workflow users would find it easier to share and combine methods encoded in multiple workflow languages, the common framework would stimulate conceptual convergence and increased workflow component sharing, and the many workflow communities could share a substantial part of the effort of delivering good quality graphical workflow editors in browsers. The paper examines whether such a common framework is feasible and presents an initial design for a web-based editor, tested with a preliminary prototype. It is not a fait accompli but rather an urgent rallying cry to explore collaboratively a generic web-based framework before investing in many divergent individual implementations.

In this paper we present the architecture of a framework for building Science Gateways supporting official standards both for user authentication and authorization and for middleware-independent job and data management. Two use cases of the customization of the Science Gateway framework for Semantic-Web-based life science applications are also described.

In the last 10 years, a new way of doing science is spreading in the world thank to the development of virtual research communities across many geographic and administrative boundaries. A virtual research community is a widely dispersed group of researchers and associated scientific instruments working together in a common virtual environment. This new kind of scientific environment, usually addressed as a "collaboratory", is based on the availability of high-speed networks and broadband access, advanced virtual tools and Grid-middleware technologies which, altogether, are the elements of the e-Infrastructures. The European Commission has heavily invested in promoting this new way of collaboration among scientists funding several international projects with the aim of creating e-Infrastructures to enable the European Research Area and connect the European researchers with their colleagues based in Africa, Asia and Latin America. In this paper we describe the actual status of these e-Infrastructures and present a complete picture of the virtual research communities currently using them. Information on the scientific domains and on the applications supported are provided together with their geographic distribution. © 2011 Springer Science+Business Media B.V.

In this chapter, we describe a successful methodology to support ­e-Science applications on e-Infrastructures put in practice in the EELA-2 project co-funded by the European Commission and involving European and Latin American countries. The heterogeneous requirements of the e-Science applications, coming from several scientific fields, makes difficult to provide them with a support able to satisfy all the different needs. Usually, the grid middleware adopted, gLite in the case of EELA-2, provides applications with general tools not able to meet specific requirements. For this reason, a really powerful e-Infrastructure has to offer some additional services to complete and integrate the functionalities of the grid middleware. These services have to both increase the set of functionalities offered by the e-Infrastructure and make easier the tasks of developing and deploying new applications. Following this methodology, EELA-2 deployed 53 e-Science applications out of the 61 supported in total, in its enriched e-Infrastructure during its life.

A measure to estimate the value that Grids can provide to potential users can be obtained by assessing the resources availability, middleware overhead and infrastructure reliability incurred when running an application in a transcontinental e-Infrastructure like EGEE. Celebrating the recent MoU between EELA-2 [1] and EUAsiaGrid [2] projects, both co-funded by EC under the Seventh Framework Programme, this paper aims at providing a comparative study between their respective Grid infrastructures. Current monitoring tools provide information on the resources status. These figures are useful for Grid managers in order to check the availability of the services but not for end users because they do not provide any indication on the execution of users’ applications, such as the average job delay. In our approach, we randomly submitted 10 jobs per day during 1 week both project´s infrastructures and measured its total execution time. No special requirements were set on the JDL files and we did not carry about the level of availability of the computing resources (CEs) neither about the number of jobs concurrently running at a giving moment. We let the core Workload Management System (WMS) of each project to automatically choose which CE to submit the jobs, considering both EUAsiaGrid and EELA-2 infrastructures as single entities. The analysis of the results can be used to measure the quality of services provided by both projects to its respective user communities.

The heterogeneous requirements of e-Science applications belonging to several scientific domains makes difficult to provide them with a support able to satisfy all the different needs. Usually, the grid middleware adopted provides applications with general tools unable to meet specific requirements. For this reason, a really powerful e-Infrastructure has to offer some additional services to complete and integrate the functionalities of the basic grid middleware. These services have both to increase the set of functionalities offered by the e-Infrastructure and to make the task of developing and deploying new applications easier. Following this methodology, EELA-2, a project co-funded by the European Commission and involving European and Latin American countries, has fully deployed 53 e-Science applications. This paper describes the services developed to achieve this outstanding result.

The increasing digital divide makes scientists, educators, and students from many parts of the worlds not able to take advantage of last ICT developments. Therefore, many countries are increasingly marginalised as the world of education and science becomes increasingly Internetdependent. The Grid INFN Laboratory for Dissemination Activities (GILDA) provides a Grid training infrastructure used to spreads the Grid technology to a wider range of users. So far several scientific communities such as biologists, physicists and many others have been successfully supported. Currently, GILDA is the choice infrastructure for the EUAsiaGrid project training activity which aims at creating new Grid communities in Asia. The training on GILDA allows new users to experienced this new technology and understand their requirements before to perform expensive investments. In this contribution we will report on the latest status of GILDA services and on the training activities recently carried out in the supported projects.

This paper describes the EGI ‘Applications on Demand service’, a new offering from EGI specifically for individual researchers, small research teams and early phase research infrastructures to support them in scientific data analysis. The described service is available through the EGI Marketplace and, through a lightweight registration and user identity vetting process, allows user-friendly access to a growing number of scientific applications, tools and application hosting frameworks (science gateways; Virtual Research Environments; portal) that are configured to use a dedicated pool of cloud computing and High Throughput Compute clusters donated by members of the EGI federation. The service operates as an open and extensible ‘hub’ for providers and user support teams who wish to federate and share applications and services with individual researchers or small, fragmented communities — typically called ‘the long tail of science’. At the time of writing the service is under integration into the European Open Science Cloud through the EOSC-hub H2020 project.

The EOSC Early Adopter Programme (EAP) is a programme launched by EOSC-hub for research communities interested in exploring the latest state-of-art technologies and services offered by the European Open Science Cloud (EOSC). 13 pilots were selected in two calls spanning several scientific disciplines: agriculture, marine, material sciences, disaster mitigation, health science, biology, astronomy, light pollution etc. The pilots exploited generic and thematic services and resources from EOSC-hub and its partners and completed 25 integrations with 9 different EOSC-hub services. Three of the pilots registered new thematic services in EOSC, and two more will follow in 2021, demonstrating how an operational EOSC can boost the research in Europe and the adoption of new paradigms to deal with the increasing complexity of science. The majority of the pilots will continue either as service provider, or with continued integration after the end of EOSC-hub. The follow-up was made possible by individual agreements that the pilots made with EGI and EUDAT, or with INFRAEOSC-07 H2020 projects (EGI-ACE, DICE, C-SCALE).