Search

In recent years, cloud sync & share storage services, provided by academic and research institutions, have become a daily workplace environment for many local user groups in the High Energy Physics (HEP) community. These, however, are primarily disconnected and deployed in isolation from one another, even though new technologies have been developed and integrated to further increase the value of data. The EU-funded CS3MESH4EOSC project is connecting locally and individually provided sync and share services, and scaling them up to the European level and beyond. It aims to deliver the ScienceMesh service, an interoperable platform to easily sync and share data across institutions and extend functionalities by connecting to other research services using streamlined sets of interoperable protocols, APIs and deployment methodologies. This supports multiple distributed application workflows: data science environments, collaborative editing and data transfer services. In this paper, we present the architecture of ScienceMesh and the technical design of its reference implementation, a platform that allows organizations to join the federated service infrastructure easily and to access application services outof-the-box. We discuss the challenges faced during the process, which include diversity of sync & share platforms (Nextcloud, Owncloud, Seafile and others), absence of global user identities and user discovery, lack of interoperable protocols and APIs, and access control and protection of data endpoints. We present the rationale for the design decisions adopted to tackle these challenges and describe our deployment architecture based on Kubernetes, which enabled us to utilize monitoring and tracing functionalities. We conclude by reporting on the early user experience with ScienceMesh. [ABSTRACT FROM AUTHOR]

With containers being the de-facto standard to package, distribute, and run applications, Helm charts are on the rise for application deployment in managed clusters (e.g., Kubernetes, OpenShift), providing developers and operators with a rich ecosystem of tools to utilize, as well as the means to configure applications and roll changes out in a programmatic way. This paper describes the reboot of the ScienceBox project: The containerized software bundle providing the ability to deploy CERN storage and analysis services on any cloud or local infrastructure. While the service offering provided through ScienceBox remains unchanged, we evolved the original implementation to make use of Helm charts across the entire stack and incorporated a major architectural update to CERNBox, replacing the previous PHP backend with a distributed microservices architecture. ScienceBox maintains the flexibility to be deployed in a few clicks for demonstration purposes, or scaled out on managed clusters for the provisioning of a multi-user service. It also removes external dependencies by deploying additional services (e.g., IDP, DBs) and facilitates the configuration of the service stack by providing automated configuration scripts, making the package fully self-contained and easy to use. [ABSTRACT FROM AUTHOR]