Search

The High Luminosity Large Hadron Collider provides a data challenge. The amount of data recorded from the experiments and transported to hundreds of sites will see a thirty fold increase in annual data volume. A systematic approach to contrast the performance of different Third Party Copy(TPC) transfer protocols arises. Two contenders, XRootD-HTTPS and the GridFTP are evaluated in their performance for transferring files from one server to an-other over 100Gbps interfaces. The benchmarking is done by scheduling pods on the Pacific Research Platform Kubernetes cluster to ensure reproducible and repeatable results. This opens a future pathway for network testing of any TPC transfer protocol., 7 pages, 8 figures

Large scientific data centers have recently begun providing a number of different types of data storage in order to satisfy the various needs of their users. Users with interactive accounts, for example, might want a POSIX interface for easy access to the data from their interactive machines. Grid computing sites, on the other hand, likely need to provide an X509-based storage protocol, like SRM and GridFTP, since the data management system is built upon them. Meanwhile, an experiment producing large amounts of data typically demands a service that provides archival storage for the safe keeping of their unique data. To access these various types of data, users must use specific sets of commands tailored to their respective storage, making access to their data complex and difficult. BNLBox is an attempt to provide a unified and easy to use storage service for all BNL users, to store their important documents, code and data. It is a cloud storage system with an intuitive web interface for novice users. It provides an automated synchronization feature that enables users to upload data to their cloud storage without manual intervention, freeing them to focus on analysis rather than data management software. It provides a POSIX interface for local interactive users, which simplifies data access from batch jobs as well. At the same time, it also provides users with a straightforward mechanism for archiving large data sets for later processing. The storage space can be used for both code and data within the compute job environment. This paper will describe various aspects of the BNLBox storage service.

The XRootD software framework is essential for data access at WLCG sites. The WLCG community is exploring and expanding XRootD functionality. This presents a particular challenge at the RAL Tier-1 as the Echo storage service is a Ceph based Erasure Coded object store. External access to Echo uses gateway machines which run GridFTP and caching servers. Local jobs access Echo via caches on every worker node, but it is clear there are inefficiencies in the system. Remote jobs also access data via XRootD on Echo. For CMS jobs this is via the AAA service. ATLAS, who are consolidating their storage at fewer sites, are increasingly accessing job input data remotely. This paper describes the continuing work to optimise both local and remote data access by testing different caching methods.

23rd International Conference on Computing in High Energy and Nuclear Physics, CHEP 2018, Sofia, Bulgaria, 9 Jul 2018 - 13 Jul 2018; The European physical journal / Web of Conferences Web of Conferences : proceedings proceedings 214, 04045 (2019). doi:10.1051/epjconf/201921404045, GridFTP transfers and the corresponding Grid Security Infrastructure (GSI)-based authentication and authorization system have been data transfer pillars of the Worldwide LHC Computing Grid (WLCG) for more than a decade. However, in 2017, the end of support for the Globus Toolkit - the reference platform for these technologies - was announced. This has reinvigorated and expanded efforts to replace these pillars. We present an end-to-end alternate utilizing HTTP-based WebDAV as the transfer protocol, and bearer tokens for distributed authorization.This alternate ecosystem, integrating significant pre-existing work and ideas in the area, adheres to common industry standards to the fullest extent possible, with minimal agreed-upon extensions or common interpretations of the core protocols. The bearer token approach allows resource providers to delegate authorization decisions to the LHC experiments for experiment-dedicated storage areas.This demonstration touches the entirety of the stack - from multiple storage element implementations to FTS3 to the Rucio data management system. We show how the traditional production and user workflows can be reworked utilizing bearer tokens, eliminating the need for GSI proxy certificates for storage interactions., Published by EDP Sciences, Les Ulis

In this paper we introduce and describe the highly concurrent xDFS file transfer protocol and examine its cross-platform and cross-language implementation in native code for both Linux and Windows in 32 or 64-bit multi-core processor architectures. The implemented xDFS protocol based on xDotGrid.NET framework is fully compared with the Globus GridFTP protocol. We finally propose the xDFS protocol as a new paradigm of distributed systems for Internet services, and data-intensive Grid and Cloud applications. Also, we incrementally consider different developmental methods of the optimum file transfer systems, and their advantages and disadvantages. The vision of this paper tries as possible to minimize the overhead concerned with the file transfer protocol itself and to examine optimal software design patterns of that protocol. In all disk-to-disk tests for transferring a 2GB file with or without parallelism, the xDFS throughput at minimum 30% and at most 53% was superior to the GridFTP., 25 pages, 20 figures

WebDav and HTTP are becoming popular protocols for data access in the High Energy Physics community. The most used Grid and Cloud storage solutions provide such kind of interfaces, in this scenario tuning and performance evaluation became crucial aspects to promote the adoption of these protocols within the Belle II community. In this work, we present the results of a large-scale test activity, made with the goal to evaluate performances and reliability of the WebDav protocol, and study a possible adoption for the user analysis. More specifically, we considered a pilot infrastructure composed by a set of storage elements configured with the WebDav interface, hosted at the Belle II sites. The performance tests include a comparison with xrootd and gridftp. As reference tests we used a set of analysis jobs running under the Belle II software framework, accessing the input data with the ROOT I/O library, in order to simulate as much as possible a realistic user activity. The final analysis shows the possibility to achieve promising performances with WebDav on different storage systems, and gives an interesting feedback, for Belle II community and for other high energy physics experiments.

We present the Multiscale Coupling Library and Environment: MUSCLE 2. This multiscale component-based execution environment has a simple to use Java, C++, C, Python and Fortran API, compatible with MPI, OpenMP and threading codes. We demonstrate its local and distributed computing capabilities and compare its performance to MUSCLE 1, file copy, MPI, MPWide, and GridFTP. The local throughput of MPI is about two times higher, so very tightly coupled code should use MPI as a single submodel of MUSCLE 2; the distributed performance of GridFTP is lower, especially for small messages. We test the performance of a canal system model with MUSCLE 2, where it introduces an overhead as small as 5% compared to MPI., Comment: 18 pages, 22 figures, submitted to journal

The Florida CMS Tier2 center, one of the CMS Tier2 centers, has been using the Lustre filesystem for its data storage backend system since 2004. Recently, the data access pattern at our site has changed greatly due to various new access methods that include file transfers through the GridFTP servers, read access from the worker nodes, and the remote read access through the xrootd servers. In order to optimize the file access performance, we have to consider all the possible access patterns and each pattern needs to be studied separately. In this presentation, we report on our work to optimize file access with the Lustre filesystem at the Florida CMS T2 using an approach based on analyzing these access patterns.

Observation has lead to a conclusion that the physics analysis jobs run by LHCb physicists on a local computing farm (i.e. non-grid) require more efficient access to the data which resides on the Grid. Our experiments have shown that the I/O bound nature of the analysis jobs in combination with the latency due to the remote access protocols (e.g. rfio, dcap) cause a low CPU efficiency of these jobs. In addition to causing a low CPU efficiency, the remote access protocols give rise to high overhead (in terms of amount of data transferred). This paper gives an overview of the concept of pre-fetching and caching of input files in the proximity of the processing resources, which is exploited to cope with the I/O bound analysis jobs. The files are copied from Grid storage elements (using GridFTP), while concurrently performing computations, inspired from a similar idea used in the ATLAS experiment. The results illustrate that this file staging approach is relatively insensitive to the original location of the data, and a significant improvement can be achieved in terms of the CPU efficiency of an analysis job. Dealing with scalability of such a solution on the Grid environment is discussed briefly.

We present a data transfer system for the grid environment built on top of the open source FDT tool (Fast Data Transfer) developed by Caltech in collaboration with the National University of Science and Technology (Pakistan). The enhancement layer above FDT consists of a client program - fdtcp (FDT copy) and a fdtd service (FDT daemon). This pair of components allows for GSI authenticated data transfers and offers to the user (or data movement production service) interface analogous to grid middle-ware data transfer services - SRM (i.e. srmcp) or GridFTP (i.e. globus-url-copy). fdtcp/fdtd enables third-party, batched file transfers. An important aspect is monitoring by means of the MonALISA active monitoring light-weight library ApMon, providing real-time monitoring and arrival time estimates as well as powerful troubleshooting mechanism. The actual transfer is carried out by the FDT application, an efficient application capable of reading and writing at disk speed over wide area networks. FDT's excellent performance was demonstrated e.g. during SuperComputing 2009 Bandwidth Challenge. We also discuss the storage technology interface layer, specifically focusing on the open source Hadoop distributed file system (HDFS), presenting the recently developed FDT-HDFS sequential write adapter. The integration with CMS PhEDEx is described as well. The PhEDEx project (Physics Experiment Data Export) is responsible for facilitating large-scale CMS data transfers across the grid. Ongoing and future development involves interfacing with next generation network services developed by OGF NSI-WG, GLIF and DICE groups, allowing for network resource reservation and scheduling.

Cloud computing has recently emerged as a new computing paradigm aiming at providing reliable, flexible, IT infrastructure and services based on externalized, virtual resources. In this context, users may upload Virtual Machine (VM) images into a Cloud storage service, from which they are propagated on demand to the physical nodes on which they are supposed to run. It is therefore important for the Cloud storage service to provide efficient support for VM storage in a context where a large number of clients may concurrently upload a large number of VMs, each of which may subsequently be needed by a large number of computing nodes. This paper addresses the problem of building such an efficient distributed repository for Cloud Virtual Machines. To meet this goal, our approach leverages BlobSeer, a system for efficient management of massive data concurrently accessed at a large-scale, as a storage back end for the Cloud VM repository. As a case study, we consider the Nimbus Cloud environment, whose repository currently relies on the GridFTP high-performance file transfer protocol. We have integrated BlobSeer as a back-end storage layer for GridFTP and evaluated our prototype on the Grid'5000 testbed.; Le Cloud computing a récemment émergé comme un nouveau paradigme qui vise à fournir une infrastructure informatique et des services fiables et flexibles en s'appuyant sur des ressources virtuelles. Dans ce contexte, les utilisateurs peuvent télécharger des images de machines virtuelles vers un service de stockage Cloud, d'où elles sont propagées sur demande vers les noeuds physiques sur lesquels elles vont être exécutées. En consequence, il est essentiel que le service de gestion des données Cloud permette le stockage efficace des machines virtuelles quand de nombreux utilisateurs téléchargent des machines virtuelles simultanément. Cet article porte sur le problème de la conception d'un tel service de stockage pour les machines virtuelles dans le Cloud. Notre approche s'appuie sur l'utilisation de BlobSeer, un système dédié à la gestion des données accedées par de nombreux clients à large échelle, comme back-end de stockage pour les machines virtuelles. Nous avons pris comme étude de cas la plate-forme Nimbus dont le système de stockage repose actuellement sur le protocole de transfert de fichiers GridFTP. Nous avons intégré BlobSeer comme couche de stockage pour GridFTP et nous avons évalué notre prototype sur la plate-forme Grid'5000.

—This paper presents the generalised Token Based Networking (gTBN) architecture, which enables dynamic binding of communities and their applications to specialised network services. gTBN uses protocol independent tokens to provide decoupling of authorisation from time of usage as well as identification of network traffic. The tokenised traffic allows specialised software components uploaded into network elements to execute services specific to communities. A reference implementation of gTBN over IPv4 is proposed as well as the presentation of our experiments. These experiments include validation tests of our test bed with common grid applications such as GridFTP, OpenMPI, and VLC. In addition, we present a firewalling use case based on gTBN.

This is the author’s version of a work that was accepted for publication in Optical Switching and Networking. Changes resulting from the publishing process, such as peer review, editing, corrections, structural formatting, and other quality control mechanisms may not be reflected in this document. Changes may have been made to this work since it was submitted for publication. A definitive version was subsequently published in Optical Switching and Networking, 6, 9 (2009) DOI: 10.1016/j.osn.2008.05.003, Selected papers from First International Symposium on Advanced Networks and Telecommunication Systems, ANTS 2007, Computing and networking resources virtualization is the main objective of Grid services. Such a concept is already used in the context of Web-services on the Internet. In the next few years, a large number of applications belonging to various domains (biotechnology, banking, finance, car and aircraft manufacturing, nuclear energy etc.) will also benefit from Grid services. Admission control is a key functionality for Quality of Service (QoS) provision in IP networks, and more specifically for Grid services provision. Service differentiation (DS) is a widely deployed technique on the Internet. It operates at the packet level on a best-effort mode. Flow-Aware Networking (FAN) that operates at the scale of the IP flows relies on implicit flow differentiation through priority fair queuing (PFQ). It may be seen as an alternative to DS. A Grid session may be seen as a succession of parallel TCP/IP flows characterized by data transfers with much larger volume than usual TCP/IP flows. In this paper, we propose an extension of FAN for the Grid environment called Grid over FAN (GoFAN). We compare, by means of computer simulations, the efficiency of Grid over DS (GoDS) and GoFAN. Two variants of GoFAN architectures based on different fair queuing algorithms are considered. As a first step, we provide two short surveys on QoS for Grid environment and on QoS in IP networks respectively.

This paper describes the grid-oriented storage (GOS) architecture and its implementations. A GOS- specific file system (GOS-FS), the single-purpose intent of a GOS OS, and secure interfaces via grid security infrastructure (GSI) motivate and enable this new architecture. As an FTP server, GOS with a slimmed OS, with a total volume of around 150 MB, outperforms the standard GridFTP by 20-40 percent. As a file server, GOS-FS acts as a network/grid interface, enabling a user to perform searches and access resources without downloading them locally. In the real-world tests between Cambridge and Beijing, where the transfer distance is 10, 000 km, the multistreamed GOS-FS file opening/saving resulted in a remarkable performance increase of about 2-25 times, compared to the single-streamed network file system (NFSv4). GOS is expected to be a variant of or successor to the well-used network-attached storage (NAS) and/or storage area network (SAN) products in the grid era.

Grid is a highly distributed heterogeneous environment which interoperates towards completing a common goal. As such, grids performance is highly dependant on the speed of data transfer between systems of which grid consists. Most commonly used and a “de facto” standard for data transfer on grid systems is GridFTP [7]. While working on CRO-GRID Infrastructure [3] project, we faced problems with small data transfer speeds when transferring small files. In this paper we present a program we have developed which uses GridFTP and speeds up data transfer of large number of small or medium sized files. We also present empirical results of the performance measurements achieved while using the implemented system.

The data transfer in the Grid at CERN (the European Organization for Nuclear Research) has seen constant improvement, be it through optimizing the existing tools, GridFTP and GFAL2 or by adding new tools such as XRootd. Unfortunately, all these have reached the maximum limit in terms of throughput. They are limited not by the network infrastructure, but by the fact that they use a single source for transfers, despite the existence of multiple replicas. In this paper we take on the challenge of evaluating the effects of using multiple sources, over the throughput, by comparing the download speed of the tools mentioned above with Aria2 and an under development version of XRootd, both supporting the use of multiple sources.