289 results on '"GridFTP"'
Search Results
2. SNAG: SDN-Managed Network Architecture for GridFTP Transfers Using Application-Awareness.
- Author
-
Nadig, Deepak, Ramamurthy, Byrav, and Bockelman, Brian
- Subjects
SOFTWARE-defined networking ,DISTRIBUTED computing ,QUALITY of service ,LASER interferometers ,WIDE area networks ,COMPUTER architecture - Abstract
Increasingly, academic campus networks support large-scale data transfer workflows for data-intensive science. These data transfers rely on high-performance, scalable, and reliable protocols for moving large amounts of data over a high-bandwidth, high-latency network. GridFTP is a widely used protocol for wide area network (WAN) data movement. However, as the GridFTP protocol does not share connection information with the network-layer, network operators have reduced flexibility, particularly in identifying/managing flows across the network. We address this problem by deploying a production “ application-aware ” software defined network (SDN) for managing GridFTP transfers for data-intensive science workflows. We first propose a novel application-aware architecture called SNAG (SDN-managed Network Architecture for GridFTP transfers). SNAG combines application-layer and network-layer collaboration (termed “application-awareness”) with SDN-enabled network management to classify, monitor and to manage network resources actively. Until now, our SNAG deployment has successfully classified over 1.5 Billion GridFTP connections at the Holland Computing Center (HCC), University of Nebraska-Lincoln (UNL). Next, we develop an application-aware SDN system to provide differentiated network services for distributed computing workflows. At HCC, we also demonstrate how our system ensures the quality of service (QoS) for high-throughput workflows such as Compact Muon Solenoid (CMS) and Laser Interferometer Gravitational-Wave Observatory (LIGO). Further, we also demonstrate how application-aware SDN can be exploited to create policy-driven approaches to achieve accurate resource accounting for each workflow. We present strategies for implementing differentiated network services and discuss their capacity improvement benefits. Lastly, we provide some guidelines and recommendations for developing application-aware SDN architectures for general-purpose applications. [ABSTRACT FROM AUTHOR]
- Published
- 2022
- Full Text
- View/download PDF
3. Development of Hadoop Massive Data Migration System
- Author
-
Zheng, Ming, Zhuo, Mugui, Kacprzyk, Janusz, Series Editor, Pal, Nikhil R., Advisory Editor, Bello Perez, Rafael, Advisory Editor, Corchado, Emilio S., Advisory Editor, Hagras, Hani, Advisory Editor, Kóczy, László T., Advisory Editor, Kreinovich, Vladik, Advisory Editor, Lin, Chin-Teng, Advisory Editor, Lu, Jie, Advisory Editor, Melin, Patricia, Advisory Editor, Nedjah, Nadia, Advisory Editor, Nguyen, Ngoc Thanh, Advisory Editor, Wang, Jun, Advisory Editor, Xu, Zheng, editor, Choo, Kim-Kwang Raymond, editor, Dehghantanha, Ali, editor, Parizi, Reza, editor, and Hammoudeh, Mohammad, editor
- Published
- 2020
- Full Text
- View/download PDF
4. GridFTP frontend with redirection for DMlite
- Author
-
A. K. Kiryanov
- Subjects
WLCG ,Grid ,GridFTP ,DPM ,DMLite ,data storage ,access protocol ,Applied mathematics. Quantitative methods ,T57-57.97 ,Mathematics ,QA1-939 - Abstract
One of the most widely used storage solutions in WLCG is a Disk Pool Manager (DPM) developed and supported by SDC/ID group at CERN. Recently DPM went through a massive overhaul to address scalability and extensibility issues of the old code. New system was called DMLite. Unlike the old DPM that was based on daemons, DMLite is arranged as a library that can be loaded directly by an application. This approach greatly improves performance and transaction rate by avoiding unnecessary inter-process communication via network as well as threading bottlenecks. DMLite has a modular architecture with its core library providing only the very basic functionality. Backends (storage engines) and frontends (data access protocols) are implemented as plug-in modules. Doubtlessly DMLite wouldn't be able to completely replace DPM without GridFTP as it is used for most of the data transfers in WLCG. In DPM GridFTP support was implemented in a Data Storage Interface (DSI) module for Globus GridFTP server. In DMLite an effort was made to rewrite a GridFTP module from scratch in order to take advantage of new DMLite features and also implement new functionality. The most important improvement over the old version is a redirection capability. With old GridFTP frontend a client needed to contact SRM on the head node in order to obtain a transfer URL (TURL) before reading or writing a file. With new GridFTP frontend this is no longer necessary: a client may connect directly to the GridFTP server on the head node and perform file I/O using only logical file names (LFNs). Data channel is then automatically redirected to a proper disk node. This renders the most often used part of SRM unnecessary, simplifies file access and improves performance. It also makes DMLite a more appealing choice for non-LHC VOs that were never much interested in SRM. With new GridFTP frontend it's also possible to access data on various DMLite-supported backends like HDFS, S3 and legacy DPM.
- Published
- 2015
- Full Text
- View/download PDF
5. GridFTP GUI: An Easy and Efficient Way to Transfer Data in Grid
- Author
-
Liu, Wantao, Kettimuthu, Rajkumar, Tieman, Brian, Madduri, Ravi, Li, Bo, Foster, Ian, Akan, Ozgur, Series editor, Bellavista, Paolo, Series editor, Cao, Jiannong, Series editor, Dressler, Falko, Series editor, Ferrari, Domenico, Series editor, Gerla, Mario, Series editor, Kobayashi, Hisashi, Series editor, Palazzo, Sergio, Series editor, Sahni, Sartaj, Series editor, Shen, Xuemin (Sherman), Series editor, Stan, Mircea, Series editor, Xiaohua, Jia, Series editor, Zomaya, Albert, Series editor, Coulson, Geoffrey, Series editor, Doulamis, Anastasios, editor, Mambretti, Joe, editor, Tomkos, Ioannis, editor, and Varvarigou, Theodora, editor
- Published
- 2010
- Full Text
- View/download PDF
6. Coordinated Co-allocator Model for Data Grid in Multi-sender Environment
- Author
-
Bhuvaneswaran, R. S., Katayama, Yoshiaki, Takahashi, Naohisa, Hutchison, David, editor, Kanade, Takeo, editor, Kittler, Josef, editor, Kleinberg, Jon M., editor, Mattern, Friedemann, editor, Mitchell, John C., editor, Naor, Moni, editor, Nierstrasz, Oscar, editor, Pandu Rangan, C., editor, Steffen, Bernhard, editor, Sudan, Madhu, editor, Terzopoulos, Demetri, editor, Tygar, Dough, editor, Vardi, Moshe Y., editor, Weikum, Gerhard, editor, Dan, Asit, editor, and Lamersdorf, Winfried, editor
- Published
- 2006
- Full Text
- View/download PDF
7. A Parallel Data Storage Interface to GridFTP
- Author
-
Sánchez, Alberto, Pérez, María S., Gueant, Pierre, Montes, Jesús, Herrero, Pilar, Hutchison, David, editor, Kanade, Takeo, editor, Kittler, Josef, editor, Kleinberg, Jon M., editor, Mattern, Friedemann, editor, Mitchell, John C., editor, Naor, Moni, editor, Nierstrasz, Oscar, editor, Pandu Rangan, C., editor, Steffen, Bernhard, editor, Sudan, Madhu, editor, Terzopoulos, Demetri, editor, Tygar, Dough, editor, Vardi, Moshe Y., editor, Weikum, Gerhard, editor, Meersman, Robert, editor, and Tari, Zahir, editor
- Published
- 2006
- Full Text
- View/download PDF
8. A Recursive-Adjustment Co-allocation Scheme in Data Grid Environments
- Author
-
Yang, Chao-Tung, Yang, I-Hsien, Li, Kuan-Ching, Hsu, Ching-Hsien, Hutchison, David, editor, Kanade, Takeo, editor, Kittler, Josef, editor, Kleinberg, Jon M., editor, Mattern, Friedemann, editor, Mitchell, John C., editor, Naor, Moni, editor, Nierstrasz, Oscar, editor, Pandu Rangan, C., editor, Steffen, Bernhard, editor, Sudan, Madhu, editor, Terzopoulos, Demetri, editor, Tygar, Dough, editor, Vardi, Moshe Y., editor, Weikum, Gerhard, editor, Hobbs, Michael, editor, Goscinski, Andrzej M., editor, and Zhou, Wanlei, editor
- Published
- 2005
- Full Text
- View/download PDF
9. Performance Analysis of Applying Replica Selection Technology for Data Grid Environments
- Author
-
Yang, Chao-Tung, Chen, Chun-Hsiang, Li, Kuan-Ching, Hsu, Ching-Hsien, Hutchison, David, editor, Kanade, Takeo, editor, Kittler, Josef, editor, Kleinberg, Jon M., editor, Mattern, Friedemann, editor, Mitchell, John C., editor, Naor, Moni, editor, Nierstrasz, Oscar, editor, Pandu Rangan, C., editor, Steffen, Bernhard, editor, Sudan, Madhu, editor, Terzopoulos, Demetri, editor, Tygar, Dough, editor, Vardi, Moshe Y., editor, Weikum, Gerhard, editor, and Malyshkin, Victor, editor
- Published
- 2005
- Full Text
- View/download PDF
10. Low-Variance Latency Through Forward Error Correction on Wide-Area Networks
- Author
-
Paul Lu and Nooshin Eghbal
- Subjects
business.industry ,Computer science ,Throughput ,GridFTP ,Forward error correction ,Variance (accounting) ,Latency (engineering) ,Macro ,business ,Computer network ,Data transmission - Abstract
High bandwidth-delay product (BDP) networks present many performance challenges. We present the design, implementation, and evaluation of UDT+FEC, a software system that provides high throughput, low latency, and low-variance latency on wide-area networks (WAN), especially for large data transfers. Using 2D-XOR forward error correction (FEC), implemented as an extension of the UDP-based Data Transfer (UDT) system, we show that it is possible to match the throughput and latency of other state-of-the-art tools, but also provide the added property of lower variance in the interarrival times of messages at the receiver. With a variety of macro- and micro-benchmarks, we quantify the relative advantages of UDT+FEC, when compared to using GridFTP combined with CUBIC, BBR, and parallel streams.We find that UDT+FEC has substantially lower variance in the latency of message arrivals (approximated by message interarrival times, for one-way transmissions) when compared to different combinations of CUBIC, BBR, and parallel streams.
- Published
- 2021
11. A High-Performance GriFTP Server at Desktop Cost
- Author
-
Vazhkudai, Sudharshan [ORNL]
- Published
- 2007
12. Transferring a petabyte in a day
- Author
-
Franck Cappello, Rajkumar Kettimuthu, Zhengchun Liu, Ian Foster, Katrin Heitmann, and D. J. Wheeler
- Subjects
Service (systems architecture) ,Computer Networks and Communications ,Computer science ,Volume (computing) ,Petabyte ,020206 networking & telecommunications ,02 engineering and technology ,GridFTP ,computer.software_genre ,Supercomputer ,Pipeline (software) ,Hardware and Architecture ,Transfer (computing) ,0202 electrical engineering, electronic engineering, information engineering ,Operating system ,020201 artificial intelligence & image processing ,computer ,Software - Abstract
Extreme-scale simulations and experiments can generate large amounts of data, whose volume can exceed the compute and/or storage capacity at the simulation or experimental facility. With the emergence of ultra-high-speed networks, researchers are considering pipelined approaches in which data are passed to a remote facility for analysis. Here we examine an extreme-scale cosmology simulation that, when run on a large fraction of a leadership computer, generates data at a rate of one petabyte per elapsed day. Writing those data to disk is inefficient and impractical, and in situ analysis poses its own difficulties. Thus we implement a pipeline in which data are generated on one supercomputer and then transferred, as they are generated, to a remote supercomputer for analysis. We use the Swift scripting language to instantiate this pipeline across Argonne National Laboratory and the National Center for Supercomputing Applications, which are connected by a 100 Gb/s network; and we demonstrate that by using the Globus transfer service we can achieve a sustained rate of 93 Gb/s over a 24-hour period, thus attaining our performance goal of one petabyte moved in 24 h. This paper describes the methods used and summarizes the lessons learned in this demonstration.
- Published
- 2018
13. Systematic benchmarking of HTTPS third party copy on 100Gbps links using XRootD
- Author
-
Brian Bockelman, F. Würthwein, Edgar Fajardo, Diego Davila, Aashay Arora, and Richard Gao
- Subjects
FOS: Computer and information sciences ,Protocol (science) ,Third party ,010308 nuclear & particles physics ,Physics ,QC1-999 ,Volume (computing) ,High Luminosity Large Hadron Collider ,Benchmarking ,GridFTP ,computer.software_genre ,01 natural sciences ,Scheduling (computing) ,Computer Science - Distributed, Parallel, and Cluster Computing ,Transfer (computing) ,0103 physical sciences ,Operating system ,Distributed, Parallel, and Cluster Computing (cs.DC) ,010306 general physics ,computer - Abstract
The High Luminosity Large Hadron Collider provides a data challenge. The amount of data recorded from the experiments and transported to hundreds of sites will see a thirty fold increase in annual data volume. A systematic approach to contrast the performance of different Third Party Copy(TPC) transfer protocols arises. Two contenders, XRootD-HTTPS and the GridFTP are evaluated in their performance for transferring files from one server to an-other over 100Gbps interfaces. The benchmarking is done by scheduling pods on the Pacific Research Platform Kubernetes cluster to ensure reproducible and repeatable results. This opens a future pathway for network testing of any TPC transfer protocol., 7 pages, 8 figures
- Published
- 2021
14. A Parallel Data Stream Layer for Large Data Workloads on WANs
- Author
-
Nooshin Eghbal and Paul Lu
- Subjects
business.industry ,Computer science ,computer.internet_protocol ,Data stream mining ,020206 networking & telecommunications ,Throughput ,02 engineering and technology ,GridFTP ,Internet protocol suite ,Packet loss ,020204 information systems ,0202 electrical engineering, electronic engineering, information engineering ,File transfer ,Network File System ,business ,Virtual network ,computer ,Computer network - Abstract
A variety of workloads on wide-area networks (WAN) can benefit from parallel data streams. Whether between public and private clouds or between high-performance computing centres, high bandwidth-delay-product (BDP) networks with even small amounts of packet loss (e.g., due to congestion) can suffer reduced TCP/IP throughput. Therefore, we design, implement, and evaluate the open-source Parallel Data Streams (PDS) user-level tool. PDS makes trade-offs (e.g., TCP fairness) for specific workloads, when performance is the key goal. Unlike the well-known GridFTP, PDS also supports tools such as rsync, git, Virtual Network Computing (VNC), and the Network File System (NFS). We also quantify and contribute a performance evaluation of PDS, using a combination of emulated and real WANs. We establish that PDS achieves comparable performance to GridFTP for file transfer, but additional functionality via other tools. For example, PDS can transfer a 14 GB file on a WAN between Alberta and Quebec (maximum 1 Gbps; over 3,100 km) at 861 Mbps, using rsync and 8 parallel, cleartext, TCP streams. In comparison, rsync over a single SSH stream achieves 274 Mbps.
- Published
- 2020
15. Instant GridFTP.
- Author
-
Kettimuthu, Rajkumar, Lacinski, Lukasz, Link, Mike, Pickett, Karl, Tuecke, Steve, and Foster, Ian
- Abstract
A foundational need in high-performance computing is to move large (multi-gigabyte and even terabyte) datasets between sites. Simple file transfer mechanisms such as FTP and SCP are not sufficient from either a reliability or performance perspective. GridFTP is the de facto standard protocol for transferring large data files in production Grid/HPC environments. GridFTP extends the standard FTP protocol to provide a high-performance, secure, reliable data transfer protocol optimized for high-bandwidth wide-area networks. The Glob us GridFTP implementation has become the preeminent high-performance data transfer tool for the Grid community, with large facilities and projects using it to transfer billions of files per year. We report here on a new product, Glob us Connect Multi User (GCMU), that greatly streamlines Glob us GridFTP installation and configuration. GCMU packages a GridFTP server, My Proxy Online Certificate Authority, and other components in a manner that avoids the need for any end-user or system administrator involvement in security configuration or credential management. We describe the GCMU design and a GridFTP protocol extension that simplifies transfers across security domains. We also explain how GCMU interacts with the Glob us Online software-as-a-service solution. By enabling "instant GridFTP," this work makes the powerful Glob us GridFTP tool accessible to nonexpert users and to smaller laboratories and projects. [ABSTRACT FROM PUBLISHER]
- Published
- 2012
- Full Text
- View/download PDF
16. The BNLBox Cloud Storage Service
- Author
-
Mizuki Karasawa, Ognian Novakov, Hironori Ito, Robert Hancock, Tim Chou, Tejas Rao, Ofer Rind, Alexandr Zaytsev, Zhenping Liu, Yingzi Wu, and Guangwei Che
- Subjects
Database ,business.industry ,Data management ,Interface (computing) ,Physics ,QC1-999 ,GridFTP ,computer.software_genre ,Upload ,Data access ,POSIX ,Synchronization (computer science) ,business ,computer ,Cloud storage - Abstract
Large scientific data centers have recently begun providing a number of different types of data storage in order to satisfy the various needs of their users. Users with interactive accounts, for example, might want a POSIX interface for easy access to the data from their interactive machines. Grid computing sites, on the other hand, likely need to provide an X509-based storage protocol, like SRM and GridFTP, since the data management system is built upon them. Meanwhile, an experiment producing large amounts of data typically demands a service that provides archival storage for the safe keeping of their unique data. To access these various types of data, users must use specific sets of commands tailored to their respective storage, making access to their data complex and difficult. BNLBox is an attempt to provide a unified and easy to use storage service for all BNL users, to store their important documents, code and data. It is a cloud storage system with an intuitive web interface for novice users. It provides an automated synchronization feature that enables users to upload data to their cloud storage without manual intervention, freeing them to focus on analysis rather than data management software. It provides a POSIX interface for local interactive users, which simplifies data access from batch jobs as well. At the same time, it also provides users with a straightforward mechanism for archiving large data sets for later processing. The storage space can be used for both code and data within the compute job environment. This paper will describe various aspects of the BNLBox storage service.
- Published
- 2020
17. XRootD and Object Store: A new paradigm
- Author
-
George Patargias, Katy Ellis, Robert Appleyard, Tim Adye, Chris Brew, Alastair Dewhurst, and Ian Johnson
- Subjects
Service (systems architecture) ,010308 nuclear & particles physics ,Gateway (telecommunications) ,Node (networking) ,Physics ,QC1-999 ,Echo (computing) ,GridFTP ,computer.software_genre ,01 natural sciences ,Software framework ,Data access ,Server ,0103 physical sciences ,Operating system ,010306 general physics ,computer - Abstract
The XRootD software framework is essential for data access at WLCG sites. The WLCG community is exploring and expanding XRootD functionality. This presents a particular challenge at the RAL Tier-1 as the Echo storage service is a Ceph based Erasure Coded object store. External access to Echo uses gateway machines which run GridFTP and caching servers. Local jobs access Echo via caches on every worker node, but it is clear there are inefficiencies in the system. Remote jobs also access data via XRootD on Echo. For CMS jobs this is via the AAA service. ATLAS, who are consolidating their storage at fewer sites, are increasingly accessing job input data remotely. This paper describes the continuing work to optimise both local and remote data access by testing different caching methods.
- Published
- 2020
18. Third-party transfers in WLCG using HTTP
- Author
-
Brian Bockelman, Paul Millar, Alessandra Forti, Andrea Ceccanti, Dmitry Litvintsev, and Fabrizio Furano
- Subjects
FOS: Computer and information sciences ,cs.DC ,File Transfer Protocol ,Physics ,QC1-999 ,Interoperability ,020207 software engineering ,02 engineering and technology ,GridFTP ,Security token ,computer.software_genre ,01 natural sciences ,OpenID Connect ,Computing and Computers ,010305 fluids & plasmas ,Computer Science - Distributed, Parallel, and Cluster Computing ,Server ,0103 physical sciences ,WebDAV ,0202 electrical engineering, electronic engineering, information engineering ,Operating system ,Distributed, Parallel, and Cluster Computing (cs.DC) ,Protocol (object-oriented programming) ,computer - Abstract
Since its earliest days, the Worldwide LHC Computational Grid (WLCG) has relied on GridFTP to transfer data between sites. The announcement that Globus is dropping support of its open source Globus Toolkit (GT), which forms the basis for several FTP client and servers, has created an opportunity to reevaluate the use of FTP. HTTP-TPC, an extension to HTTP compatible with WebDAV, has arisen as a strong contender for an alternative approach. In this paper, we describe the HTTP-TPC protocol itself, along with the current status of its support in different implementations, and the interoperability testing done within the WLCG DOMA working group's TPC activity. This protocol also provides the first real use-case for token-based authorisation for this community. We will demonstrate the benefits of such authorisation by showing how it allows HTTP-TPC to support new technologies (such as OAuth, OpenID Connect, Macaroons and SciTokens) without changing the protocol. We will also discuss the next steps for HTTP-TPC and the plans to use the protocol for WLCG transfers., 7 pages, 3 figures, to appear in the proceedings of CHEP 2020
- Published
- 2019
- Full Text
- View/download PDF
19. Bridging the gap between peak and average loads on science networks
- Author
-
Sam Nickolay, Ian Foster, Rajkumar Kettimuthu, and Eun-Sung Jung
- Subjects
Bridging (networking) ,010504 meteorology & atmospheric sciences ,Computer Networks and Communications ,Slowdown ,Computer science ,Testbed ,020206 networking & telecommunications ,02 engineering and technology ,GridFTP ,01 natural sciences ,Hardware and Architecture ,Server ,0202 electrical engineering, electronic engineering, information engineering ,Software ,Simulation ,0105 earth and related environmental sciences - Abstract
Backbone networks are typically overprovisioned in order to support peak loads. Research and education networks (RENs), for example, are often designed to operate at 20–30% of capacity. Thus, Internet2 upgrades its backbone interconnects when the weekly 95th-percentile load is reliably above 30% of link capacity, and analysis of ESnet traffic between major laboratories shows a substantial gap between peak and average utilization. As science data volumes increase exponentially, it is unclear whether this overprovisioning trend can continue into the future. Even if overprovisioning is possible, it may not be the most cost-effective (and desirable) approach going forward. Under the current mode of free access to RENs, traffic at peak load may include both flows that need to be transferred in near-real time–for example, for computation and instrument monitoring and steering–and flows that are less time-critical, for example, archival and storage replication operations. Thus, peak load does not necessarily indicate the capacity that is absolutely required at that moment. We thus examine how data transfers are impacted when the average network load is increased while the network capacity is kept at the current levels. We also classify data transfers into on-demand (time-critical) and best-effort (less time-critical) and study the impact on both classes for different proportions of both the number of on-demand transfers and amount of bandwidth allocated for on-demand transfers. For our study, we use real transfer logs from production GridFTP servers to do simulation-based experiments as well as real experiments on a testbed. We find that when the transfer load is doubled and the network capacity is fixed at the current level, the gap between peak and average throughput decreases by an average of 18% in the simulation experiments and 16% in the testbed experiments, and the average slowdown experienced by the data transfers is under 1.5×. Furthermore, when transfers are classified as on-demand or best-effort, on-demand transfers experience almost no slowdown and the mean slowdown experienced by best-effort transfers is under 2× in the simulation experiments and under 1.2× in the testbed experiments.
- Published
- 2018
20. Maximizing the performance of scientific data transfer by optimizing the interface between parallel file systems and advanced research networks
- Author
-
F. Alex Feltus, Walter B. Ligon, and Nicholas Mills
- Subjects
0301 basic medicine ,File system ,Computer Networks and Communications ,Computer science ,Distributed computing ,Interface (computing) ,05 social sciences ,InfiniBand ,GridFTP ,STREAMS ,computer.software_genre ,03 medical and health sciences ,030104 developmental biology ,Hardware and Architecture ,0502 economics and business ,computer ,050203 business & management ,Software ,Data transmission ,BeeGFS - Abstract
The large amount of time spent transferring experimental data in fields such as genomics is hampering the ability of scientists to generate new knowledge. Often, computer hardware is capable of faster transfers but sub-optimal transfer software and configurations are limiting performance. This work seeks to serve as a guide to identifying the optimal configuration for performing genomics data transfers. A wide variety of tests narrow in on the optimal data transfer parameters for parallel data streaming across Internet2 and between two CloudLab clusters loading real genomics data onto a parallel file system. The best throughput was found to occur with a configuration using GridFTP with at least 5 parallel TCP streams with a 16 MiB TCP socket buffer size to transfer to/from 4–8 BeeGFS parallel file system nodes connected by InfiniBand.
- Published
- 2018
21. mdtmFTP and its evaluation on ESNET SDN testbed
- Author
-
Wenji Wu, Phil DeMar, Liang Zhang, and Eric Pouyoul
- Subjects
Multi-core processor ,Computer Networks and Communications ,Computer science ,business.industry ,Testbed ,Big data ,020206 networking & telecommunications ,02 engineering and technology ,GridFTP ,computer.software_genre ,Computer architecture ,Hardware and Architecture ,Asynchronous communication ,0202 electrical engineering, electronic engineering, information engineering ,Operating system ,Batch processing ,020201 artificial intelligence & image processing ,business ,computer ,Software - Abstract
To address the high-performance challenges of data transfer in the big data era, we are developing and implementing mdtmFTP: a high-performance data transfer tool for big data. mdtmFTP has four salient features. First, it adopts an I/O centric architecture to execute data transfer tasks. Second, it more efficiently utilizes the underlying multicore platform through optimized thread scheduling. Third, it implements a large virtual file mechanism to address the lots-of-small-files (LOSF) problem. Finally, mdtmFTP integrates multiple optimization mechanisms, includingzero copy, asynchronous I/O, pipelining, batch processing, and pre-allocated buffer poolsto enhance performance. mdtmFTP has been extensively tested and evaluated within the ESNET 100G testbed. Evaluations show that mdtmFTP can achieve higher performance than existing data transfer tools, such as GridFTP, FDT, and BBCP. An I/O centric architecture is proposed to execute data transfer tasks.Optimized thread scheduling makes more efficient utilization of the underlying multicore platform.A large virtual file mechanism is implemented to address the lots-of-small-files (LOSF) problem.Multiple optimization mechanisms, includingzero copy, asynchronous I/O, pipelining, batch processing, and pre-allocated buffer poolsare integrated to enhance performance.
- Published
- 2018
22. MataNui–A Distributed Storage Infrastructure for Scientific Data.
- Author
-
Kloss, Guy K.
- Subjects
DISTRIBUTION (Probability theory) ,INTERNET ,SCIENCE databases ,DATA management ,REMOTE access networks ,COMPUTER network protocols - Abstract
In science and engineering the problem of űsanityƇ in data management is quite common. Particularly, if partners within a project are geographically distributed, and require access to this data. These partners would ideally like to access or store data using a local server, while still retaining remote access. Furthermore, data does not űlive alone,Ƈ but has a űpartnerƇ: Meta- data. Currently available solutions cater for different needs, and do not fully address the problems in our own environment. Particularly, seamless integration within a Grid infrastructure is essential. For these reasons, the presented MataNui data server infrastructure offers full storage of files along with arbitrarily exten- sive meta-data for each stored item. The server can replicate its content between geographically distributed locations, making each usable for full read/write access, enabling researchers to use the one closest to their own network. Files are accessible through GridFTP as the predominant protocol for Grid Computing. For full access to meta-data and server side search queries a (RESTful) Web Service is provided. The current implementation features very high performance in data throughput on storage, retrieval and query operations. Through the Web Service front end it is straight forward to integrate it with scientific workflow environments or scientific data management applications. [ABSTRACT FROM AUTHOR]
- Published
- 2013
- Full Text
- View/download PDF
23. AN ADAPTIVE META-SCHEDULER FOR DATA-INTENSIVE APPLICATIONS IN GRID.
- Author
-
Vimala S. and Sasikala T.
- Subjects
GRID computing ,INFORMATION retrieval ,INTERNET administration - Abstract
Grid computing becomes the de facto platform for scientific computations that incorporates geographically and organizationally dispersed, heterogeneous resources. These scientific and data intensive computations require large and multiple datasets to be transferred to the data storage and compute nodes. As there is rapid growth in communication through internet, the data transfer becomes the major bottleneck for the end-to-end performance for these scientific applications. A most practical way of increasing the throughput is using multiple parallel streams. Currently GridFTP protocol is designed for point-to-single point parallel data transfer. However the issue of simultaneous -multiple files to multiple locations is not studied so far. In this paper, we design an optimized Meta-scheduler by which multiple files can be transferred simultaneously to the destined compute nodes. A LBLC scheduling algorithm is designed to transfer multiple files to multiple locations simultaneously. A greedy method is followed at every stage. The Optimized proposed model gives better results compared to the non-optimized data transfer. [ABSTRACT FROM AUTHOR]
- Published
- 2013
- Full Text
- View/download PDF
24. RACAM: design and implementation of a recursively adjusting co-allocation method with efficient replica selection in Data Grids.
- Author
-
Yang, Chao-Tung, Yang, I-Hsien, and Chen, Chun-Hsiang
- Subjects
GRID computing ,BIOINFORMATICS ,BANDWIDTHS ,CLIENT/SERVER computing ,DOWNLOADING - Abstract
Data Grids enable the sharing, selection, and connection of a wide variety of geographically distributed computational and storage resources for addressing large-scale data-intensive scientific application needs in, for instance, high-energy physics, bioinformatics, and virtual astrophysical observatories. Data sets are replicated in Data Grids and distributed among multiple sites. Unfortunately, data sets of interest sometimes are significantly large in size, and may cause access efficiency overhead. A co-allocation architecture was developed in order to enable parallel downloading of data sets from multiple servers. Several co-allocation strategies have been coupled and used to exploit download rate by specifying among various client-server divides files into multiple blocks of equal sizes to link and address dynamic rate fluctuations. However, one major obstacle, the idle time of faster servers having to wait for the slowest server to deliver the final block, makes it important to reduce differences in finishing time among replica servers. In this paper, we propose a dynamic co-allocation method, called Recursively Adjusting Co-Allocation Method (RACAM), to improve the performance of parallel data file transfer. Our approach reduces the idle time spent waiting for the slowest server and decreases data transfer completion time. We also provide an effective scheme for reducing the cost of reassembling data blocks. Copyright © 2010 John Wiley & Sons, Ltd. [ABSTRACT FROM AUTHOR]
- Published
- 2010
- Full Text
- View/download PDF
25. Optimization of Data Transfer for Grid Using GridFTP.
- Author
-
Radic, Branimir, Kajic, Vedran, and Imamagic, Emir
- Subjects
DISTRIBUTED computing ,COMPUTER systems ,DATABASES ,DATA transmission systems ,STANDARDS ,COMPUTER files - Abstract
Grid is a highly distributed heterogeneous environment which interoperates towards completing a common goal. As such, grids performance is highly dependant on the speed of data transfer between systems of which grid consists. Most commonly used and a "de facto" standard for data transfer on grid systems is GridFTP. While working on CRO-GRID Infrastructure project, we faced problems with small data transfer speeds when transferring small files. In this paper we present a program we have developed which uses GridFTP and speeds up data transfer of large number of small or medium sized files. We also present empirical results of the performance measurements achieved while using the implemented system. [ABSTRACT FROM AUTHOR]
- Published
- 2007
- Full Text
- View/download PDF
26. Improving GridFTP transfers by means of a multiagent parallel file system.
- Author
-
Sánchez, Alberto, Pérez, María S., Montes, Jesús, Herrero, Pilar, and Cortes, Toni
- Subjects
COMPUTER systems ,COMPUTER input-output equipment ,DATA transmission systems ,INTERNET servers ,LATENT structure analysis - Abstract
Data transfer in a grid environment has become one critical activity in a large number of applications that require access to huge volumes of data. In these scenarios, characterized by large latencies, poor performance and complex dependencies, the use of approaches such as multiagents or parallel I/O can provide a great benefit. However, all the attempts to improve the performance of data transfer in grids should achieve the interoperability with already established data transfer schemes. GridFTP is one of the most known and used data grid transfer protocols. This paper describes MAPFS-DSI, a modification of the GridFTP server, based on a multiagent parallel file system. MAPFS-DSI increases the performance of data transfers, but keeping the interoperability with existing GridFTP servers. [ABSTRACT FROM AUTHOR]
- Published
- 2007
27. MyCoG.NET: a multi-language CoG toolkit.
- Author
-
Paventhan, A., Takeda, Kenji, Cox, Simon J., and Nicole, Denis A.
- Subjects
WIDE area networks ,PYTHON programming language ,WORKFLOW software ,GRID computing ,MICROSOFT .NET Framework - Abstract
Grid application developers utilize Commodity Grid (CoG) toolkits to access Globus Grid services. Existing CoG toolkits are language-specific and have, for example, been developed for Java, Python and the Matlab scripting environment. In this paper we describe MyCoG.NET, a CoG toolkit supporting multi-language programmability under the Microsoft .NET framework. MyCoG.NET provides a set of classes and APIs to access Globus Grid services from languages supported by the .NET Common Language Runtime. We demonstrate its programmability using FORTRAN, C
++ , C# and Java, and discuss its performance over LAN and WAN infrastructures. We present a Grid application, in the field of experimental aerodynamics, as a case study to show how MyCoG.NET can be exploited. We demonstrate how scientists and engineers can create and use domain-specific workflow activity sets for rapid application development using Windows Workflow Foundation. We also show how users can easily extend and customize these activities. [ABSTRACT FROM AUTHOR]- Published
- 2007
- Full Text
- View/download PDF
28. Improvements on dynamic adjustment mechanism in co-allocation data grid environments.
- Author
-
Yang, Chao-Tung, Yang, I-Hsien, Li, Kuan-Ching, and Wang, Shih-Yu
- Subjects
- *
DATA compression , *GRID computing , *DISTRIBUTED computing , *CODING theory , *DATABASE management , *ELECTRONIC file management , *COMPUTER systems , *HIGH performance computing , *ELECTRONIC data processing - Abstract
Several co-allocation strategies have been coupled and used to exploit rate differences among various client-server links and to address dynamic rate fluctuations by dividing files into multiple blocks of equal sizes. However, a major obstacle, the idle time of faster servers having to wait for the slowest server to deliver the final block, makes it important to reduce differences in finishing time among replica servers. In this paper, we propose a dynamic co-allocation scheme, namely Recursive-Adjustment Co-Allocation scheme, to improve the performance of data transfer in Data Grids. Our approach reduces the idle time spent waiting for the slowest server and decreases data transfer completion time. [ABSTRACT FROM AUTHOR]
- Published
- 2007
- Full Text
- View/download PDF
29. Grid-Oriented Storage: A Single-Image, Cross-Domain, High-Bandwidth Architecture.
- Author
-
Frank Zhigang Wang, Sining Wu, Na Helian, Parker, Michael Andrew, Yike Guo, Yuhui Deng, and Khare, Vineet R.
- Subjects
- *
INFORMATION retrieval , *BANDWIDTHS , *COMPUTER architecture , *FILE Transfer Protocol (Computer network protocol) , *CLIENT/SERVER computing , *DATA transmission systems , *NETWORK File System (Computer network protocol) , *BIT rate , *COMPUTER networks - Abstract
This paper describes the Grid-Oriented Storage (GOS) architecture and its implementations. A GOS-specific File System (GOS-FS), the single-purpose intent of a GOS OS, and secure interfaces via Grid Security Infrastructure (GSI) motivate and enable this new architecture. As an FTP server, GOS with a slimmed OS, with a total volume of around 150 MB, outperforms the standard GridFTP by 20-40 percent. As a file server, GOS-FS acts as a network/grid interface, enabling a user to perform searches and access resources without downloading them locally. In the real-world tests between Cambridge and Beijing, where the transfer distance is 10,000 km, the multistreamed GOS-FS file opening/saving resulted in a remarkable performance increase of about 2-25 times, compared to the single-streamed Network File System (NFSv4). GOS is expected to be a variant of or successor to the well-used network-attached storage (NAS) and/or storage area network (SAN) products in the Grid era. [ABSTRACT FROM AUTHOR]
- Published
- 2007
- Full Text
- View/download PDF
30. Lark: An effective approach for software-defined networking in high throughput computing clusters
- Author
-
Dale W. Carder, Zhe Zhang, Todd Tannenbaum, and Brian Bockelman
- Subjects
Ethernet ,Bandwidth management ,Computer Networks and Communications ,Computer science ,Network virtualization ,020206 networking & telecommunications ,02 engineering and technology ,GridFTP ,Network layer ,computer.software_genre ,Grid ,Scheduling (computing) ,Hardware and Architecture ,Wide area network ,Network address ,0202 electrical engineering, electronic engineering, information engineering ,Systems architecture ,Operating system ,020201 artificial intelligence & image processing ,High-throughput computing ,Software-defined networking ,computer ,Software ,Private network - Abstract
High throughput computing (HTC) systems are widely adopted in scientific discovery and engineering research. They are responsible for scheduling submitted batch jobs to utilize the cluster resources. Current systems mostly focus on managing computing resources like CPU and memory; however, they lack flexible and fine-grained management mechanisms for network resources. This has increasingly been an urgent need as current batch systems may be distributed among dozens of sites around the globe like Open Science Grid. The Lark project was motivated by this need to re-examine how the HTC layer interacts with the network layer. In this paper, we present the system architecture of Lark and its implementation as a plugin of HTCondor which is a popular HTC software project. Lark achieves lightweight network virtualization at per-job granularity for HTCondor by utilizing Linux container and virtual Ethernet devices; this provides each batch job with a unique network address in a private network namespace. We extended HTCondor’s description language, ClassAds, so users can specify networking requirements in the job submission script. HTCondor can perform matchmaking to make sure user-specified network requirements and resource-specific policies are fulfilled. We also extended the job agent, condor_starter , so that it can manage and configure the job’s network environment. Given this important building block as the core, we implement bandwidth management functionality at both the host and network levels utilizing software-defined networking (SDN). In addition to HTCondor, Wide area network bandwidth management for GridFTP traffic is designed and implemented. Our experiments and evaluations show that Lark can effectively manage network resources simultaneously for both applications inside the cluster environment. By not resorting to heavyweight VMs, we keep startup overheads minimal compared to “regular” batch jobs. This mechanism provides the users with better predictability of their job execution and the administrators more policy flexibility in allocation of network resources.
- Published
- 2017
31. RAMSYS: Resource-Aware Asynchronous Data Transfer with Multicore SYStems
- Author
-
Shudong Jin, Tan Li, Dantong Yu, and Yufei Ren
- Subjects
Input/output ,020203 distributed computing ,Multi-core processor ,Computer science ,business.industry ,Distributed computing ,Pipeline (computing) ,Locality ,020206 networking & telecommunications ,Cloud computing ,02 engineering and technology ,GridFTP ,Thread (computing) ,Replication (computing) ,Scheduling (computing) ,Instruction set ,Data access ,Computational Theory and Mathematics ,Hardware and Architecture ,Asynchronous communication ,Signal Processing ,Scalability ,0202 electrical engineering, electronic engineering, information engineering ,business ,Communications protocol - Abstract
High-speed data transfer is vital to data-intensive computing that often requires moving large data volumes efficiently within a local data center and among geographically dispersed facilities. Effective utilization of the abundant resources in modern multicore environments for data transfer remains a persistent challenge, particularly, for Non-Uniform Memory Access (NUMA) systems wherein the locality of data accessing is an important factor. This requires rethinking how to exploit parallel access to data and to optimize the storage and network I/Os. We address this challenge and present a novel design of asynchronous processing and resource-aware task scheduling in the context of high-throughput data replication. Our software allocates multiple sets of threads to different stages of the processing pipeline, including storage I/O and network communication, based on their capacities. Threads belonging to each stage follow an asynchronous model, and attain high performance via multiple locality-aware and peer-aware mechanisms, such as task grouping, buffer sharing, affinity control and communication protocols. Our design also integrates high performance features to enhance the scalability of data transfer in several scenarios, e.g., file-level sorting, block-level asynchrony, and thread-level pipelining. Our experiments confirm the advantages of our software under different types of workloads and dynamic environments with contention for shared resources, including a 28-160 percent increase in bandwidth for transferring large files, 1.7-66 times speed-up for small files, and up to 108 percent larger throughput for mixed workloads compared with three state of the art alternatives, GridFTP , BBCP and Aspera .
- Published
- 2017
32. Building an open source cloud environment with auto-scaling resources for executing bioinformatics and biomedical workflows
- Author
-
Oscar Torreno, Dieter Kranzlmüller, Oswaldo Trelles, and Michael Krieger
- Subjects
0301 basic medicine ,Service (systems architecture) ,SIMPLE (military communications protocol) ,Computer Networks and Communications ,business.industry ,Computer science ,Software as a service ,Distributed computing ,Cloud computing ,02 engineering and technology ,GridFTP ,Bioinformatics ,03 medical and health sciences ,030104 developmental biology ,Workflow ,Hardware and Architecture ,0202 electrical engineering, electronic engineering, information engineering ,020201 artificial intelligence & image processing ,business ,Software - Abstract
Fast and simple access to computing resources for in-silico experiments and processing of large amounts of data have become crucial factors for researchers in the bioinformatics and biomedical domains. Cloud computing offers the possibility of accessing computing resources for a flexible amount of time and with varying requirements. We discuss how a full cloud stack ranging from Infrastructure-as-a-Service (IaaS) via Platform-as-a-Service (PaaS) to Software-as-a-Service (SaaS) can be built on open source technologies including the integration of high-throughput data transfers in all service types. Based on the scaling capabilities provided by the tools used on the IaaS level we devise a strategy to build a dynamically scaling PaaS offering for building and running workflows using Galaxy. Finally, applications — available as SaaS offerings — for two distinct use cases from the bioinformatics and biomedical domains are presented to demonstrate the feasibility and performance of our solution.
- Published
- 2017
33. The Impact of Large-Data Transfers in Shared Wide-Area Networks: An Empirical Study
- Author
-
Paul Lu and Hamidreza Anvari
- Subjects
Computer science ,business.industry ,Distributed computing ,020206 networking & telecommunications ,Workload ,02 engineering and technology ,GridFTP ,Empirical research ,Wide area ,Analytics ,Data file ,0202 electrical engineering, electronic engineering, information engineering ,General Earth and Planetary Sciences ,Network File System ,020201 artificial intelligence & image processing ,business ,General Environmental Science - Abstract
Computational science sometimes requires large data files to be transferred over high bandwidth-delay-product (BDP) wide-area networks (WANs). Experimental data (e.g., LHC, SKA), analytics logs, and filesystem backups are regularly transferred between research centres and between private-public clouds. Fortunately, a variety of tools (e.g., GridFTP, UDT, PDS) have been developed to transfer bulk data across WANs with high performance. However, using large-data transfer tools could adversely affect other network applications on shared networks. Many of the tools explicitly ignore TCP fairness to achieve high performance. Users have experienced high-latency and low-bandwidth situations when a large-data transfer is underway. But there have been few empirical studies that quantify the impact of the tools. As an extension of our previous work using synthetic background traffic, we perform an empirical analysis of how the bulk-data transfer tools perform when competing with a non-synthetic, application-based workload (e.g., Network File System). Conversely, we characterize and show that, for example, NFS performance can drop from 29 Mb/s to less than 10 Mb/s (for a single stream) when competing with bulk-data transfers on a shared network.
- Published
- 2017
34. High-performance remote access to climate simulation data: a challenge problem for data grid technologies
- Author
-
Chervenak, Ann, Deelman, Ewa, Kesselman, Carl, Allcock, Bill, Foster, Ian, Nefedova, Veronika, Lee, Jason, Sim, Alex, Shoshani, Arie, Drach, Bob, Williams, Dean, and Middleton, Don
- Subjects
- *
REMOTE access networks , *COMPUTER input-output equipment , *MODELS & modelmaking , *CLIMATOLOGY - Abstract
In numerous scientific disciplines, terabyte and petabyte-scale data collections are emerging as critical community resources. A new class of “data grid” infrastructure is required to support management, transport, distributed access to, and analysis of these datasets by potentially thousands of users. Researchers who face this challenge include the climate modeling community, which performs long-duration computations accompanied by frequent output of very large files that must be further analyzed. We describe the Earth System Grid-I prototype, which brings together advanced analysis, replica management, data transfer, request management, and other technologies to support high-performance, interactive analysis of replicated data. We present performance results that demonstrate our ability to manage the location and movement of large datasets from the user’s desktop. We report on experiments conducted over SciNET at SC’2000, where we achieved peak performance of 1.55 Gb/s and sustained performance of 512.9 Mb/s for data transfers between Texas and California. Finally, we describe the development of the next-generation Earth System Grid-II (ESG-II) project. Important issues for ESG-II include security requirements for production environments, efficient data filtering and transport, metadata services for discovery of relevant climate datasets, and sophisticated request or workflow management for complex tasks. [Copyright &y& Elsevier]
- Published
- 2003
- Full Text
- View/download PDF
35. Data management and transfer in high-performance computational grid environments
- Author
-
Allcock, Bill, Bester, Joe, Bresnahan, John, Chervenak, Ann L., Foster, Ian, Kesselman, Carl, Meder, Sam, Nefedova, Veronika, Quesnel, Darcy, and Tuecke, Steven
- Subjects
- *
INFORMATION storage & retrieval systems , *MANAGEMENT - Abstract
An emerging class of data-intensive applications involve the geographically dispersed extraction of complex scientific information from very large collections of measured or computed data. Such applications arise, for example, in experimental physics, where the data in question is generated by accelerators, and in simulation science, where the data is generated by supercomputers. So-called Data Grids provide essential infrastructure for such applications, much as the Internet provides essential services for applications such as e-mail and the Web. We describe here two services that we believe are fundamental to any Data Grid: reliable, high-speed transport and replica management. Our high-speed transport service, GridFTP, extends the popular FTP protocol with new features required for Data Grid applications, such as striping and partial file access. Our replica management service integrates a replica catalog with GridFTP transfers to provide for the creation, registration, location, and management of dataset replicas. We present the design of both services and also preliminary performance results. Our implementations exploit security and other services provided by the Globus Toolkit. [Copyright &y& Elsevier]
- Published
- 2002
- Full Text
- View/download PDF
36. Implementation of Research Data Platform: in the Perspective of Data Transfer
- Author
-
Min Ki Kim, Min-Ho Lee, Jin Young Kim, YoungHo Shin, and Sungho Shin
- Subjects
Open science ,Government ,Knowledge management ,Computer science ,business.industry ,05 social sciences ,Perspective (graphical) ,Convergence (economics) ,GridFTP ,010501 environmental sciences ,01 natural sciences ,0502 economics and business ,business ,Data Linkage ,050203 business & management ,0105 earth and related environmental sciences ,Data transmission ,Research data - Abstract
Recent explosive increase in research data and increasing complexity of science and technology are continuing to expand convergence research and international collaboration among various fields. Developed countries are actively promoting open science policies to share research results and processes to create new knowledge and value through convergence researches. South Korea is also establishing and promoting similar strategies to encourage the sharing and utilization of national research data by the federal government. In 2018, a national research data platform was designed, and a prototype of the platform was constructed. It has been operated to verify the platform. It is important to link and collect more research data from various research institutions for the success of the national research data platform. We examine the communication and network technologies for the platform to link repositories outside and present some implications.
- Published
- 2019
37. Data Transfer between Scientific Facilities – Bottleneck Analysis, Insights and Optimizations
- Author
-
Yuanlai Liu, Nageswara S. V. Rao, Rajkumar Kettimuthu, Zizhong Chen, Zhengchun Liu, and Ian Foster
- Subjects
Computer science ,business.industry ,Distributed computing ,Concurrency ,020206 networking & telecommunications ,02 engineering and technology ,GridFTP ,Bottleneck ,Overhead (business) ,Computer data storage ,Data_FILES ,0202 electrical engineering, electronic engineering, information engineering ,Benchmark (computing) ,File transfer ,020201 artificial intelligence & image processing ,business ,Data transmission - Abstract
Wide area file transfers play an important role in many science applications. File transfer tools typically deliver the highest performance for datasets with a small number of large files, but many science datasets consist of many small files. Thus it is important to understand the factors that contribute to the decrease in wide area data transfer performance for datasets with many small files. To this end, we (i) benchmark the performance of subsystems involved in end-to-end file transfer between two HPC facilities for a many-file dataset that is representative of production science transfers; (ii) characterize the per-file overhead introduced by different subsystems; (iii) identify potential dependencies and bottlenecks; (iv) study the effectiveness of transferring many files concurrently as a means of reducing per-file overheads; and (v) prototype a prefetching mechanism as an alternative of concurrency to reduce the per-file overhead on source storage system. We show that both concurrency and prefetching can help reduce the per-file overhead significantly. A reasonable level of concurrency combined with prefetching can bring the per-file overhead down to a negligible level.
- Published
- 2019
38. Web UI System of TWAREN High-Speed Data Transfer Network
- Author
-
Che-Nan Yang, Li-Chi Ku, Ta-Yuan Chou, and Te-Lung Liu
- Subjects
Software ,Command-line interface ,business.industry ,Computer science ,Broadband ,Bandwidth (computing) ,ComputerApplications_COMPUTERSINOTHERSYSTEMS ,GridFTP ,business ,Computer network ,Data transmission - Abstract
This paper demonstrates the construction of a Web UI system of Data Transfer Network (DTN) on TWAREN. On the hardware aspect, via the TWAREN backbone, we connect 6 access nodes for high-speed data transfer through dedicated broadband lines. On the software aspect, we adopt widely-used tools, such as Fast Data Transfer (FDT) and GridFTP so that effective utilization of network bandwidth can be yielded. We also develop a Web UI system for users instead of Command Line Interface (CLI). Using the proposed system, which can be easily studied and operated, users can share research data more efficiently and effectively.
- Published
- 2019
39. Development of Hadoop Massive Data Migration System
- Author
-
Ming Zheng and Mugui Zhuo
- Subjects
Data processing ,Distributed computing ,Synchronization (computer science) ,Process (computing) ,Mode (statistics) ,Dynamic priority scheduling ,GridFTP ,Data migration ,Data transmission - Abstract
At present, the amount of data generated by high-energy physics experiments is increasing. When Hadoop, a large data processing platform, is used to process high-energy physics data, it is faced with the actual needs of data migration. However, the existing migration tools do not support data transmission between HDFS and other file systems, and their performance has obvious defects. Starting from the requirements of synchronization and archiving of high energy physical data, this paper designs and implements a universal mass data migration system. By extending the access mode of HDFS data, MapReduce is used to migrate data directly between HDFS data nodes and other storage systems/media. In addition, the system designs and implements the dynamic priority scheduling model, and evaluates and chooses the dynamic priority of multi-task. The system has been applied to data migration in large high altitude air shower Observatory (LHAASO) cosmic ray and other physical experiments. The actual operation results show that the system has good performance and can meet the data migration requirements of various experiments.
- Published
- 2019
40. APRIL: An Application-Aware, Predictive and Intelligent Load Balancing Solution for Data-Intensive Science
- Author
-
Byrav Ramamurthy, David Swanson, Brian Bockelman, and Deepak Nadig
- Subjects
021103 operations research ,Computer science ,business.industry ,Deep learning ,Big data ,0211 other engineering and technologies ,020206 networking & telecommunications ,02 engineering and technology ,GridFTP ,Predictive analytics ,Load balancing (computing) ,computer.software_genre ,Load management ,Recurrent neural network ,Workflow ,Server ,0202 electrical engineering, electronic engineering, information engineering ,Artificial intelligence ,Data mining ,business ,computer - Abstract
In this paper, we propose an application-aware intelligent load balancing system for high-throughput, distributed computing, and data-intensive science workflows. We leverage emerging deep learning techniques for time-series modeling to develop an application-aware predictive analytics system for accurately forecasting GridFTP connection loads. Our solution integrates with a major U.S. CMS Tier-2 site; we use a real dataset representing 670 million GridFTP transfer connections measured over 18 months to drive our predictive analytics solution. First, we perform extensive analysis on this dataset and use the connection loads as an example to study the temporal dependencies between various user-roles and workflow memberships. We use the analysis to motivate the design of a gated recurrent unit (GRU) based deep recurrent neural network (RNN) for modeling long-term temporal dependencies and predicting connection loads. We develop a novel application-aware, predictive and intelligent load balancer, APRIL, that effectively integrates application metadata and load forecast information to maximize server utilization. We conduct extensive experiments to evaluate the performance of our deep RNN predictive analytics system and compare it with other approaches such as ARIMA and multi-layer perceptron (MLP) predictors. The results show that our forecasting model, depending on the user-role, performs between 5.88%–92.6% better than the alternatives. We also demonstrate the effectiveness of APRIL by comparing it with the load balancing capabilities of an existing production Linux Virtual Server (LVS) cluster. Our approach improves server utilization, on an average, between 0.5 to 11 times, when compared with its LVS counterpart.
- Published
- 2019
41. Measurements and analytics of wide-area file transfers over dedicated connections
- Author
-
Raj Kettimuthu, Qiang Liu, Satyabrata Sen, Ian Foster, Zhengchun Liu, and Nageswara S. V. Rao
- Subjects
Ethernet ,Wide area ,Computer science ,Analytics ,business.industry ,Synchronous optical networking ,Computation ,File transfer ,Lustre (file system) ,GridFTP ,business ,Computer network - Abstract
Distributed scientific and big-data computations are becoming increasingly dependent on access to remote files. Wide-area file transfers are supported by two basic schemes: (i) application-level tools, such as GridFTP, that provide transport services between file systems housed at geographically separated sites, and (ii) file systems mounted over wide-area networks, using mechanisms such as LNet routers that make them transparently available. In both cases, the file transfer performance critically depends on the configuration consisting of host, file, IO, and disk subsystems, which are complex by themselves, as well as on their complex compositions implemented using buffers and IO-network data transitions. We present extensive file transfer rate measurements collected over dedicated 10 Gbps connections with 0-366 ms round-trip times, using GridFTP and XDD file transfer tools, and Lustre file system extended over wide-area networks using LNet routers. Our test configurations are composed of: three types of host systems; XFS, Lustre, and ext3 file systems; and Ethernet and SONET wide-area connections. We present analytics based on the convexity-concavity of throughput profiles which provide insights into throughput and its superior or inferior trend compared to linear interpolations. We propose the utilization-concavity coefficient, a scalar metric that characterizes the overall performance of any file transfer method consisting of specific configuration and scheme. Our results enable performance optimizations by highlighting the significant roles of (i) buffer sizes and parallelism in GridFTP and XDD, and (ii) buffer utilization and credit mechanism in LNet routers.
- Published
- 2019
42. Bootstrapping a new LHC data transfer ecosystem
- Author
-
Derek Weitzel, Brian Bockelman, Wei Yang, Andrew Hanushevsky, Mario Lassnig, Oliver Keeble, and Paul Millar
- Subjects
Delegate ,Data management ,computer: communications ,QC1-999 ,02 engineering and technology ,GridFTP ,Security token ,01 natural sciences ,010305 fluids & plasmas ,0103 physical sciences ,0202 electrical engineering, electronic engineering, information engineering ,ddc:530 ,communications [computer] ,Grid Security Infrastructure ,Worldwide LHC Computing Grid ,activity report ,Authentication ,business.industry ,Physics ,Computing and Computers ,CERN LHC Coll ,WebDAV ,Grid computing ,020201 artificial intelligence & image processing ,data management ,business ,Computer network - Abstract
23rd International Conference on Computing in High Energy and Nuclear Physics, CHEP 2018, Sofia, Bulgaria, 9 Jul 2018 - 13 Jul 2018; The European physical journal / Web of Conferences Web of Conferences : proceedings proceedings 214, 04045 (2019). doi:10.1051/epjconf/201921404045, GridFTP transfers and the corresponding Grid Security Infrastructure (GSI)-based authentication and authorization system have been data transfer pillars of the Worldwide LHC Computing Grid (WLCG) for more than a decade. However, in 2017, the end of support for the Globus Toolkit - the reference platform for these technologies - was announced. This has reinvigorated and expanded efforts to replace these pillars. We present an end-to-end alternate utilizing HTTP-based WebDAV as the transfer protocol, and bearer tokens for distributed authorization.This alternate ecosystem, integrating significant pre-existing work and ideas in the area, adheres to common industry standards to the fullest extent possible, with minimal agreed-upon extensions or common interpretations of the core protocols. The bearer token approach allows resource providers to delegate authorization decisions to the LHC experiments for experiment-dedicated storage areas.This demonstration touches the entirety of the stack - from multiple storage element implementations to FTS3 to the Rucio data management system. We show how the traditional production and user workflows can be reworked utilizing bearer tokens, eliminating the need for GSI proxy certificates for storage interactions., Published by EDP Sciences, Les Ulis
- Published
- 2019
43. Throughput Analytics of Data Transfer Infrastructures
- Author
-
Zhengchun Liu, Nageswara S. V. Rao, Ian Foster, Rajkumar Kettimuthu, and Qiang Liu
- Subjects
020203 distributed computing ,Computer science ,business.industry ,Distributed computing ,Testbed ,02 engineering and technology ,GridFTP ,computer.software_genre ,Software framework ,Software ,020204 information systems ,0202 electrical engineering, electronic engineering, information engineering ,File transfer ,business ,Communications protocol ,Host (network) ,Throughput (business) ,computer - Abstract
To support increasingly distributed scientific and big-data applications, powerful data transfer infrastructures are being built with dedicated networks and software frameworks customized to distributed file systems and data transfer nodes. The data transfer performance of such infrastructures critically depends on the combined choices of file, disk, and host systems as well as network protocols and file transfer software, all of which may vary across sites. The randomness of throughput measurements makes it challenging to assess the impact of these choices on the performance of infrastructure or its parts. We propose regression-based throughput profiles by aggregating measurements from sites of the infrastructure, with RTT as the independent variable. The peak values and convex-concave shape of a profile together determine the overall throughput performance of memory and file transfers, and its variations show the performance differences among the sites. We then present projection and difference operators, and coefficients of throughput profiles to characterize the performance of infrastructure and its parts, including sites and file transfer tools. In particular, the utilization-concavity coefficient provides a value in the range [0, 1] that reflects overall transfer effectiveness. We present results of measurements collected using (i) testbed experiments over dedicated 0–366 ms 10 Gbps connections with combinations of TCP versions, file systems, host systems and transfer tools, and (ii) Globus GridFTP transfers over production infrastructure with varying site configurations.
- Published
- 2019
44. Comparative Performance Evaluation of High-performance Data Transfer Tools
- Author
-
Eun-Sung Jung, Ian Fosterz, Deepak Nadig, Rajkumar Kettimuthu, Byrav Ramamurthy, and S.V. Nageswara Rao
- Subjects
Polynomial regression ,020203 distributed computing ,Computer science ,business.industry ,02 engineering and technology ,GridFTP ,Machine learning ,computer.software_genre ,Scheduling (computing) ,Data sharing ,Workflow ,Software ,020204 information systems ,0202 electrical engineering, electronic engineering, information engineering ,Artificial intelligence ,Predictability ,business ,computer ,Data transmission - Abstract
Data transfer in wide-area networks has been long studied in different contexts, from data sharing among data centers to online access to scientific data. Many software tools and platforms have been developed to facilitate easy, reliable, fast, and secure data transfer over wide area networks, such as GridFTP, FDT, bbcp, mdtmFTP, and XDD. However, few studies have shown the full capabilities of existing data transfer tools from the perspective of whether such tools have fully adopted state-of-the-art techniques through meticulous comparative evaluations. In this paper, we evaluate the performance of the four highperformance data transfer tools (GridFTP, FDT, mdtmFTP, and XDD) in various environments. Our evaluation suggests that each tool has strengths and weaknesses. FDT and GridFTP perform consistently in diverse environments. XDD and mdtmFTP show improved performance in limited environments and datasets during our evaluation. Unlike other studies on data transfer tools, we also evaluate the predictability of the tools’ performance, an important factor for scheduling different stages of science workflows. Performance predictability also helps in (auto)tuning the configurable parameters of the data transfer tool. We apply statistical learning techniques such as linear/polynomial regression, and k-nearest neighbors (kNN), to assess the performance predictability of each tool using its control parameters. Our results show that we can achieve good prediction performance for GridFTP and mdtmFTP using linear regression and kNN, respectively.
- Published
- 2018
45. Large Data Transfer Predictability and Forecasting using Application-Aware SDN
- Author
-
David Swanson, Deepak Nadig, Brian Bockelman, and Byrav Ramamurthy
- Subjects
business.industry ,Computer science ,020206 networking & telecommunications ,02 engineering and technology ,GridFTP ,Predictive analytics ,computer.software_genre ,Network management ,Exploratory data analysis ,Mean absolute percentage error ,0202 electrical engineering, electronic engineering, information engineering ,Autoregressive integrated moving average ,Data mining ,Predictability ,business ,Software-defined networking ,computer - Abstract
Network management for applications that rely on large-scale data transfers is challenging due to the volatility and the dynamic nature of the access traffic patterns. Predictive analytics and forecasting play an important role in providing effective resource allocation strategies for large data transfers. We propose a predictive analytics solution for large data transfers using an application-aware software defined networking (SDN) approach. We perform extensive exploratory data analysis to characterize the GridFTP connection transfers dataset and present various strategies for its use with statistical forecasting models. We develop a univariate autoregressive integrated moving average (ARIMA) based prediction framework for forecasting GridFTP connection transfers. Our prediction model tightly integrates with an application-aware SDN solution to preemptively drive network management decisions for GridFTP resource allocation at a U.S. CMS Tier-2 site. Further, our framework has a mean absolute percentage error (MAPE) ranging from 6% to 10% when applied to make rolling forecasts.
- Published
- 2018
46. Long-haul secure data transfer using hardware-assisted GridFTP
- Author
-
Mohammad Javad Rashti, Rajkumar Kettimuthu, and Gerald Sabin
- Subjects
Remote direct memory access ,Computer Networks and Communications ,business.industry ,Computer science ,ComputerSystemsOrganization_COMPUTER-COMMUNICATIONNETWORKS ,CPU time ,020206 networking & telecommunications ,02 engineering and technology ,GridFTP ,computer.software_genre ,TCP offload engine ,Network interface controller ,Large segment offload ,Hardware and Architecture ,Packet loss ,0202 electrical engineering, electronic engineering, information engineering ,Operating system ,File transfer ,020201 artificial intelligence & image processing ,business ,computer ,Software ,Computer hardware ,Computer network - Abstract
Extreme-scale scientific collaborations require high-performance wide-area end-to-end data transports to enable fast and secure transfer of high data volumes among collaborating institutions. GridFTP is the de facto protocol for large-scale data transfer in science environments. Existing predominant network transport protocols such as TCP have serious limitations that consume significant CPU power and prevent GridFTP from achieving high throughput on long-haul networks with high latency and potential packet loss, reordering and jitter. On the other hand, protocols such as UDT that address some of the TCP shortcomings demand high computing resources on data transfer nodes. These limitations have caused underutilization of existing high-bandwidth links in scientific and collaborative grids. To address this situation, we have enhanced Globus GridFTP, the most widely used GridFTP implementation, by developing transport offload engines such as UDT and iWARP on SmartNIC, a programmable 10GbE network interface card (NIC). Our results show significant reduction in server utilization and full line-rate sustained bandwidth in high-latency networks, as measured for up to 100 ms of network latency. In our work, we also offload OpenSSL on SmartNIC to reduce host utilization for secure file transfers. The offload engine can provide line-rate data channel encryption/decryption on top of UDT offload without consuming additional host CPU resources. Lower CPU utilization leads to increased server capacity, which allows data transfer nodes to support higher network and data-processing rates. Alternatively, smaller or fewer DTNs can be used for a particular data rate requirement. We enhance GridFTP long-haul file transfers using hardware offloading.UDT offload improves long-haul file transfer throughput and reduces CPU utilization.UDT offload has a faster convergence to peak BW compared to host-based UDT and TCP.UDT shows lower sensitiveness to packet loss and burst compared to TCP.SSL-offload over UDT provides the same BW and host utilization as regular UDT.
- Published
- 2016
47. Application-Level Optimization of Big Data Transfers through Pipelining, Parallelism and Concurrency
- Author
-
Tevfik Kosar, Engin Arslan, Jangyoung Kim, and Esma Yildirim
- Subjects
020203 distributed computing ,Multi-core processor ,Computer Networks and Communications ,business.industry ,Computer science ,Distributed computing ,Pipeline (computing) ,Big data ,Clock rate ,02 engineering and technology ,Parallel computing ,GridFTP ,Computer Science Applications ,File size ,Hardware and Architecture ,Transfer (computing) ,0202 electrical engineering, electronic engineering, information engineering ,Concurrent computing ,020201 artificial intelligence & image processing ,business ,Throughput (business) ,Software ,Information Systems - Abstract
In end-to-end data transfers, there are several factors affecting the data transfer throughput, such as the network characteristics (e.g., network bandwidth, round-trip-time, background traffic); end-system characteristics (e.g., NIC capacity, number of CPU cores and their clock rate, number of disk drives and their I/O rate); and the dataset characteristics (e.g., average file size, dataset size, file size distribution). Optimization of big data transfers over inter-cloud and intra-cloud networks is a challenging task that requires joint-consideration of all of these parameters. This optimization task becomes even more challenging when transferring datasets comprised of heterogeneous file sizes (i.e., large files and small files mixed). Previous work in this area only focuses on the end-system and network characteristics however does not provide models regarding the dataset characteristics. In this study, we analyze the effects of the three most important transfer parameters that are used to enhance data transfer throughput: pipelining, parallelism and concurrency . We provide models and guidelines to set the best values for these parameters and present two different transfer optimization algorithms that use the models developed. The tests conducted over high-speed networking and cloud testbeds show that our algorithms outperform the most popular data transfer tools like Globus Online and UDT in majority of the cases.
- Published
- 2016
48. Token-based authorization in StoRM WebDAV
- Author
-
Diego Michelotto, Andrea Ceccanti, and Enrico Vianello
- Subjects
Authentication ,010308 nuclear & particles physics ,business.industry ,Physics ,QC1-999 ,Data management ,GridFTP ,Security token ,Computer security ,computer.software_genre ,01 natural sciences ,OpenID Connect ,0103 physical sciences ,WebDAV ,Key (cryptography) ,Data center ,010306 general physics ,business ,computer - Abstract
At the end of May 2017 the Globus Alliance announced that the open-source Globus Toolkit (GT) would be no longer supported by the Globus team at the University of Chicago. This announcement had an obvious impact on WLCG, given the central role of the Globus Security Infrastructure (GSI) and GridFTP in the WLCG data management framework, so discussions started in the appropriate forums on the search for alternatives. At the same time, support for token-based authentication and authorization has emerged as a key requirement for storage elements powering WLCG data centers. In this contribution, we describe the work done to enable token-based authentication and authorization in the StoRM WebDAV service, describing and highlighting the differences between support for external OpenID connect providers, groupbased and capability-based authorization schemes, and locally-issued authorization tokens. We discuss how StoRM WebDAV token-based authorization is being exploited in several contexts, from WLCG DOMA activities to other scientific experiments hosted at the INFN Tier-1 data center. In this contribution, we also describe the methodology used to compare Globus GridFTP and StoRM WebDAV and we present initial results confirming how HTTP represent a viable alternative to GridFTP for data transfers also performance-wise.
- Published
- 2020
49. Testing the limits of HTTPS single point third party copy transfer over the WAN
- Author
-
Frank Wuerthwein, Brian Bockelman, and Edgar Fajardo
- Subjects
Large Hadron Collider ,Third party ,010308 nuclear & particles physics ,business.industry ,Physics ,QC1-999 ,Big data ,GridFTP ,computer.software_genre ,01 natural sciences ,Server ,Transfer (computing) ,0103 physical sciences ,Operating system ,Single point ,010306 general physics ,business ,Protocol (object-oriented programming) ,computer - Abstract
LHC data is constantly being moved between computing and storage sites to support analysis, processing, and simulation; this is done at a scale that is currently unique within the science community. For example, the CMS experiment on the LHC manages approximately 200PB of storage across 100 sites and, on a daily basis, moves 1PB between sites via GridFTP as primary protocol. This paper describes the performance results we have achieved by exploring alternatives to the GridFTP protocol for these data movements. In particular the HTTPS third party copy over Xrootd data servers as a possible replacement of GridFTP for LHC big data movements.
- Published
- 2020
50. Cross-geography scientific data transferring trends and behavior
- Author
-
Zhengchun Liu, Rajkumar Kettimuthu, Nageswara S. V. Rao, and Ian Foster
- Subjects
End user ,Computer science ,020206 networking & telecommunications ,Throughput ,02 engineering and technology ,GridFTP ,Data science ,Resource (project management) ,Wide area network ,020204 information systems ,Transfer (computing) ,0202 electrical engineering, electronic engineering, information engineering ,Data Corruption ,File transfer - Abstract
Wide area data transfers play an important role in many science applications but rely on expensive infrastructure that often delivers disappointing performance in practice. In response, we present a systematic examination of a large set of data transfer log data to characterize transfer characteristics, including the nature of the datasets transferred, achieved throughput, user behavior, and resource usage. This analysis yields new insights that can help design better data transfer tools, optimize networking and edge resources used for transfers, and improve the performance and experience for end users. Our analysis shows that (i) most of the datasets as well as individual files transferred are very small; (ii) data corruption is not negligible for large data transfers; and (iii) the data transfer nodes utilization is low. Insights gained from our analysis suggest directions for further analysis.
- Published
- 2018
Catalog
Discovery Service for Jio Institute Digital Library
For full access to our library's resources, please sign in.