Search

The paper introduces FlashPage, a high-speed SSD caching system designed for ultra-fast media, with the goal of enhancing web page delivery in proxy servers. Traditional SSD caching schemes, designed primarily for slow HDD-based primary storage, encounter difficulties when applied to capacity-class SSDs as primary storage. This limits the high-performance capabilities of caching media. To address this issue, FlashPage operates within the Linux virtual filesystem layer, shortening the hit-handling path and minimizing lookup overhead. It incorporates a compact radix tree to fast locate cached data. These approaches reduce the software overhead for a 4kB read hit by over 5 times. FlashPage also employs novel admission and eviction policies to minimize flash wear while maintaining a high hit rate. As a second-level storage cache, FlashPage predicts the hotness of potential demotion candidates in the first-level storage cache (i.e., page cache), achieving a 10.1% higher hit rate and reducing write traffic by 10.4% compared to LRU. Evaluations using Varnish and Squid HTTP caches show its effectiveness, with up to 29.6% and 38.2% faster web request processing compared to Bcache and DM-Cache, state-of-the-art caching schemes in mainline Linux kernels.

The article introduces a cost-based data prefetching and task scheduling framework in Big Data platforms over underlying tiered storage system. Topics discussed include solutions to task scheduling, resource scheduling and data prefetching problems over tiered storage, pruning algorithms for reducing scheduling time, YARN cluster resource request model, storage-tier-aware task scheduling approach, and implementation of Trident scheduling and prefetching components in Apache Hadoop and Spark.

This paper presents DITIS, a simulator for distributed and tiered file-based storage systems. In particular, DITIS can model a distributed storage system with up to three levels of storage tiers and up to three additional levels of caches. Each tier and cache can be configured with different number and type of storage media devices (e.g., HDD, SSD, NVRAM, DRAM), each with their own performance characteristics. The simulator utilizes the provided characteristics in fine-grained performance cost models (which are distinct for each device type) in order to compute the duration time of each I/O request processed on each tier. At the same time, DITIS simulates the overall flow of requests through the different layers and storage nodes of the system using numerous pluggable policies that control every aspect of execution, ranging from request routing and data redundancy to cache and tiering strategies. For performing the simulation, DITIS adapts an extended version of the Actor Model, during which key components of the system exchange asynchronous messages with each other, much like a real distributed multi-threaded system. The ability to simulate the execution of a workload in such an accurate and realistic way brings multiple benefits for its users, since DITIS can be used to better understand the behavior of the underlying file system as well as evaluate different storage setups and policies. [ABSTRACT FROM AUTHOR]

Storing large amounts of data for different users has become the new normal in a modern distributed cloud storage environment. Storing data successfully requires a balance of availability, reliability, cost, and performance. Typically, systems design for this balance with minimal information about the data that will pass through them. We propose a series of methods to derive groupings from data that have predictive value, informing layout decisions for data on disk. Unlike previous grouping work, we focus on dynamically identifying groupings in data that can be gathered from active systems in real time with minimal impact using spatiotemporal locality. We outline several techniques we have developed and discuss how we select particular techniques for particular workloads and application domains. Our statistical and machine-learning-based grouping algorithms answer questions such as “What can a grouping be based on?” and “Is a given grouping meaningful for a given application?” We design our models to be flexible and require minimal domain information so that our results are as broadly applicable as possible. We intend for this work to provide a launchpad for future specialized system design using groupings in combination with caching policies and architectural distinctions such as tiered storage to create the next generation of scalable storage systems.

Multi-tiered storage, where each tier consists of one type of storage device (e.g., SSD, HDD, or disk arrays), is a commonly used approach to achieve both high performance and cost efficiency in large-scale systems that need to store data with vastly different access characteristics. By aligning the access characteristics of the data, either fixed-sized extents or variable-sized files, to the characteristics of the storage devices, a higher performance can be achieved for any given cost. This article presents ExaPlan, a method to determine both the data-to-tier assignment and the number of devices in each tier that minimize the system's mean response time for a given budget and workload. In contrast to other methods that constrain or minimize the system load, ExaPlan directly minimizes the system's mean response time estimated by a queueing model. Minimizing the mean response time is typically intractable as the resulting optimization problem is both nonconvex and combinatorial in nature. ExaPlan circumvents this intractability by introducing a parameterized data placement approach that makes it a highly scalable method that can be easily applied to exascale systems. Through experiments that use parameters from real-world storage systems, such as CERN and LOFAR, it is demonstrated that ExaPlan provides solutions that yield lower mean response times than previous works. It supports standalone SSDs and HDDs as well as disk arrays as storage tiers, and although it uses a static workload representation, we provide empirical evidence that underlying dynamic workloads have invariant properties that can be deemed static for the purpose of provisioning a storage system. ExaPlan is also effective as a load-balancing tool used for placing data across devices within a tier, resulting in an up to 3.6-fold reduction of response time compared with a traditional load-balancing algorithm, such as the Longest Processing Time heuristic. [ABSTRACT FROM AUTHOR]

In the current situation of the economic crisis, when companies target budgetcuttings in a context of an explosive data growth, the IT community must evaluate potentialtechnology developments not only on their technical advantages, but on their economiceffects as well. More then ever, the old cliché “doing more things with fewer resources” istrue today. Many IT companies started building very large facilities, called data centers(DCs) or Internet DC (IDCs), which provide businesses a wide range of solutions forsystems deployment and operation. In recent years, the IT departments around the worldhave moved from data center and infrastructure consolidation to virtualization.Data center virtualization is the process of aligning available resources with the actualneeds of the offered services, moving from physical servers to virtual servers, sharing andprovisioning servers, networks, storage, and applications. By taking advantage of threebasic innovations — virtualization, tiered storage architectures and dynamic provisioningsoftware — an organization can achieve greater efficiencies in their current computingenvironment.Such a unified computing architecture offers end-to-end virtualization; all structures areoptimized for virtualized environments, from the CPU to the aggregation layer. Incombination with embedded management, this new approach increases responsiveness andreduces the opportunities for human error, improving consistency and reducing server andnetwork deployment times.

SAN (Storage Area Network, a high-speed, block level storage device) based centralized data storage system of SST-1 (Steady State superconducting Tokamak) has envisaged to address the need of availability of SST-1 operation & experimental data centrally for archival as well as retrieval [2] . Considering the initial data volume requirement, ∼10 TB (Terabytes) capacity of SAN based data storage system has configured/installed with optical fiber backbone with compatibility considerations of existing Ethernet network of SST-1. The SAN based data storage system has been designed/configured with 3-tiered architecture and GFS (Global File System) cluster file system with multipath support. Tier-1 is of ∼3 TB (frequent access and low data storage capacity) comprises of Fiber channel (FC) based hard disks for optimum throughput. Tier-2 is of ∼6 TB (less frequent access and high data storage capacity) comprises of SATA based hard disks. Tier-3 will be planned later to store offline historical data. In the SAN configuration two tightly coupled storage servers (with cluster configuration) are working together to achieve increase performance, reliability, distribute workload and provides access to the files from any server regardless of the physical location of the file. Different RAID (Redundant Array of Independent Disks) groups are created with both, FC and SATA hard disks to increase reliability, security and performance of the storage system. The adopted SAN based data storage for SST-1 is a modular, robust, and allows future expandability. The storage modules can be added as and when required without changing the existing storage architecture. The data read/write time of the configured system is adequate enough to cater the present throughput requirements of individual subsystems. Sufficient redundancy in terms of hardware has been incorporated to assure the uninterrupted experimental data availability 24 × 7 within intranet. The installed storage system can be expanded up to ∼100 TB of capacity with the existing controller pair presents in the SAN configuration. [ABSTRACT FROM AUTHOR]

SUMMARY We propose a new method for tiered storage and evaluate its characteristics. In this method, the fast tier is divided into two areas and the data in each area are managed on the basis of two I/O (input/output) measurement periods. When I/Os to these areas increase, the proposed method allocates the frequently accessed areas to a fast tier in advance, thereby improving storage I/O performance. We evaluate the proposed method by simulating storage I/Os and data migration between tiers. When I/O locality is high or frequently accessed areas rarely move, the proposed method is more effective and improves storage I/O performance by up to 44%. [ABSTRACT FROM AUTHOR]

SUMMARY We evaluated the storage I/O response time with the volume tiering method and the SSD cache method on tiered storage having SSD and HDD. The simulation results show that the volume tiering method improves the storage performance with low I/O locality workload and small SSD capacity. The SSD cache method is effective with read-intensive and high I/O locality workloads. Utilizing both methods simultaneously reduces the storage I/O response time by up to 20% with write-intensive workloads and large SSD capacity compared to using the volume tiering method or the SSD cache method separately. [ABSTRACT FROM AUTHOR]

Background: Large streamed datasets, characteristic of life science applications, are often resource-intensive to process, transport and store. We propose a pipeline model, a design pattern for scientific pipelines, where an incoming stream of scientific data is organized into a tiered or ordered "data hierarchy". We introduce the HASTE Toolkit, a proof-of-concept cloud-native software toolkit based on this pipeline model, to partition and prioritize data streams to optimize use of limited computing resources., Findings: In our pipeline model, an "interestingness function" assigns an interestingness score to data objects in the stream, inducing a data hierarchy. From this score, a "policy" guides decisions on how to prioritize computational resource use for a given object. The HASTE Toolkit is a collection of tools to adopt this approach. We evaluate with 2 microscopy imaging case studies. The first is a high content screening experiment, where images are analyzed in an on-premise container cloud to prioritize storage and subsequent computation. The second considers edge processing of images for upload into the public cloud for real-time control of a transmission electron microscope., Conclusions: Through our evaluation, we created smart data pipelines capable of effective use of storage, compute, and network resources, enabling more efficient data-intensive experiments. We note a beneficial separation between scientific concerns of data priority, and the implementation of this behaviour for different resources in different deployment contexts. The toolkit allows intelligent prioritization to be `bolted on' to new and existing systems - and is intended for use with a range of technologies in different deployment scenarios., (© The Author(s) 2021. Published by Oxford University Press GigaScience.)

Summary: Multitiered storage systems, which are made up of heterogeneous devices, are widely used in distributed environments to accelerate the I/O performance of upper big data applications. It raises new challenges in efficient data migration through smart caching mechanisms among heterogeneous storage levels, such as MEM‐SSD‐HDD. To optimize the cache policy scheduling mechanism on the distributed tiered storage architecture, we proposed a general framework with five layers, including a tiered storage system layer, a cache migration policy layer, a cache policy adaptive scheduling layer, a data access pattern layer, and a big data application layer. The framework prototype has been designed and implemented on the widely used distributed hybrid storage system named Alluxio. To meet the demands of the big data application layer, on the one hand, we designed a couple of cache eviction policies and promotion policies covering various access patterns on the cache migration policy layer (several proposed eviction policies have been adopted by the Alluxio open‐source community). On the other hand, two adaptive cache policy scheduling algorithms for selecting appropriate policies in various scenarios are designed and implemented on the cache policy adaptive scheduling layer. The scheduling algorithms are designed based on the hit ratio statistics and data access pattern model prediction, respectively. Experimental results show that the proposed cache policies are very effective for various big data applications, such as Spark SQL. The proposed cache policy scheduling algorithms with various eviction policies can improve around 20% hit ratio than that with a single eviction policy. [ABSTRACT FROM AUTHOR]