Search

Large-scale high performance computing (HPC) systems typically consist of many thousands of CPUs and storage units used by hundreds to thousands of users simultaneously. Applications from large numbers of users have diverse characteristics, such as varying computation, communication, memory, and I/O intensity. A good understanding of the performance characteristics of each user application is important for job scheduling and resource provisioning. Among these performance characteristics, I/O performance is becoming increasingly important as data sizes rapidly increase and large-scale applications, such as simulation and model training, are widely adopted. However, predicting I/O performance is difficult because I/O systems are shared among all users and involve many layers of software and hardware stack, including the application, network interconnect, operating system, file system, and storage devices. Furthermore, updates to these layers and changes in system management policy can significantly alter the I/O behavior of applications and the entire system. To improve the prediction of the I/O performance on HPC systems, we propose integrating information from several different system logs and developing a regression-based approach to predict the I/O performance. Our proposed scheme can dynamically select the most relevant features from the log entries using various feature selection algorithms and scoring functions, and can automatically select the regression algorithm with the best accuracy for the prediction task. The evaluation results show that our proposed scheme can predict the write performance with up to 90% prediction accuracy and the read performance with up to 99% prediction accuracy using the real logs from the Cori supercomputer system at NERSC.

High-performance storage devices, such as Non-Volatile Memory express Solid-State Drives (NVMe SSDs), have been widely adopted in data centers. Especially, multiple storage devices provide higher I/O performance compared with a single device. However, the performance can be reduced in the case of workloads with mixed read and write requests (e.g., key-value stores) even though multiple storage devices are adopted. This is because read requests can be blocked until the processing for write requests is finished. In this article, we propose an I/O isolation scheme to improve the performance of the key-value store for multiple SSDs. In our scheme, we classify files of the key-value store and deploy files to the separated storage devices according to the characteristics of each file. Thus, read/write operations are performed in different storage devices. In addition, we propose two different device mapping methods, namely fixed and adaptive device mapping to deploy files to the proper device. We implement our scheme in RocksDB with multiple storage devices (six NVMe SSDs) and extend our scheme on an open-channel SSD, which reveals internal hardware architecture to verify the effectiveness of read/write isolation within a single storage device. The experimental results demonstrate that our scheme improves performance by up to 29% and 26% in the open-channel SSD and multiple storage devices, respectively, compared with the existing scheme.

General-purpose graphics processing units (GPGPUs) have been widely adapted to the industry due to the high parallelism of graphics processing units (GPUs) compared with central processing units (CPUs). Especially, a GPGPU device has been adopted for various scientific workloads which have high parallelism. To handle the ever increasing demand, multiple applications are often run simultaneously in multiple GPGPU devices. However, when multiple applications are running concurrently, the overall performance of GPGPU devices varies significantly due to the different characteristics of GPGPU applications. To improve the efficiency, it is critical to anticipate the performance of applications and find optimal scheduling policy. In this paper, we analyze various types of scientific applications and identify factors that impact the performance during the concurrent execution of the applications in GPGPU devices. Our analysis results show that each application has distinct characteristic. By considering distinct characteristics of applications, a certain combination of applications has better performance compared with the others when executed concurrently in multiple GPGPU devices. Based on the finding of our analysis, we propose a simulator which predicts the performance of GPGPU devices when multiple applications are running concurrently. Our simulator collects performance metrics during the execution of applications and predicts the performance of certain combinations using the performance metrics. The experimental result shows that the best combination of applications can increase the performance by 39.44% and 65.98% compared with the average of combinations and the worst case, respectively when using a single GPGPU device. When utilizing multiple GPGPU devices, our result shows that the performance improve can be 24.78% and 39.32% compared with the average and the worst combinations, respectively.

Cloud computing is being widely adopted in the industry due to providing more computation power and improve resource utilization. In cloud computing systems, many users execute various types of applications that produce a large amount of data. To handle a large amount of data, cloud computing systems provide various, high-performance, and large-scale clustered storage devices. With the large capacity, improving the performance of storage maintenance is an important issue since the large capacity can increase the suspend time during the maintenance significantly. As a storage maintenance technique, checking a bad block in which the data cannot be accessed anymore prevents I/O failure of the application. However, an existing bad block checker (e.g., badblocks in Linux) takes a long time, even when storage devices provide parallelism (e.g., multiple disks, multi-channel SSD, etc). It is because the existing bad block checker performs I/O and check operations in a serialized manner. To reduce the checking time, we propose an efficient and parallel bad block checker for exploiting the parallelism of storage devices. In our scheme, we enable parallel I/O and check operations for the bad block instead of the serialized operations. To do this, we first divide a series of check operations into parallel tasks (i.e., independent tasks). Second, we create a thread pool in which multiple workers fetch their tasks concurrently. Finally, we enable each checker to perform its own check and I/O operations in parallel. We implement and evaluate our checker on a 32-core machine with a disk array and an NVMe SSD. The experimental results show that our proposed bad block checker improves the performance by up to 3.7 $$\times$$ and 7.8 $$\times$$ in the disk array and NVMe SSD, respectively, compared with the existing bad block checker.

Non-volatile random access memory (NVRAM) is a promising approach to persistent data storage with outstanding advantages over traditional storage devices, such as hard disk drives (HDDs) and solid state drives (SSDs). Some of its biggest advantages are its DRAM-like read latency and microsecond-level write latency, which are several hundred times faster than those in the original block device. However, one of the issues with using NVRAM as a storage device is designing an indexing system for its data stores to fully utilize NVRAM characteristics. The state-of-the-art indexing systems of non-volatile key-value stores are usually based on B+-trees or their variants, which were originally designed for block-based storage devices with better sequential performance than random performance. The semantics of B+-tree require data being sorted into leaf nodes and inner nodes and frequent splitting and merging to keep balanced. However, all the sorting, splitting, and merging operations cause extra write to NVRAM, which decreases its performance. In this article, we propose NV-Skiplist, a skiplist-based indexing system for key-value stores on NVRAM that fully uses the features of both NVRAM and DRAM. NV-Skiplist constructs its bottom layer in non-volatile memory to maintain data persistence and support range scans. It builds its upper layers in DRAM to retain rapid index searching and prevent consistently large overhead. We also propose a multiranged variant of NV-Skiplist to increase its search performance and scalability. We evaluate the performance of NV-Skiplist and wB+-tree which is a state-of-art scheme on an NVRAM emulator on a server with an Intel Xeon E5-2620 v2 processor. The results show that our design outperforms the original tree-based, non-volatile key-value stores up to 48%.

RDMA is increasingly becoming popular not only in HPC but also in data centers where high throughput and low latency are critical requirements. RDMA supports several types of transports, each of which has different characteristics, so that users can choose the right one to meet their requirements. Reliable connected (RC) transport has advantages on usability but disadvantages on scalability while unreliable datagram (UD) transport is scalable but hard to use. Dynamically connected (DC) transport has been newly introduced to address these limitations when using one of the existing transports while delivering both usability and scalability. However, despite all of these merits, DC transport is not yet generally adopted in the related fields due to hardware dependency. To eliminate the hardware dependency, in this paper, we design and implement SoftDC, a totally software-based DC transport. SoftDC uses the basic RDMA primitives to emulate the behavior of DC transport; UD transport for connection and RC transport for data transfer. We build and evaluate a SoftDC transport-based application to prove its effectiveness compared with RC transport-based one. Our experimental results show that our scheme has potential to provide the advantages of both RC and UD transports although our prototype has large connection establishment costs that can be amortized across a large data transfer.

A key-value store is an essential component that is increasingly demanded in many scale-out environments, including social networks, online retail environments, and cloud services. Modern key-value storage engines provide many features, including transaction, versioning, and replication. In storage engines, transaction processing provides atomicity and durability by using write-ahead logging (WAL), which flushes log data before the data page is written to persistent storage in synchronous commit. However, according to our observation, WAL is a performance bottleneck in key-value storage engines since the flushing of log data to persistent storage incurs a significant overhead of lock contention and fsync() calls, even with the various optimizations in the existing scheme. In this article, we propose an approach to improve the performance of key-value storage by optimizing the existing flushing scheme combined with group commit and consolidate array. Our scheme aggregates the multiple flushing of log data into a large request on the fly and completes the request early. This scheme is an efficient group commit that reduces the number of frequent lock acquisitions and fsync() calls in the synchronous commit while supporting the same transaction level that the existing scheme provides. Furthermore, we integrate our flushing scheme into the replication system and evaluate it by using multiple nodes. We implement our scheme on the WiredTiger storage engine. The experimental results show that our scheme improves the performance of the key-value workload compared to the existing scheme.

Modern storage systems are facing an important challenge of making the best use of fast storage devices. Even though the underlying storage devices are being enhanced, the traditional storage stack falls short of utilizing the enhanced characteristics, as it has been optimized specifically for hard disk drives. In this article, we optimize the storage stack to maximize the benefit of low latency that fast storage devices provide. Our approach is to simplify the I/O path from application to the fast storage device by removing inefficient layers and the conventional block I/O. The proposed stack consists of three layers: an optimized device driver, a low-latency file system called L2FS, and a simplified VFS. The device driver provides a simple file I/O API to the file system instead of the existing block I/O API. L2FS, a variant of EXT4, performs low-latency I/O operations by using the file I/O API that our optimized device driver provides. We implement our storage stack on Linux 3.14.3 and evaluate it with multiple benchmarks. The results show that our system improves the throughput by up to 6.6 times and reduces the latency by an average of 54% compared to the existing storage stack on fast storage.

Emerging non-volatile memory (NVM) technology with high throughput and scalability has considerable attraction in cloud and enterprise storage systems. The industry and academic communities made the NVMe specification to elicit the highest performance on NVM devices. While the technology is commercially viable, it is important to consider the performance of NVM devices with NVMe specification according to different I/O configurations and analyze workloads on the storage to exploit better performance. This paper provides the results of empirical evaluation and analysis of the performance on a recent NVM express solid state drive (NVMe SSD) developed by Samsung electronics, a flash-based PCIe-attached SSD built to follow NVMe specification. The maximum throughput is 2.5 GB/s and 800 MB/s for reading and writing 4 kb, respectively. We analyze the performance of NVMe SSD in terms of different performance metrics with microbenchmark and database workloads. We also perform comparison study of NVMe SSD with SATA SSD. We anticipate that the experimental results and performance analysis will provide the implications on various storage systems.

Lately, fast storage devices are rapidly increasing in social network services, cloud platforms, etc. Unfortunately, the traditional Linux I/O stack is designed to maximize performance on disk-based storage. Emerging byte-addressable and low-latency non-volatile memory technologies (e.g., phase-change memories, MRAMs, and the memristor) provide very different characteristics, so the disk-based I/O stack cannot lead to high performance. This paper presents a high performance I/O stack for the fast storage devices. Our scheme is to remove the concept of block and to simplify the whole I/O path and software stack, which results in only two layers that are the byte-capable interface and the byte-aware file system called BAFS. We aim to minimize I/O latency and maximize bandwidth by eliminating the unnecessary layers and supporting byte-addressable I/O without requiring changes to applications. We have implemented a prototype and evaluated its performance with multiple benchmarks. The experimental results show that our I/O stack achieves 6.2 times on average and up to 17.5 times performance gains compared to the existing Linux I/O stack.