Search

Recently, power-aware distributed file systems for efficient big data processing have increasingly moved toward power proportional designs. However, inefficient gear-shifting in such systems is an important issue that can seriously degrade their performance. To address this issue, we propose and evaluate an efficient gear-shifting power proportional distributed file system. The proposed system utilizes flexible data placement that reduces the amount of reflected data and has an architecture that improves the metadata management to achieve high-efficiency gear-shifting. Extensive empirical experiments using actual machines based on the HDFS demonstrated that the proposed system gains up to \(22\,\%\) better throughput-per-watt performance. Moreover, a suitable metadata management setting corresponding to the amount of data updated while in low gear is found from the experimental results.

In the big data era, the distributed file system is getting more and more significant due to the characteristics of\ud its scale-out capability, high availability, and high performance. Different distributed file systems may have different design goals. For example, some of them are designed to have good performance for small file operations, such as GlusterFS, while some of them are designed for large file operations, such as Hadoop distributed file system. With the divergence of big data applications, a distributed file system may provide good performance for some applications but fails for some other applications, that is, there has no universal distributed file system that can produce good performance for all applications. In this\ud paper, we propose a hybrid file system framework, HybridFS, which can deliver satisfactory performance for all applications. HybridFS is composed of multiple distributed file systems with the integration of advantages of these distributed file systems. In HybridFS, on top of multiple distributed file systems, we have designed a metadata management server to perform three functions: file placement, partial metadata store, and dynamic file migration. The file placement is performed based on a decision tree. The partial metadata store is performed for files whose size is less than a few hundred Bytes to increase throughput. The dynamic file migration is performed to balance the storage usage of distributed file systems without throttling performance. We have implemented HybridFS in java on eight nodes and choose Ceph, HDFS, and GlusterFS as designated distributed file systems. The experimental results show that, in the best case, HybridFS can have up to 30% performance improvement of read/write operations over a single distributed file system. In addition, if the difference of storage usage among multiple distributed file systems is less than 40%, the performance of HybridFS is guaranteed, that is, no performance degradation.

MemFS is a fully-symmetrical, in-memory distributed runtime file system. Its design is based on uniformly distributing file stripes across the storage nodes belonging to an application by means of a distributed hash function, purposefully sacrificing data locality for balancing both network traffic and memory consumption. This way, reading and writing files can benefit from full network bisection bandwidth, while data distribution is balanced across the storage servers.

Parallel disk I/O subsystems are becoming more important in today’s large-scale parallel machines. Parallel disk systems provide a significant boost in I/O performance reducing the gap between processor and disk speeds. We describe a Unix-like file I/O user interface, implemented in a parallel file I/O subsystem on an MIMD machine, the nCUBE 2. Based on message passing, we develop parallel disk read/write algorithms to achieve higher parallelism when more disk drives are used. We use the closed queuing network model to analyze the effect of some tunable system parameters for our parallel file system. We then performed simulation experiments in order to obtain more realistic performance data, by comparing the original vendor-supplied file system with ours. The results indicate that the speedup in I/O performance is almost equal to the number of disk drives we use. Thus, our user-level parallel file I/O approach will provide scalable I/O performance.

The Bullet server is a file server that outperforms traditional file servers by more than a factor of three. It achieves high throughput and low delay by a software design radically different from that of file servers currently in use. Whereas files are normally stored as a sequence of disk blocks, each Bullet server file is stored contiguously, both on disk and in the server's random access memory cache. Furthermore, it uses the concept of an immutable file to improve performance, to enable caching, and to provide a clean semantic model to the user. The authors describe the design and implementation of the Bullet server in detail, present measurements of its performance, and compare this performance to that of the SUN file server running on the same hardware. >