Probabilistic scheduling of dynamic I/O requests via application clustering for burst‐buffers equipped high‐performance computing.

Summary: Burst‐buffering is a promising storage solution that introduces an intermediate high‐throughput storage buffer layer to mitigate the I/O bottleneck problem that the current high‐performance computing (HPC) platforms suffer. The existing Markov‐Chain based probabilistic I/O scheduling utilizes the load state of burst‐buffers and the periodic characteristics of applications to reduce I/O congestion due to the limited capacity of burst‐buffers. However, this probabilistic approach requires consistent I/O characteristics of applications, including similar I/O duration and long application length, in order to obtain an accurate I/O load estimation. These consistency conditions do not often hold in realistic situations. In this paper, we propose a generic framework of dynamic probabilistic I/O scheduling based on application clustering (DPSAC) to make applications meet the consistency requirements. According to the I/O phase length of each application, our scheme first deploys a one‐dimensional K‐means clustering algorithm to cluster the applications into clusters. Next, it calculates the expected workload of each cluster through the probabilistic model of applications and then partitions the burst‐buffers proportionally. Then, to handle dynamic changes (join and exit) of applications, it updates the clusters based on a heuristic strategy. Finally, it applies the probabilistic I/O scheduling, which is based on the distribution of application workload and the state of burst‐buffers, to schedule I/O for all the concurrent applications to mitigate I/O congestion. The simulation results on synthetic data show that our DPSAC is effective and efficient. [ABSTRACT FROM AUTHOR]