Performance-Aware Big Data Management for Remote Sensing Systems.

Remote sensing data, whose dimensions increase exponentially and turn into big data with the new technologies, cause significant difficulties in transferring, storing, and processing because of consisting of gigantic coarse-grained files. This article proposes a novel two-phase big data management system on the geo-distributed private cloud that takes advantage of network topology and resource utilization in a distributed manner. The system optimizes resource allocation to facilitate efficient and extensive data analysis for remote sensing applications by minimizing file fragmentation, resulting in faster analysis. In order to simulate the proposed system, different network topologies are created using virtual machines. Moreover, the proposed method named performance-aware assignment is compared with well-known methods such as random assignment, Hungarian algorithm, and Hadoop Distributed File System, also famous in the big data era. The experimental results indicate that performance-aware assignment outperforms random assignment, Hungarian algorithm, and Hadoop Distributed File System, achieving 36%, 26%, and 71% more stored data, respectively, within the same time while also exhibiting lower IOPS values. In addition, it optimizes resource usage in data centers, which is particularly important for preventing resource exhaustion. [ABSTRACT FROM AUTHOR]