Structured Big Data Management System Supported Cross-Domain Query

We design a structured big data management system which can deal with large-scale structured datasets and supports the cross-domain collaborative query. The system employs the HDFS as the storage layer. And it realizes a scheduling engine in reference with the splitting technology of the massive parallel processing (MPP) database. Using this engine, tasks can be split and distributed to different sub-nodes for parallel execution. Through cross-domain querymodule, users can execute SQL commands on the datasets of different datacenters or network domains. Meanwhile, the system supports the distributed deployment, so as to reduce the construction cost by making full use of existing software and hardware resources and equipments. We test the system functions and performance on a 80 nodes cluster, and compares with Hive. The result suggested that system performance is improved by 2-3 times than Hive and the function designed can be performed correctly.