Supporting Social Data Observatory with Customizable Index Structures on HBase - Architecture and Performance

The intensive research activities in social data analysis in recent years suggest the necessity and great potential of a public social data observatory. To effectively support a social data observatory, the storage platform must satisfy its special requirements for loading and storage of Terabyte-level datasets, as well as efficient evaluation of queries involving analysis of the texts of millions of social updates. Traditional inverted indexing techniques do not meet such requirements due to their targeted use cases in text retrieval scenarios. To address these problems, we propose a general indexing framework, IndexedHBase, to build specially customized index structures for facilitating efficient queries, and employ the HBase system for distributed data storage. IndexedHBase is used to support the Truthy system that collects and analyzes data obtained through the Twitter streaming API. To handle the special queries in Truthy, we develop a parallel query evaluation strategy that can explore the customized index structures efficiently. We evaluate the performance of IndexedHBase on FutureGrid, and compare it with Riak, a widely adopted commercial NoSQL database system. The results show that IndexedHBase provides a data loading speed that is 6 times faster than Riak, and is significantly more efficient in evaluating queries involving large result sets.