Data-Centric Benchmarking

In data management, both system designers and users casually resort to performance evaluation. Performance evaluation by experimentation on a real system is generally referred to as benchmarking. The aim of this chapter is to present an overview of the major past and present state-of-the-art data-centric benchmarks. This review includes the TPC standard benchmarks, but also alternative or more specialized benchmarks. Surveyed benchmarks are categorized into three families: transaction benchmarks aimed at on-line transaction processing (OLTP), decision-support benchmarks aimed at on-line analysis processing (OLAP), and big data benchmarks. Issues, tradeoffs, and future trends in data-centric benchmarking are also discussed.