Search

Cloud vendors provide ready-to-use distributed DBMS solutions as a service. While the provisioning of a DBMS is usually fully automated, customers typically still have to make important design decisions which were traditionally made by the database administrator such as finding an optimal partitioning scheme for a given database schema and workload. In this paper, we introduce a new learned partitioning advisor based on Deep Reinforcement Learning (DRL) for OLAP-style workloads. The main idea is that a DRL agent learns the cost tradeoffs of different partitioning schemes and can thus automate the partitioning decision. In the evaluation, we show that our advisor is able to find non-trivial partitionings for a wide range of workloads and outperforms more classical approaches for automated partitioning design.

Attempts to improve database performance have led to the use of high-speed and large nonvolatile memory as a durable database medium instead of existing storage devices. For such database systems, the cost of memory access instead of I/O processing decreases, and the CPU cost increases relative to the most suitable join method selected for a database query to minimize query execution time. Moreover, for reducing database administration overhead, the cost model is also required to be capable of application to different generation CPUs through minor modification. In this paper, the measurement-based cost calculation method (MBCC) for solving these requirements is described. This cost calculation formula using MBCC is based on the behavior of the instruction issuing part in the CPU instruction pipeline and the tendency between the statistical information measured by the CPU performance monitor The cost calculation formulas are formed into parts for each element separated into elements repeatedly appearing in the access path of the join, and the cost is calculated combining the parts for an arbitrary number of join tables. In addition, MBCC enables the join cost calculation formula to support a CPU with architecture from a different generation without the need to re-measure the statistical information of the CPU. The evaluation of the accuracy of MBCC cost calculations revealed that the difference between the predicted cross point and the measured cross point was reduced by 74% to 95% compared with the difference between the cross point obtained by the conventional method and the measured cross point, and the updated cost evaluation formulas estimated the cost of joining different generation-based CPUs accurately in 66% of the test cases. In conclusion, the database system using the proposed cost calculation method can select the best join method and can be applied with CPUs from different generations.

Configuration tuning is vital to optimize the performance of database management system (DBMS). It becomes more tedious and urgent for cloud databases (CDB) due to the diverse database instances and query workloads, which make the database administrator (DBA) incompetent. Although there are some studies on automatic DBMS configuration tuning, they have several limitations. Firstly, they adopt a pipelined learning model but cannot optimize the overall performance in an end-to-end manner. Secondly, they rely on large-scale high-quality training samples which are hard to obtain. Thirdly, there are a large number of knobs that are in continuous space and have unseen dependencies, and they cannot recommend reasonable configurations in such high-dimensional continuous space. Lastly, in cloud environment, they can hardly cope with the changes of hardware configurations and workloads, and have poor adaptability. To address these challenges, we design an end-to-end automatic CDB tuning system, CDBTune, using deep reinforcement learning (RL). CDBTune utilizes the deep deterministic policy gradient method to find the optimal configurations in high-dimensional continuous space. CDBTune adopts a try-and-error strategy to learn knob settings with a limited number of samples to accomplish the initial training, which alleviates the difficulty of collecting massive high-quality samples. CDBTune adopts the reward-feedback mechanism in RL instead of traditional regression, which enables end-to-end learning and accelerates the convergence speed of our model and improves efficiency of online tuning. We conducted extensive experiments under 6 different workloads on real cloud databases to demonstrate the superiority of CDBTune. Experimental results showed that CDBTune had a good adaptability and significantly outperformed the state-of-the-art tuning tools and DBA experts.

Database tuning is a crucial task to address the performance of information systems that deal with a considerable amount of information stored in databases. Current tuning tools are very platform-specific and do not provide adequate support for the database administrator to reason about performance improvement suggestions. In this paper, we discuss several architectural and implementation decisions of Outer-Tuning, our framework that supports database tuning. Outer-Tuning follows a model-driven development and a modular architecture design, which enabled several benefits. This paper contributes with: (i) the architectural design model adopted in Outer-Tuning, which combines imperative and declarative programming; (ii) the discussions and steps to integrate several software components; and (iii) the actual framework implementation. We assess our framework with an experiment using the TPC-H benchmark. The results evidence that Outer-Tuning infers useful tuning actions and supports the DBA by providing a more semantic environment to create and adapt tuning heuristics using concepts closer to his/her domain, and also relevant information on the rationale of the tuning actions through a friendly web interface.

Many modern database-backed web applications are built upon Object Relational Mapping (ORM) frameworks. While such frame- works ease application development by abstracting persistent data as objects, such convenience comes with a performance cost. In this paper, we studied 27 real-world open-source applications built on top of the popular Ruby on Rails ORM framework, with the goal to understand the database-related performance inefficiencies in these applications. We discovered a number of inefficiencies rang- ing from physical design issues to how queries are expressed in the application code. We applied static program analysis to identify and measure how prevalent these issues are, then suggested techniques to alleviate these issues and measured the potential performance gain as a result. These techniques significantly reduce database query time (up to 91%) and the webpage response time (up to 98%). Our study provides guidance to the design of future database engines and ORM frameworks to support database application that are performant yet without sacrificing programmability.

Database workloads severely underutilize their micro-architectural resources. This results in inefficient use of the chip's limited power budget. While ARM's server-grade processor is a promising energy-efficient alternative, it nevertheless wastes significant portion of the power-budget when running a database workload. In this paper, we systematically explore the ways of improving energy efficiency when running database workloads. We start by describing software-only and hardware-only optimizations, and then discuss two hardware/software co-design approaches. We finally propose combining a diverse set of hardware/software co-designed systems into a heterogeneous, energy-efficient database machine.

SQL-On-Hadoop systems translate a given query into several MapReduce jobs. Each job executes a different set of query operators over different input data sets, which leads to distinct resource consumption patterns. Once each job has a different resource consumption pattern they should receive tailor made tuning setup. However, SQL-On-Hadoop systems propagate the same tuning to every job in the query plan because they are not able to apply a specific tuning setup per job. Propagating the same tuning through the query plan is a problem because it drives the query to sub-optimal performance and drives tuning advisors to re-profile similar jobs several times. In our research we characterize this problem and propose a solution. Preliminary results show that our approach can reduce the number of profiles required by tuning advisors in 67% for TPC-H.

Database management system (DBMS) configuration tuning is an essential aspect of any data-intensive application effort. But this is historically a difficult task because DBMSs have hundreds of configuration "knobs" that control everything in the system, such as the amount of memory to use for caches and how often data is written to storage. The problem with these knobs is that they are not standardized (i.e., two DBMSs use a different name for the same knob), not independent (i.e., changing one knob can impact others), and not universal (i.e., what works for one application may be sub-optimal for another). Worse, information about the effects of the knobs typically comes only from (expensive) experience. To overcome these challenges, we present an automated approach that leverages past experience and collects new information to tune DBMS configurations: we use a combination of supervised and unsupervised machine learning methods to (1) select the most impactful knobs, (2) map unseen database workloads to previous workloads from which we can transfer experience, and (3) recommend knob settings. We implemented our techniques in a new tool called OtterTune and tested it on two DBMSs. Our evaluation shows that OtterTune recommends configurations that are as good as or better than ones generated by existing tools or a human expert.

Emerging next-generation non-volatile memory (NVM) technologies, including PCM and STT-MRAM, provide low latency, high bandwidth, non-volatility, and high capacity. Recently, NVM has drawn much attention from the database community to improve transaction processing (e.g., write-ahead logging). As a complement to existing work, we investigate NVM for atomic page updates in database systems. Atomic page update is required for consistent recovery, and database systems generally use redundant writes to support atomic page update (e.g., doublewrite buffer in MySQL/InnoDB). However, redundant writes negatively affect the database and lifetime of storage devices (e.g., flash-based SSD). In this paper, we present LSBM, a log-structured buffer manager which updates pages atomically and efficiently using NVM. LSBM updates pages in an out-of-place manner to achieve atomicity and performs page buffering to improve performance. LSBM also reclaims the buffered pages on the fly to reduce the write traffic to storage. We implement LSBM using an NVDIMM as NVM and ported it to MySQL/InnoDB. The experimental results demonstrate that LSBM improves database performance and reduces write traffic on OLTP workloads.

The database performance is one of the most important quality indicators that companies are looking for to choose their appropriate database management systems. Quantifying this performance is usually performed by the means of mathematical cost models. Due to the importance of these models, for each evolution of the database technology pushes researchers to revisit or to propose cost models in order to integrate new dimensions brought by that evolution. As a consequence, a huge number of cost models exists. To exploit them, we need to find their respective scientific papers. This situation is in contradiction with Era Sharing, because it reduces reuse of these cost models by researchers, students (from third world countries), etc. and even more it penalizes the reproduction of experiments that intensively use these cost models. In this paper, we propose a framework for cost models dedicated to query processing and optimization. We first propose a common repository, called, MetricStore, to store metrics of cost model units. Secondly, thanks to model-driven engineering facilities, the repository offers capabilities aiming at publishing, searching and reusing cost models through a suitable user interface. Tool support is fully available.

Despite having a significant impact on overall data system performance, database drivers connecting the application to the database system have not innovated at the same pace as the database systems themselves. This work describes a database driver designed for the requirements of cloud-based systems requiring flexibility, high availability, scaling, and performance. The unique contribution is a rule-based query routing system that supports real-time configurations and optimizations without requiring any changes to the application code or database system. With the increasing migration of applications and databases to the cloud as well as different database technologies such as NoSQL systems, this flexibility allows application owners to optimize and migrate legacy applications to exploit the advantages of new database technologies. Experimental results demonstrate how queries cached by the driver can improve query response times by an order of magnitude and reduce the overall load on the database system by up to 50+.

In the verifiable database (VDB) model, a computationally weak client (database owner) delegates his database management to a database service provider on the cloud, which is considered untrusted third party, while users can query the data and verify the integrity of query results. Since the process can be computationally costly and has a limited support for sophisticated query types such as aggregated queries, we propose in this paper a framework that helps bridge the gap between security and practicality trade-offs. The proposed framework remodels the verifiable database problem using Stackelberg security game. In the new model, the database owner creates and uploads to the database service provider the database and its authentication structure (AS). Next, the game is played between the defender (verifier), who is a trusted party to the database owner and runs scheduled randomized verifications using Stackelberg mixed strategy, and the database service provider. The idea is to randomize the verification schedule in an optimized way that grants the optimal payoff for the verifier while making it extremely hard for the database service provider or any attacker to figure out which part of the database is being verified next. We have implemented and compared the proposed model performance with a uniform randomization model. Simulation results show that the proposed model outperforms the uniform randomization model. Furthermore, we have evaluated the efficiency of the proposed model against different cost metrics.

Aiming at the characteristics of telemetry data with different types, varied parameters and various processing algorithms, a database-based design method is proposed. This method can store the variable information in the database, and the database information could be loaded before the software runs. At the same time, the database and the software will not be changed while information varies. Besides, aiming at the characteristics of large telemetry data and high frame frequency, a data output frequency control method based on the identification bit is proposed. The test results show that this method can effectively control the output frequency of data according to user needs.

Query optimization is hard and the current proliferation of "modern" hardware does nothing to make it any easier. In addition, the tools that are commonly used by performance engineers, such as compiler intrinsics, static analyzers or hardware performance counters are neither integrated with data management systems nor easy to learn. This fact makes it (unnecessarily) hard to educate engineers, to prototype and to optimize database query plans for modern hardware. To address this problem, we developed a system called Candomble that lets database performance engineers interactively examine, optimize and evaluate query plans using a touch-based interface. Candomble puts attendants in the place of a physical query optimizer that has to rewrite a physical query plan into a better equivalent plan. Attendants experience the challenges when ad-hoc optimizing a physical plan for processing devices such as GPUs and CPUs and capture their gained knowledge in rules to be used by a rule-based optimizer.

Running an online transaction processing (OLTP) system is one of the most daunting tasks required of database administrators (DBAs). As businesses rely on OLTP databases to support their mission-critical and real-time applications, poor database performance directly impacts their revenue and user experience. As a result, DBAs constantly monitor, diagnose, and rectify any performance decays. Unfortunately, the manual process of debugging and diagnosing OLTP performance problems is extremely tedious and non-trivial. Rather than being caused by a single slow query, performance problems in OLTP databases are often due to a large number of concurrent and competing transactions adding up to compounded, non-linear effects that are difficult to isolate. Sudden changes in request volume, transactional patterns, network traffic, or data distribution can cause previously abundant resources to become scarce, and the performance to plummet. This paper presents a practical tool for assisting DBAs in quickly and reliably diagnosing performance problems in an OLTP database. By analyzing hundreds of statistics and configurations collected over the lifetime of the system, our algorithm quickly identifies a small set of potential causes and presents them to the DBA. The root-cause established by the DBA is reincorporated into our algorithm as a new causal model to improve future diagnoses. Our experiments show that this algorithm is substantially more accurate than the state-of-the-art algorithm in finding correct explanations.

Database consistency and recoverability require guaranteeing write atomicity for one or more pages. However, contemporary database systems consider write operations non-atomic. Thus, many database storage engines have traditionally relied on either journaling or copy-on-write approaches for atomic propagation of updated pages to the storage. This reliance achieves write atomicity at the cost of various write amplifications such as redundant writes, tree-wandering, and compaction. This write amplification results in reduced performance and, for flash storage, accelerates device wear-out. In this paper, we propose a flash storage interface, SHARE. Being able to explicitly remap the address mapping inside flash storage using SHARE interface enables host-side database storage engines to achieve write atomicity without causing write amplification. We have implemented SHARE on a real SSD board, OpenSSD, and modified MySQL/InnoDB and Couchbase NoSQL storage engines to make them compatible with the extended SHARE interface. Our experimental results show that this SHARE-based MySQL/InnoDB and Couchbase configurations can significantly boost database performance. In particular, the inevitable and costly Couchbase compaction process can complete without copying any data pages.

NoSQL database systems nowadays need to make trade-offs to optimize for their applications by considering consistency, availability, or partition tolerance. While a hybrid database system can use various kinds of database software and take advantage of their features for individual applications and workloads, how to make the database software work together to achieve the highest performance is still a challenging problem. In this paper, we propose Attila, a data-oriented load balancer that monitors the performance of each database node, dynamically detects the hot spots in a hybrid database system, and migrates data to improve the overall throughput. We also provide an extendable and flexible database interface for integrating NoSQL databases and adding database operations. Our experiment results show that, in average, the additional overhead of Attila is about 140--200 microseconds, residing in request transmission and parsing, which is acceptable, considering the improved throughput from the hybrid database system.

Modern in-memory database systems perform transactions an order of magnitude faster than conventional database systems. While in-memory database systems can read the database without I/O, database updates can generate a substantial amount of I/O, since updates must normally be written to persistent secondary storage to ensure that they are durable. In this paper we present a study of storage managers for in-memory database systems, with the goal of characterizing their I/O efficiency. We model the storage efficiency of two classes of storage managers: those that perform in-place updates in secondary storage, and those that use copy-on-write. Our models allow us to make meaningful, quantitative comparisons of storage managers' I/O efficiencies under a variety of conditions.

Hot standby techniques are widely used to implement highly available database systems. These techniques make use of two separate copies of the database, an active copy and a backup that is managed by the standby. The two database copies are stored independently and synchronized by the database systems that manage them. However, database systems deployed in computing clouds often have access to reliable persistent storage that can be shared by multiple servers. In this paper we consider how hot standby techniques can be improved in such settings. We present SHADOW systems, a novel approach to hot standby high availability. Like other database systems that use shared storage, SHADOW systems push the task of managing database replication out of the database system and into the underlying storage service, simplifying the database system. Unlike other systems, SHADOW systems also provide write offloading, which frees the active database system from the need to update the persistent database. Instead, that responsibility is placed on the standby system. We present the results of a performance evaluation using a SHADOW prototype on Amazon's cloud, showing that write offloading enables SHADOW to outperform traditional hot standby replication and even a standalone DBMS that does not provide high availability.

A fundamental problem in database systems is choosing the best physical design, i.e., a small set of auxiliary structures that enable the fastest execution of future queries. Almost all commercial databases come with designer tools that create a number of indices or materialized views (together comprising the physical design) that they exploit during query processing. Existing designers are what we call nominal; that is, they assume that their input parameters are precisely known and equal to some nominal values. For instance, since future workload is often not known a priori, it is common for these tools to optimize for past workloads in hopes that future queries and data will be similar. In practice, however, these parameters are often noisy or missing. Since nominal designers do not take the influence of such uncertainties into account, they find designs that are sub-optimal and remarkably brittle. Often, as soon as the future workload deviates from the past, their overall performance falls off a cliff, leading to customer discontent and expensive redesigns. Thus, we propose a new type of database designer that is robust against parameter uncertainties, so that overall performance degrades more gracefully when future workloads deviate from the past. Users express their risk tolerance by deciding on how much nominal optimality they are willing to trade for attaining their desired level of robustness against uncertain situations. To the best of our knowledge, this paper is the first to adopt the recent breakthroughs in the theory of robust optimization to build a practical framework for solving some of the most fundamental problems in databases, replacing today's brittle designs with a principled world of robust designs that can guarantee predictable and consistent performance.

Bare-metal clouds are an emerging infrastructure-as-a-service (IaaS) that leases physical machines (bare-metal instances) rather than virtual machines, allowing resource-intensive applications to have exclusive access to physical hardware. Unfortunately, bare-metal instances require time-consuming or OS-specific tasks for deployment due to the lack of virtualization layers, thereby sacrificing several beneficial features of traditional IaaS clouds such as agility, elasticity, and OS transparency. We present BMcast, an OS deployment system with a special-purpose de-virtualizable virtual machine monitor (VMM) that supports quick and OS-transparent startup of bare-metal instances. BMcast performs streaming OS deployment while allowing direct access to physical hardware from the guest OS, and then disappears after completing the deployment. Quick startup of instances improves agility and elasticity significantly, and OS transparency greatly simplifies management tasks for cloud customers. Experimental results have confirmed that BMcast initiated a bare-metal instance 8.6 times faster than image copying, and database performance on BMcast during streaming OS deployment was comparable to that on a state-of-the-art VMM without performing deployment. BMcast incurred zero overhead after de-virtualization.

The choice of a particular NoSQL database imposes a specific distributed software architecture and data model, and is a major determinant of the overall system throughput. NoSQL database performance is in turn strongly influenced by how well the data model and query capabilities fit the application use cases, and so system-specific testing and characterization is required. This paper presents a method and the results of a study that selected among three NoSQL databases for a large, distributed healthcare organization. While the method and study considered consistency, availability, and partition tolerance (CAP) tradeoffs, and other quality attributes that influence the selection decision, this paper reports on the performance evaluation method and results. In our testing, a typical workload and configuration produced throughput that varied from 225 to 3200 operations per second between database products, while read operation latency varied by a factor of 5 and write latency by a factor of 4 (with the highest throughput product delivering the highest latency). We also found that achieving strong consistency reduced throughput by 10-25% compared to eventual consistency.

Large amounts of data often require expensive and time-consuming analysis. Therefore, highly scalable and efficient techniques are necessary to process, analyze and discover useful information. Database sampling has proven to be a powerful method to surpass these limitations. Using only a sample of the original large database brings the benefit of obtaining useful information faster, at the potential expense of lower accuracy. In this paper, we demonstrate \vfds, a novel fast database sampling system that maintains the referential integrity of the data. The system is developed over the open-source database management system, MySQL. We present various scenarios to demonstrate the effectiveness of VFDS in approximate query answering, sample size, and execution time, on both real and synthetic databases.

From social networking websites to bank transactions, we interact with data-intensive applications every day. Such applications are typically hosted on an application server that interacts with a database server to manipulate persistent data. To make such applications efficient, developers face the daunting task of mastering the intricacies of both the programming system and the database management system. For instance, while many application features can be implemented in either the application or pushed into the database, it is difficult for a programmer to decide where to place a given computation as the decision is typically workload-driven. Unfortunately, making the wrong choice often results in drastic performance hit.In this talk, I will show how examining both the programming system and the database management system at the same time allows us to significantly improve the performance of data-intensive applications. To illustrate such cross-system optimization opportunities, I have designed, built, and evaluated three systems: Query By Synthesis, a tool that converts functionality written as imperative code into relational queries; Sloth, a system that combines queries embedded in applications into batches; and Pyxis, a system that seamlessly moves computation between application and database servers. Using real-world examples, I will show that these systems allow orders of magnitude performance improvement and graceful adaptation to changing server environments while preserving the high-level programming interface to the developer. I will furthermore highlight research opportunities and challenges in applying similar techniques to other system problems.

The increasing heterogeneity in hardware systems gives developers many opportunities to add more functionality and computational power to the system. As a consequence, modern database systems will need to be able to adapt to a wide variety of heterogeneous architectures. While porting single operators to accelerator architectures is well-understood, a more generic approach is needed for the whole database system. In prior work, we presented a generic hardware-oblivious database system, where the operators can be executed on the main processor as well as on a large number of accelerator architectures. However, to achieve fully heterogeneous query processing, placement decisions are needed for the database operators. We enhance the presented system with heterogeneity-aware operator placement (HOP) to take a major step towards designing a database system that can efficiently exploit highly heterogeneous hardware environments. In this demonstration, we are focusing on the placement-integration aspect as well as presenting the resulting database system.

Object-Relational Mapping (ORM) provides developers a conceptual abstraction for mapping the application code to the underlying databases. ORM is widely used in industry due to its convenience; permitting developers to focus on developing the business logic without worrying too much about the database access details. However, developers often write ORM code without considering the impact of such code on database performance, leading to cause transactions with timeouts or hangs in large-scale systems. Unfortunately, there is little support to help developers automatically detect suboptimal database accesses. In this paper, we propose an automated framework to detect ORM performance anti-patterns. Our framework automatically flags performance anti-patterns in the source code. Furthermore, as there could be hundreds or even thousands of instances of anti-patterns, our framework provides sup- port to prioritize performance bug fixes based on a statistically rigorous performance assessment. We have successfully evaluated our framework on two open source and one large-scale industrial systems. Our case studies show that our framework can detect new and known real-world performance bugs and that fixing the detected performance anti- patterns can improve the system response time by up to 98%.

There has been tremendous growth over the last few years of applications employing column-oriented databases (a common type of non-relational database) to store unstructured or semi-structured data. These applications have queries present in the implementation which need to be tested. Despite many literature works focus on testing relational database applications having SQL queries, there has been very little effort in testing of applications having non-relational database queries, in particular, column-oriented database queries. This paper proposes to apply fault injection-based testing approach (mutation analysis) for Cassandra, which is a popular column-oriented non-relational database. We propose a set of mutation operators for queries written in Cassandra Query Language (CQL) and mutant killing criteria. We also perform an initial evaluation for two types of CQL commands: key space and columnfamily creation and management. Our proposed approach will be the beginning step for developers and practitioners for applying fault injection based testing concept in the vast area of testing applications having non-relational database queries.

As the amount of memory in database systems grows, entire database tables, or even databases, are able to fit in the system's memory, making in-memory database operations more prevalent. This shift from disk-based to in-memory database systems has contributed to a move from row-wise to columnar data storage. Furthermore, common database workloads have grown beyond online transaction processing (OLTP) to include online analytical processing and data mining. These workloads analyze huge datasets that are often irregular and not indexed, making traditional database operations like joins much more expensive.In this paper we explore using dedicated hardware to accelerate in-memory database operations. We present hardware to accelerate the selection process of compacting a single column into a linear column of selected data, joining two sorted columns via merging, and sorting a column. Finally, we put these primitives together to accelerate an entire join operation. We implement a prototype of this system using FPGAs and show substantial improvements in both absolute throughput and utilization of memory bandwidth. Using the prototype as a guide, we explore how the hardware resources required by our design change with the desired throughput.

Storage Class Memory (SCM) has the potential to significantly improve database performance. This potential has been well documented for throughput [4] and response time [25, 22]. In this paper we show that SCM has also the potential to significantly improve restart performance, a shortcoming of traditional main memory database systems. We present SOFORT, a hybrid SCM-DRAM storage engine that leverages full capabilities of SCM by doing away with a traditional log and updating the persisted data in place in small increments. We show that we can achieve restart times of a few seconds independent of instance size and transaction volume without significantly impacting transaction throughput.

We propose a scheme for outsourcing Private Information Retrieval (PIR) to untrusted servers while protecting the privacy of the database owner as well as that of the database clients. We observe that by layering PIR on top of an Oblivious RAM (ORAM) data layout, we provide the ability for the database owner to perform private writes, while database clients can perform private reads from the database even while the owner is offline. Our system is compatible with existing PIR access control and pricing schemes on a per-record basis for these reads. This extends the usual ORAM model by allowing multiple database readers without requiring trusted hardware; indeed, almost all of the computation in our scheme during reads is performed by untrusted cloud servers. We make a second observation that the database owner can always conduct a private read as an ordinary database client, and the private write protocol does not have to provide a "read" functionality as a standard ORAM protocol does. Based on the two observations, we construct an end-to-end system that privately updates a 1 MB record in a 1 TB database with an amortized end-to-end response time as low as 300 ms when the database owner has a fast network connection to the database servers, and about 1 minute over a slow ADSL connection. Private read times by the database readers are on the order of seconds in either case.

Join query optimization has a direct impact on the performance of a database system. This work presents an evolutionary multi-agent system applied to the join ordering problem related to database query planning. The proposed algorithm was implemented and embedded in the core of a database management system (DBMS). Parameters of the algorithm were calibrated by means of a factorial design and an analysis based on the variance. The algorithm was compared with the official query planner of the H2 DBMS, using a methodology based on benchmark tests. The results show that the proposed evolutionary multi-agent system was able to generate solutions associated with low execution costs in the majority of the cases.

The global pool of data is growing at 2.5 quintillion bytes per day, with 90% of it produced in the last two years alone [24]. There is no doubt the era of big data has arrived. This paper explores targeted deployment of hardware accelerators to improve the throughput and energy efficiency of large-scale data processing. In particular, data partitioning is a critical operation for manipulating large data sets. It is often the limiting factor in database performance and represents a significant fraction of the overall runtime of large data queries. To accelerate partitioning, this paper describes a hardware accelerator for range partitioning, or HARP, and a hardware-software data streaming framework. The streaming framework offers a seamless execution environment for streaming accelerators such as HARP. Together, HARP and the streaming framework provide an order of magnitude improvement in partitioning performance and energy. A detailed analysis of a 32 nm physical design shows 7.8 times the throughput of a highly optimized and optimistic software implementation, while consuming just 6.9% of the area and 4.3% of the power of a single Xeon core in the same technology generation.

Database benchmarks are an important tool for database researchers and practitioners that ease the process of making informed comparisons between different database hardware, software and configurations. Large scale web services such as social networks are a major and growing database application area, but currently there are few benchmarks that accurately model web service workloads.In this paper we present a new synthetic benchmark called LinkBench. LinkBench is based on traces from production databases that store "social graph" data at Facebook, a major social network. We characterize the data and query workload in many dimensions, and use the insights gained to construct a realistic synthetic benchmark. LinkBench provides a realistic and challenging test for persistent storage of social and web service data, filling a gap in the available tools for researchers, developers and administrators.

In this paper, we study the feasibility of using performance models to support an analysis of the computational load in local database caching. Local database caching is used, for example, to relieve the computational load of a main DBMS in large deployments of web applications. This is done by caching part of the database contents in a DBMS local to the application server. While for common scenarios with a high browse-to-order ratio this frequently results in a reduction of the computational load, there are also scenarios in which there is not a clear advantage of local database caching. This is especially the case when each local write also results in a write into the main database server, thereby increasing the computational resource requirements. In this paper, two methods are presented which can be used to obtain significant computational parameters. We demonstrate how these parameters are used on two different hardware platforms and show that a reasonable prediction accuracy within actual measured results can be reached.

Imagine that your database has all the right indexes. Its buffer manager has been tuned to give a high hit ratio, the buffer fits in RAM, and the data is well distributed on disk. You're done, right? Well, no, because the application code might be poorly written. It might include delinquent design patterns. The demoed tuning tool AppSleuth will find those delinquent design patterns but it is the demo visitor's job to fix them.

The need to perform testing and tuning of database instances with production-like workloads (W), configurations (C), data (D), and resources (R) arises routinely. The further W, C, D, and R used in testing and tuning deviate from what is observed on the production database instance, the lower is the trustworthiness of the testing and tuning tasks done. For example, it is common to hear about performance degradation observed after the production database is upgraded from one software version to another. A typical cause of this problem is that the W, C, D, or R used during upgrade testing differed in some way from that on the production database. Performing testing and tuning tasks in principled and automated ways is very important, especially since---spurred by innovations in cloud computing---the number of database instances that a database administrator (DBA) has to manage is growing rapidly.We present Flex, a platform for trustworthy testing and tuning of production database instances. Flex gives DBAs a high-level language, called Slang, to specify definitions and objectives regarding running experiments for testing and tuning. Flex's orchestrator schedules and runs these experiments in an automated manner that meets the DBA-specified objectives. Flex has been fully prototyped. We present results from a comprehensive empirical evaluation that reveals the effectiveness of Flex on diverse problems such as upgrade testing, near-real-time testing to detect corruption of data, and server configuration tuning. We also report on our experiences taking some of the testing and tuning software described in the literature and porting them to run on the Flex platform.

Excellent work ([1]-[6]) has shown that memory management and transaction concurrency levels can often be tuned automatically by the database management systems. Other excellent work ([7]]-[14]) has shown how to use the optimizer to do automatic physical design or to make the optimizer itself more self-adaptive ([15]-[17]). Our performance tuning experience across various industries (finance, gaming, data warehouses, and travel) has shown that enormous additional tuning benefits (sometimes amounting to orders of magnitude) can come from reengineering application code and table design. The question is: can a tool help in this effort? We believe so. We present a tool called AppSleuth that parses application code and the tracing log for two popular database management systems in order to lead a competent tuner to the hot spots in an application. This paper discusses (i) representative application "delinquent design patterns", (ii) an application code parser to find them, (iii) a log parser to identify the patterns that are critical, and (iv) a display to give a global view of the issue. We present an extended sanitized case study from a real travel application to show the results of the tool at different stages of a tuning engagement, yielding a 300 fold improvement. This is the first tool of its kind that we know of.

peer-reviewed Managing large amounts of information is one of the most expensive, time-consuming and non-trivial activities and it usually requires expert knowledge. In a wide range of application areas, such as data mining, histogram construction, approximate query evaluation, and software validation, handling exponentially growing databases has become a dif- cult challenge, and a subset of the data is generally preferred. As a solution to the current challenges in managing large amounts of data, database sampling from the operational data available has proved to be a powerful technique. However, none of the existing sampling approaches consider the dependencies between the data in a relational database. In this paper, we propose a novel approach towards constructing a realistic testing environment, by analyzing the distribution of data in the original database along these dependencies before sampling, so that the sample database is representative to the original database.

Large system development typically involves changes to code and database components. In these environments, testing continues to be the biggest challenge. Since release cycles include both database and code changes, thorough testing of existing schemas and code base is paramount. Code needs to be tested with a static database, database changes need to be tested on a static code base, and finally, both changes need to be tested together.In this paper we present the testing framework built during the development of Morgan Stanley's enterprise-wide time-series database, Horizon. The database, built using Kx's KDB+/Q, holds both tick-by-tick data originating from real-time feeds as well as periodic data received at regular intervals from vendors. The plant consists of several hundred cooperating processes which connect to real-time feeds, create in-memory databases, persist in-memory databases to disk on a regular basis, manage data set locations, and service client connections. At present, the database plant's size is approximately 1PB with an anticipated growth of .5PB/year.

Solid state disks (or flash disks) are decreasing in cost per gigabyte and are being incorporated into many appliances, such as the Oracle Database Appliance [8]. Databases--and more specifically data warehouses--are often utilized to support large scale data analysis and decision support systems. Decision makers prefer information in real time. Traditional storage systems that are based on magnetic disks achieve high performance by utilizing many disks for parallel operations in RAID arrays. However, this performance is only possible if requests represent a reasonable fraction of the RAID stripe size, or I/O transactions will suffer from high overhead. Solid state disks have the potential to increase the speed of data retrieval for mission critical workloads that require real time applications, such as analytic dashboards. However, solid state disks behave differently than magnetic hard disks due to the limitations of rewriting NAND flash based blocks. Therefore, this work presents benchmark results for a modern relational database that stores data on solid state disks, and contrasts this performance to a ten disk RAID 10 array, a traditional storage design for high performance database data blocks. The preliminary results show that a single solid state disk is able to outperform the array for queries summarizing a data set for a variety of OLAP cube dimensions. Future work will explore the low level database performance in more detail.

This paper proposes strategies for query execution in mobile databases distributed over a mobile network. The underlying network architecture of the database is based on 3G telephone networks.

PurposeToday's database management systems implement sophisticated access control mechanisms to prevent unauthorized access and modifications. For instance, this is an important basic requirement for SOX (Sarbanes‐Oxley Act) compliance, whereby every past transaction has to be traceable at any time. However, malicious database administrators may still be able to bypass the security mechanisms in order to make hidden modifications to the database. This paper aims to address these issues.Design/methodology/approachIn this paper the authors define a novel signature of a B+‐tree, a widely‐used storage structure in database management systems, and propose its utilization for supporting the logging in databases. This additional logging mechanism is especially useful in conjunction with forensic techniques that directly target the underlying tree‐structure of an index. Several techniques for applying this signature in the context of digital forensics on B+‐trees are proposed in the course of this paper. Furthermore, the authors' signature can be used to generate exact copies of an index for backup purposes, thereby enabling the owner to completely restore data, even on the structural level.FindingsFor database systems in enterprise environments, compliance to regulatory standards such as SOX (Sarbanes‐Oxley Act), whereby every past transaction has to be traceable at any time, is a fundamental requirement. Today's database management systems usually implement sophisticated access control mechanisms to prevent unauthorized access and modifications. Nonetheless malicious database administrators would be able to bypass the security mechanisms in order to make modifications to the database, while covering their tracks.Originality/valueIn this paper, the authors demonstrate how the tree structure of the underlying store engine can be used to enhance forensic logging mechanisms of the database. They define a novel signature for B+‐trees, which are used by the InnoDB storage engine. This signature stores the structure of database storage files and can help in reconstructing previous versions of the file for forensic purposes. Furthermore, the authors' signature can be used to generate exact copies of an index for backup purposes, thus enabling the owner to completely restore data, even on the structural level. The authors applied their concept to four real‐life scenarios in order to evaluate its effectiveness.

Typically, users interact with database systems by formulating queries. However, many times users do not have a clear understanding of their information needs or the exact content of the database, thus, their queries are of an exploratory nature. In this paper, we propose assisting users in database exploration by recommending to them additional items that are highly related with the items in the result of their original query. Such items are computed based on the most interesting sets of attribute values (or faSets) that appear in the result of the original user query. The interestingness of a faSet is defined based on its frequency both in the query result and in the database instance. Database frequency estimations rely on a novel approach that employs an e-tolerance closed rare faSets representation. We report evaluation results of the efficiency and effectiveness of our approach on both real and synthetic datasets.

A key advantage of Autonomic Database Management Systems will be their ability to manage according to business policies. Translating high-level business policies into low-level tuning actions and parameters is, however, a non-trivial problem as there is little similarity in the metrics used for measuring database performance and business performance. These translations can be simplified, however, by having a model that reflects the business policies. In this paper, we utilize an economic model to implement importance policy as a parameter for the allocation of system resources. The relative importance of the workloads can therefore be utilized in allocating system resources. We simulate the model in order to demonstrate the effectiveness of the approach. We present experiments to show the impact of the relative importance of workloads on the allocation of resources, specifically main memory for buffer space and CPU.

Software testing has been commonly used in assuring the quality of database applications. It is often prohibitively expensive to manually write quality tests for complex database applications. Automated test generation techniques, such as Dynamic Symbolic Execution (DSE), have been proposed to reduce human efforts in testing database applications. However, such techniques have two major limitations: (1) they assume that the database that the application under test interacts with is accessible, which may not always be true; and (2) they usually cannot create necessary database states as a part of the generated tests.To address the preceding limitations, we propose an approach that applies DSE to generate tests for a database application. Instead of using the actual database that the application interacts with, our approach produces and uses a mock database in test generation. A mock database mimics the behavior of an actual database by performing identical database operations on itself. We conducted two empirical evaluations on both a medical device and an open source software system to demonstrate that our approach can generate, without producing false warnings, tests with higher code coverage than conventional DSE-based techniques.

Many of the popular relational database management systems (RDBMS) provide features for operating in a distributed environment, such as remote table queries and updates, and support for distributed transactions. In practice, however, much application software targets a more minimal set of functionality than is offered by the SQL standards. Independently of the database tier, engineering concepts such as the enterprise service bus and service oriented architecture have led to the development of communication middleware to support distributed applications. For applications that require reliable delivery of messages, complex event processing, and integrated archiving of data, impedance mismatches are likely to emerge between the database system and the communications middleware---for example with respect to data-types, event filtering that is based on information in the database, and in terms of coordinating access control policy. This paper describes event-based middleware functionality that is supported directly within the database system. In contrast to previous approaches (e.g. being able to name remote tables in SQL statements), the programming of event-based communication operations within the database is explicit. We present initial performance results that compare an augmented PostgreSQL database system to an environment in which a database and an event-based middleware package are used side-by-side. These results demonstrate the viability of our approach.

A database application differs form regular applications in that some of its inputs may be database queries. The program will execute the queries on a database and may use any result values in its subsequent program logic. This means that a user-supplied query may determine the values that the application will use in subsequent branching conditions. At the same time, a new database application is often required to work well on a body of existing data stored in some large database. For systematic testing of database applications, recent techniques replace the existing database with carefully crafted mock databases. Mock databases return values that will trigger as many execution paths in the application as possible and thereby maximize overall code coverage of the database application.In this paper we offer an alternative approach to database application testing. Our goal is to support software engineers in focusing testing on the existing body of data the application is required to work well on. For that, we propose to side-step mock database generation and instead generate queries for the existing database. Our key insight is that we can use the information collected during previous program executions to systematically generate new queries that will maximize the coverage of the application under test, while guaranteeing that the generated test cases focus on the existing data.

Rising energy costs in large data centers are driving an agenda for energy-efficient computing. In this paper, we focus on the role of database software in affecting, and, ultimately, improving the energy efficiency of a server. We first characterize the power-use profiles of database operators under different configuration parameters. We find that common database operations can exercise the full dynamic power range of a server, and that the CPU power consumption of different operators, for the same CPU utilization, can differ by as much as 60%. We also find that for these operations CPU power does not vary linearly with CPU utilization. We then experiment with several classes of database systems and storage managers, varying parameters that span from different query plans to compression algorithms and from physical layout to CPU frequency and operating system scheduling. Contrary to what recent work has suggested, we find that within a single node intended for use in scale-out (shared-nothing) architectures, the most energy-efficient configuration is typically the highest performing one. We explain under which circumstances this is not the case, and argue that these circumstances do not warrant a retargeting of database system optimization goals. Further, our results reveal opportunities for cross-node energy optimizations and point out directions for new scale-out architectures.

The identification and processing of similarities in the data play a key role in multiple application scenarios. Several types of similarity-aware operations have been studied in the literature. However, in most of the previous work, similarity-aware operations are studied in isolation from other regular or similarity-aware operations. Furthermore, most of the previous research in the area considers a standalone implementation, i.e., without any integration with a database system. In this demonstration we present SimDB, a similarity-aware database management system. SimDB supports multiple similarity-aware operations as first-class database operators. We describe the architectural changes to implement the similarity-aware operators. In particular, we present the way conventional operators' implementation machinery is extended to support similarity-aware operators. We also show how these operators interact with other similarity-aware and regular operators. In particular, we show the effectiveness of multiple equivalence rules that can be used to extend cost-based query optimization to the case of similarity-ware operations.

iTuned is a tool that takes a SQL workload as input and recommends good settings for database configuration parameters such as buffer pool sizes, multi-programming level, and number of I/O daemons. iTuned also provides response-surface and sensitivity-analysis plots that help the end-user analyze the impact of each parameter. iTuned has the following novel features: (i) a technique called Adaptive Sampling that proactively brings in appropriate data through planned experiments to find high-impact parameters and high-performance parameter settings, (ii) an executor that supports on-line experiments in production database environments through a cycle-stealing paradigm that places minimal overhead on the production workload, and (iii) portability across different database systems. This demonstration will focus on the interesting use-case scenarios of iTuned, and show the effectiveness of recommended configuration settings on popular database benchmarks.