Search

Over the last decades, various cost-based optimizers have been proposed to generate optimal plans for SQL queries. These optimizers are key to achieving good performance in database systems and can speed up query execution. Still, they may need enormous expert efforts and perform poorly on complicated queries. Learning-based optimizers have been shown to achieve high-quality plans by learning from past experiences. However, these solutions treat each query separately and neglect the semantic equivalence among different queries. Intuitively, a high-quality plan may be obtained for a complicated query by discovering a simple equivalent query. Motivated by this, in this paper, we present Celo, a novel constraint-enhanced learned optimizer to directly integrate the equivalent information of queries into the learning-based model. We apply denial constraints to identify equivalent queries by replacing equivalent predicates. Given a query, we augment the query plans generated by the learning-based model with the high-quality plans of its equivalent queries. Then, a more potentially well-performed plan will be predicted among the augmented query plans. Extensive experiments using real-world datasets demonstrated that Celo outperforms the previous state-of-the-art (SOTA) results even with few constraints. [ABSTRACT FROM AUTHOR]

Government collaboration tasks are integral for grassroots governance and essential for government administration. Large-scale government collaboration tasks often involve multiple departments working together to solve complex tasks that require handling large amounts of data. However, existing offline processes make it difficult to manage complex collaboration scenarios and obtain timely execution results. We introduce the concept of a spatio-temporal task tree and propose a log-based incremental update method for updating statistical values in the tree structure to fulfill the need for real-time aggregation analysis of cross-temporal task execution situations. The spatio-temporal task tree fosters collaboration across various departments and organizational hierarchies to execute complex and large-scale tasks in government effectively. By flexible adaptations to different business models, this framework addresses the shortcomings of conventional systems in promoting inter-organizational collaboration. The log-based incremental update method for calculating statistical values within a tree structure allows for the updating of multi-dimensional data statistics in seconds. This approach effectively decreases computational expenses, enhances query efficiency, and improves real-time computation response. Results from experiments demonstrate the feasibility and high availability of the proposed approach, showcasing their potential to address the challenges associated with large-scale collaborative tasks in government affairs. [ABSTRACT FROM AUTHOR]

Modern cloud-native OLAP databases adopt a storage-disaggregation architecture that separates the management of computation and storage. A major bottleneck in such an architecture is the network connecting the computation and storage layers. Computation pushdown is a promising solution to tackle this issue, which offloads some computation tasks to the storage layer to reduce network traffic. This paper presents FlexPushdownDB (FPDB), where we revisit the design of computation pushdown in a storage-disaggregation architecture, and then introduce several optimizations to further accelerate query processing. First, FPDB supports hybrid query execution, which combines local computation on cached data and computation pushdown to cloud storage at a fine granularity. Within the cache, FPDB uses a novel Weighted-LFU cache replacement policy that takes into account the cost of pushdown computation. Second, we design adaptive pushdown as a new mechanism to avoid throttling the storage-layer computation during pushdown, which pushes the request back to the computation layer at runtime if the storage-layer computational resource is insufficient. Finally, we derive a general principle to identify pushdown-amenable computational tasks, by summarizing common patterns of pushdown capabilities in existing systems, and further propose two new pushdown operators, namely, selection bitmap and distributed data shuffle. Evaluation on SSB and TPC-H shows each optimization can improve the performance by 2.2 × , 1.9 × , and 3 × respectively. [ABSTRACT FROM AUTHOR]

Abstract Government collaboration tasks are integral for grassroots governance and essential for government administration. Large-scale government collaboration tasks often involve multiple departments working together to solve complex tasks that require handling large amounts of data. However, existing offline processes make it difficult to manage complex collaboration scenarios and obtain timely execution results. We introduce the concept of a spatio-temporal task tree and propose a log-based incremental update method for updating statistical values in the tree structure to fulfill the need for real-time aggregation analysis of cross-temporal task execution situations. The spatio-temporal task tree fosters collaboration across various departments and organizational hierarchies to execute complex and large-scale tasks in government effectively. By flexible adaptations to different business models, this framework addresses the shortcomings of conventional systems in promoting inter-organizational collaboration. The log-based incremental update method for calculating statistical values within a tree structure allows for the updating of multi-dimensional data statistics in seconds. This approach effectively decreases computational expenses, enhances query efficiency, and improves real-time computation response. Results from experiments demonstrate the feasibility and high availability of the proposed approach, showcasing their potential to address the challenges associated with large-scale collaborative tasks in government affairs.

With the continuous development of network technology, the scale of data has shown explosive growth, which leads gradually to replacing traditional single machine databases with distributed databases. Distributed databases solve large-scale data storage problems through collaborative work among nodes, but due to increased communication costs between nodes, its query efficiency is not as good as a standalone database. In a distributed architecture, the data of storage nodes is only used as redundancy for multiple backups, providing data recovery in case of system failure, and it is not utilized to improve query efficiency. In response to the above issues, this article proposes an operator parallel optimization strategy for distributed databases. By splitting key physical operators, the split sub requests are evenly distributed to multiple nodes in the storage layer, which are processed in parallel by multiple nodes, thereby reducing query response time. The above strategy has been applied on distributed database CBase, and experiments have shown that the parallel optimization strategy proposed in this paper can significantly shorten SQL request query time and improve system resource utilization.

The integration of Mixed Reality (MR) technology into Autonomous Vehicles (AVs) has ushered in a new era for the automotive industry, offering heightened safety, convenience, and passenger comfort. However, the substantial and varied data generated by MR-Connected AVs (MR-CAVs), encompassing both highly dynamic and static information, presents formidable challenges for efficient data management and retrieval. In this paper, we formulate our indexing problem as a constrained optimization problem, with the aim of maximizing the utility function that represents the overall performance of our indexing system. This optimization problem encompasses multiple decision variables and constraints, rendering it mathematically infeasible to solve directly. Therefore, we propose a heuristic algorithm to address the combinatorial complexity of the problem. Our heuristic indexing algorithm efficiently divides data into highly dynamic and static categories, distributing the index across Roadside Units (RSUs) and optimizing query processing. Our approach takes advantage of the computational capabilities of edge servers or RSUs to perform indexing operations, thereby shifting the burden away from the vehicles themselves. Our algorithm strategically places data in the cache, optimizing cache hit rate and space utilization while reducing latency. The quantitative evaluation demonstrates the superiority of our proposed scheme, with significant reductions in latency (averaging 27%−49.25%), a 30.75% improvement in throughput, a 22.50% enhancement in cache hit rate, and a 32%−50.75% improvement in space utilization compared to baseline schemes.

Complex event processing (CEP) systems have gained significant importance in various domains, such as finance, logistics, and security, where the real-time analysis of event streams is crucial. However, as the volume and complexity of event data continue to grow, optimizing the performance of CEP systems becomes a critical challenge. This paper investigates the impact of indexing strategies on the performance of databases handling complex event processing. We propose a novel indexing technique, called Hierarchical Temporal Indexing (HTI), specifically designed for the efficient processing of complex event queries. HTI leverages the temporal nature of event data and employs a multi-level indexing approach to optimize query execution. By combining temporal indexing with spatial- and attribute-based indexing, HTI aims to accelerate the retrieval and processing of relevant events, thereby improving overall query performance. In this study, we evaluate the effectiveness of HTI by implementing complex event queries on various CEP systems with different indexing strategies. We conduct a comprehensive performance analysis, measuring the query execution times and resource utilization (CPU, memory, etc.), and analyzing the execution plans and query optimization techniques employed by each system. Our experimental results demonstrate that the proposed HTI indexing strategy outperforms traditional indexing approaches, particularly for complex event queries involving temporal constraints and multi-dimensional event attributes. We provide insights into the strengths and weaknesses of each indexing strategy, identifying the factors that influence performance, such as data volume, query complexity, and event characteristics. Furthermore, we discuss the implications of our findings for the design and optimization of CEP systems, offering recommendations for indexing strategy selection based on the specific requirements and workload characteristics. Finally, we outline the potential limitations of our study and suggest future research directions in this domain. [ABSTRACT FROM AUTHOR]

In autonomous driving, the fusion of multiple sensors is considered essential to improve the accuracy and safety of 3D object detection. Currently, a fusion scheme combining low-cost cameras with highly robust radars can counteract the performance degradation caused by harsh environments. In this paper, we propose the IRBEVF-Q model, which mainly consists of BEV (Bird's Eye View) fusion coding module and an object decoder module.The BEV fusion coding module solves the problem of unified representation of different modal information by fusing the image and radar features through 3D spatial reference points as a medium. The query in the object decoder, as a core component, plays an important role in detection. In this paper, Heat Map-Guided Query Initialization (HGQI) and Dynamic Position Encoding (DPE) are proposed in query construction to increase the a priori information of the query. The Auxiliary Noise Query (ANQ) then helps to stabilize the matching. The experimental results demonstrate that the proposed fusion model IRBEVF-Q achieves an NDS of 0.575 and a mAP of 0.476 on the nuScenes test set. Compared to recent state-of-the-art methods, our model shows significant advantages, thus indicating that our approach contributes to improving detection accuracy. [ABSTRACT FROM AUTHOR]

The importance and complexity of spatial join operation resulted in the availability of many join algorithms, some of which are tailored for big-data platforms like Hadoop and Spark. The choice among them is not trivial and depends on different factors. This paper proposes the first machine-learning-based framework for spatial join query optimization which can accommodate both the characteristics of spatial datasets and the complexity of the different algorithms. The main challenge is how to develop portable cost models that once trained can be applied to any pair of input datasets, because they are able to extract the important input characteristics, such as data distribution and spatial partitioning, the logic of spatial join algorithms, and the relationship between the two input datasets. The proposed system defines a set of features that can be computed efficiently for the data to catch the intricate aspects of spatial join. Then, it uses these features to train five machine learning models that are used to identify the best spatial join algorithm. The first two are regression models that estimate two important measures of the spatial join performance and they act as the cost model. The third model chooses the best partitioning strategy to use with spatial join. The fourth and fifth models further tune two important parameters, number of partitions and plane-sweep direction, to get the best performance. Experiments on large-scale synthetic and real data show the efficiency of the proposed models over baseline methods. [ABSTRACT FROM AUTHOR]

Recently, numerous machine learning (ML) techniques have been applied to address database performance management problems, including cardinality estimation, cost modeling, optimal join order prediction, hint generation, etc. In this paper, we focus on query optimizer hints employed by users in their queries in order to mask some Query Optimizer deficiencies. We treat the query optimizer hints, bound to previous queries, as significant additional query metadata and learn to automatically predict which new queries will pose similar performance challenges and should therefore also be supported by query optimizer hints. To validate our approach, we have performed a number of experiments using real-life SQL workloads and we achieved promising results.

The article informs about identifying the root causes of suboptimality in database management systems (DBMS). Topics include introducing the notion of empirical suboptimality, proposing a predictive model to explain the relationship between various factors in query optimization and suboptimality, and testing the model on multiple DBMSes to quantify the relative contribution of each cause to observed suboptimality.

Cloud computing is trusted and used by millions of users around the world. Therefore, it becomes essential to offer trusted and efficient services to its users. Query processing and query optimization are the major components to provide optimal query plans for users through different steps. So far, the mathematical evaluation including algebraic transformation and selectivity estimation models are performed to improve the security while optimization and processing of query. Meanwhile, they have several equivalence rules to have various logical form. That problem was stated by single query processing at a time, but they are not efficient for dynamic operation of cloud since they are slower. Hence, this paper presents a review of improvement made to the processing and optimization of queries with the security enhancement in the past 8 years neglecting the tedious mathematical procedures. Along with the query processing techniques, the quantitative analysis of those techniques is provided in the paper based on deterministic and non-deterministic query processing techniques. As a result, it is found that when utilizing multiple query processing techniques, the host system may get loaded and struck, while the adaptive model works better by providing the process to host system based on its processing capability. [ABSTRACT FROM AUTHOR]

Cardinality estimation is one of the most important problems in query optimization. Recently, machine learning-based techniques have been proposed to effectively estimate cardinality, which can be broadly classified into query-driven and data-driven approaches. Query-driven approaches learn a regression model from a query to its cardinality, while data-driven approaches learn a distribution of tuples, select some samples that satisfy a SQL query, and use the data distributions of these selected tuples to estimate the cardinality of the SQL query. As query-driven methods rely on training queries, the estimation quality is not reliable when there are no high-quality training queries, while data-driven methods have no such limitation and have high adaptivity. In this work, we focus on data-driven methods. A good data-driven model should achieve three optimization goals. First, the model needs to capture data dependencies between columns and support large domain sizes (achieving high accuracy). Second, the model should achieve high inference efficiency, because many data samples are needed to estimate the cardinality (achieving low inference latency). Third, the model should not be too large (achieving a small model size). However, existing data-driven methods cannot simultaneously optimize the three goals. To address the limitations, we propose a novel cardinality estimator FACE , which leverages the normalizing flow-based model to learn a continuous joint distribution for relational data. FACE can transform a complex distribution over continuous random variables into a simple distribution (e.g., multivariate normal distribution) and use the probability density to estimate the cardinality for both sequential queries and parallel queries. First, we design a dequantization method to make data more "continuous." Second, we propose encoding and indexing techniques to handle Like predicates for string data. Third, we propose a Monte Carlo method to estimate the cardinality based on the FACE model. Fourth, we propose a grouping technique to process parallel queries. Fifth, we discuss how to support join queries. Experimental results show that our method significantly outperforms existing approaches in terms of estimation accuracy while keeping similar latency and model size. [ABSTRACT FROM AUTHOR]

Query execution time prediction is essential for database query optimization tasks, such as query scheduling, progress monitoring, and resource allocation. In the query execution time prediction tasks, the query plan is often used as the modeling object of a prediction model. Although the learning-based prediction models have been proposed to capture plan features, there are two limitations need to be considered more. First, the parent–child dependencies between plan operators can be captured, but the operator’s branch independence cannot be distinguished. Second, each operator’s output row is its following operator input, but the data iterate transfer operations between operators are ignored. In this study, we propose a graph query execution time prediction model containing a plan module, a query module, a plan-query module, and a prediction module to improve prediction effectiveness. Specifically, the plan module is used to capture the data iterate transfer operations and distinguish independent of branch operators; the query module is used to learn features of query terms that have an influence on the composition of operators; the plan-query interaction module is used to learn the logical correlations of plan and query. The experiment on datasets proves the effectiveness of the operator iterate-aware and query-plan interaction method in our proposed graph query execution prediction model.

Large-scale graph processing and Stream processing are two distinct computational paradigms for big data processing. Graph processing deals with computation on graphs of billions of vertices and edges. However, large-scale graph processing frameworks mostly work on graphs that do not change over time, while on the other end of the spectrum, stream processing operates on a continuous stream of data in real-time. Modern-day graphs change very rapidly over time, and finding patterns in temporally evolving graphs could reveal a lot of insights that can not be unveiled using traditional graph computations. We have proposed a novel framework called FlowGraph which could find patterns in dynamic and temporally evolving graphs. Computations on large-scale graphs are iterative and take multiple steps before final results can be calculated, which is very different from stream processing which is one-shot computation. Therefore, the most critical bottleneck of such a system is the time required to process the query. In this work, we have proposed a query optimization technique that could reduce the time required to process the pattern. The proposed system has an optimization technique that could reduce the time required to process the pattern, especially those related to the temporal evolution of the graph. Our method shows for eight clauses the execution time is reduced by 75%, we also proved that this improvement is not affected by the scaling of the graph or the change of elements in given clauses.

Over the last decades, various cost-based optimizers have been proposed to generate optimal plans for SQL queries. These optimizers are key to achieving good performance in database systems and can speed up query execution. Still, they may need enormous expert efforts and perform poorly on complicated queries. Learning-based optimizers have been shown to achieve high-quality plans by learning from past experiences. However, these solutions treat each query separately and neglect the semantic equivalence among different queries. Intuitively, a high-quality plan may be obtained for a complicated query by discovering a simple equivalent query. Motivated by this, in this paper, we present Celo, a novel constraint-enhanced learned optimizer to directly integrate the equivalent information of queries into the learning-based model. We apply denial constraints to identify equivalent queries by replacing equivalent predicates. Given a query, we augment the query plans generated by the learning-based model with the high-quality plans of its equivalent queries. Then, a more potentially well-performed plan will be predicted among the augmented query plans. Extensive experiments using real-world datasets demonstrated that Celo outperforms the previous state-of-the-art (SOTA) results even with few constraints.

Query optimization is one of the key factors affecting the performance of database systems that aim to enact the query execution plan with minimum cost. Particularly in distributed database systems, due to the multiple copies of the data that are stored in different data nodes, resulting in the dramatic increase in the feasible query execution plans for a query statement. Because of the increasing volume of stored data, the cluster size of distributed databases also increases, resulting in poor performance of current query optimization algorithms. In this case, a dynamic perturbation-based artificial bee colony algorithm is proposed to solve the query optimization problem in distributed database systems. The improved artificial bee colony algorithm improves the global search capability by combining the selection, crossover, and mutation operators of the genetic algorithm to overcome the problem of falling into the local optimal solution easily. At the same time, the dynamic perturbation factor is introduced so that the algorithm parameters can be dynamically varied along with the process of iteration as well as the convergence degree of the whole population to improve the convergence efficiency of the algorithm. Finally, comparative experiments conducted to assess the average execution cost of Top-k query plans generated by the algorithms and the convergence speed of algorithms under the conditions of query statements in six different dimension sets. The results demonstrate that the Top-k query plans generated by the proposed method have a lower execution cost and a faster convergence speed, which can effectively improve the query efficiency. However, this method requires more execution time. [ABSTRACT FROM AUTHOR]

Indexing is an important method to improve database performance. At present, with the development of reinforcement learning algorithm, there are a series of methods to solve the index recommendation problem（ISP） by reinforcement learning. Aiming at the problem that the existing deep reinforcement learning index recommendation algorithm has long training time and unstable training, this paper proposes an index recommendation algorithm based on A2C（Advantage Actor-Critical）, PRELIA（Parallel Compensation Learning Index Advisor）. In order to improve the accuracy and efficiency of index selection and reduce the occupation of index space, the algorithm adds the characteristic matrix of the number of rows scanned by load index and normalizes the reward value. Experimental results on different data sets show that the proposed algorithm can guarantee the index recommendation quality equivalent to that of the compared algorithm, while the recommended index occupies less storage space, and the training time is more than 4 times longer than that of the baseline algorithm. [ABSTRACT FROM AUTHOR]

In order to reduce the resource occupancy and retrieval efficiency of geological drilling databases, this study proposes a distributed horizontal expansion method for query optimization of geological drilling databases by constructing a comprehensive geological data subtree, analyzing the characteristics of distributed databases and elements in geological databases, and quickly retrieving data resources based on element attributes. In addition, this study has designed a method to horizontally extend the database designed for drilling holes using a multi-constraint model in order to achieve extension optimization of the distributed geological drilling database. Experiments are conducted to verify the performance and applicability of the proposed method. The experiment shows that when the geological data capacity is 80 GB, the capacity level of the geological database can be extended to 41 × 105TB using the method proposed in this study. The retrieval efficiency is higher than 89% and the resource occupancy rate is lower than 12% after the horizontal expansion of the database. By using this research method, the horizontal expansion of the geological drilling database is more effective, and can effectively reduce the resource occupancy rate and retrieval efficiency of the geological drilling databases. This has value significance for geological drilling with efficiency improvement and development.

Complex event processing (CEP) systems have gained significant importance in various domains, such as finance, logistics, and security, where the real-time analysis of event streams is crucial. However, as the volume and complexity of event data continue to grow, optimizing the performance of CEP systems becomes a critical challenge. This paper investigates the impact of indexing strategies on the performance of databases handling complex event processing. We propose a novel indexing technique, called Hierarchical Temporal Indexing (HTI), specifically designed for the efficient processing of complex event queries. HTI leverages the temporal nature of event data and employs a multi-level indexing approach to optimize query execution. By combining temporal indexing with spatial- and attribute-based indexing, HTI aims to accelerate the retrieval and processing of relevant events, thereby improving overall query performance. In this study, we evaluate the effectiveness of HTI by implementing complex event queries on various CEP systems with different indexing strategies. We conduct a comprehensive performance analysis, measuring the query execution times and resource utilization (CPU, memory, etc.), and analyzing the execution plans and query optimization techniques employed by each system. Our experimental results demonstrate that the proposed HTI indexing strategy outperforms traditional indexing approaches, particularly for complex event queries involving temporal constraints and multi-dimensional event attributes. We provide insights into the strengths and weaknesses of each indexing strategy, identifying the factors that influence performance, such as data volume, query complexity, and event characteristics. Furthermore, we discuss the implications of our findings for the design and optimization of CEP systems, offering recommendations for indexing strategy selection based on the specific requirements and workload characteristics. Finally, we outline the potential limitations of our study and suggest future research directions in this domain.

In autonomous driving, the fusion of multiple sensors is considered essential to improve the accuracy and safety of 3D object detection. Currently, a fusion scheme combining low-cost cameras with highly robust radars can counteract the performance degradation caused by harsh environments. In this paper, we propose the IRBEVF-Q model, which mainly consists of BEV (Bird’s Eye View) fusion coding module and an object decoder module.The BEV fusion coding module solves the problem of unified representation of different modal information by fusing the image and radar features through 3D spatial reference points as a medium. The query in the object decoder, as a core component, plays an important role in detection. In this paper, Heat Map-Guided Query Initialization (HGQI) and Dynamic Position Encoding (DPE) are proposed in query construction to increase the a priori information of the query. The Auxiliary Noise Query (ANQ) then helps to stabilize the matching. The experimental results demonstrate that the proposed fusion model IRBEVF-Q achieves an NDS of 0.575 and a mAP of 0.476 on the nuScenes test set. Compared to recent state-of-the-art methods, our model shows significant advantages, thus indicating that our approach contributes to improving detection accuracy.

Multi-cultural integration is an inevitable path to enhance the inheritance and protection of the Central Plains culture and to innovate the teaching mode of vocal music in colleges and universities. In this paper, we constructed a university vocal music teaching database system based on distributed database, and used semi-connection algorithm to query and optimize the data related to the integration of Central Plains culture into vocal music teaching resources. In order to make up for the defects of the semi-connection algorithm when facing a large number of tuples, the semi-connection algorithm is improved by introducing the idea of symmetric homology to realize the massive data scheduling of the vocal music teaching resources database. For the college vocal teaching database constructed in this paper, the query optimization test and performance test are carried out, and the teaching practice application is carried out with the vocal teaching database. The results show that the query throughput rate of vocal music teaching resources reaches 3.18*103MB/s under Read-only load when the number of running rounds reaches 16 rounds, and when the number of concurrent users of the vocal music teaching database is 1000, the CPU occupancy rate and memory utilization rate are 1.37% and 1.06%, respectively. By integrating Zhongyuan culture into the vocal music teaching database, more than 70% of the students indicated that it could enhance their interest in learning about Zhongyuan culture. The use of technology to realize the integration of vocal music teaching and the culture of the Central Plains provides a new reference for the inheritance and protection of the culture of the Central Plains.

While the Web of Data in principle offers access to a wide range of interlinked data, the architecture of the Semantic Web today relies mostly on the data providers to maintain access to their data through SPARQL endpoints. Several studies, however, have shown that such endpoints often experience downtime, meaning that the data they maintain becomes inaccessible. While decentralized systems based on Peer-to-Peer (P2P) technology have previously shown to increase the availability of knowledge graphs, even when a large proportion of the nodes fail, processing queries in such a setup can be an expensive task since data necessary to answer a single query might be distributed over multiple nodes. In this paper, we therefore propose an approach to optimizing SPARQL queries over decentralized knowledge graphs, called Lothbrok. While there are potentially many aspects to consider when optimizing such queries, we focus on three aspects: cardinality estimation, locality awareness, and data fragmentation. We empirically show that Lothbrok is able to achieve significantly faster query processing performance compared to the state of the art when processing challenging queries as well as when the network is under high load. [ABSTRACT FROM AUTHOR]

Groupjoins combine execution of a join and a subsequent group-by. They are common in analytical queries and occur in about of the queries in TPC-H and TPC-DS. While they were originally invented to improve performance, efficient parallel execution of groupjoins can be limited by contention in many-core systems. Efficient implementations of groupjoins are highly desirable, as groupjoins are not only used to fuse group-by and join, but are also useful to efficiently execute nested aggregates. For these, the query optimizer needs to reason over the result of aggregation to optimally schedule it. Traditional systems quickly reach their limits of selectivity and cardinality estimations over computed columns and often treat group-by as an optimization barrier. In this paper, we present techniques to efficiently estimate, plan, and execute groupjoins and nested aggregates. We propose four novel techniques, aggregate estimates to predict the result distributions of aggregates, parallel groupjoin execution for scalable execution of groupjoins, index groupjoins, and a greedy eager aggregation optimization technique that introduces nested preaggregations to significantly improve execution plans. The resulting system has improved estimates, better execution plans, and a contention-free evaluation of groupjoins, which speeds up TPC-H and TPC-DS queries significantly. [ABSTRACT FROM AUTHOR]

Multistores are data management systems that enable query processing across different and heterogeneous databases; besides the distribution of data, complexity factors like schema heterogeneity and data replication must be resolved through integration and data fusion activities. Our multistore solution relies on a dataspace to provide the user with an integrated view of the available data and enables the formulation and execution of GPSJ queries. In this paper, we propose a technique to optimize the execution of GPSJ queries by formulating and evaluating different execution plans on the multistore. In particular, we outline different strategies to carry out joins and data fusion by relying on different schema representations; then, a self-learning black-box cost model is used to estimate execution times and select the most efficient plan. The experiments assess the effectiveness of the cost model in choosing the best execution plan for the given queries and exploit multiple multistore benchmarks to investigate the factors that influence the performance of different plans. [ABSTRACT FROM AUTHOR]

Cardinality estimation is critical for database management systems (DBMSs) to execute query optimization tasks, which can guide the query optimizer in choosing the best execution plan. However, traditional cardinality estimation methods cannot provide accurate estimates because they cannot accurately capture the correlation between multiple tables. Several recent studies have revealed that learning-based cardinality estimation methods can address the shortcomings of traditional methods and provide more accurate estimates. However, the learning-based cardinality estimation methods still have large errors when an SQL query involves multiple tables or is very complex. To address this problem, we propose a sampling-based tree long short-term memory (TreeLSTM) neural network to model queries. The proposed model addresses the weakness of traditional methods when no sampled tuples match the predicates and considers the join relationship between multiple tables and the conjunction and disjunction operations between predicates. We construct subexpressions as trees using operator types between predicates and improve the performance and accuracy of cardinality estimation by capturing the join-crossing correlations between tables and the order dependencies between predicates. In addition, we construct a new loss function to overcome the drawback that Q-error cannot distinguish between large and small cardinalities. Extensive experimental results from real-world datasets show that our proposed model improves the estimation quality and outperforms traditional cardinality estimation methods and the other compared deep learning methods in three evaluation metrics: Q-error, MAE, and SMAPE. [ABSTRACT FROM AUTHOR]

The rise of diversified data engines has created the need for federated queries. A federated query can take a query and provide data analysis based on data from various data engines. Since the query data originates from multiple data engines, federated queries usually rely on join operation and data migration to complete the query and take a long time. The challenges of optimizing federated queries lie on join order selection and data migration coordination. However, enumerating all join orders is impractical because the set of join orders grows exponentially with the number of relations to be joined. To improve the performance of federated queries, we present a deep reinforcement learning-based approach on optimizing join order and join engine selection for federated queries and design an deep Q-network-based (DQN-based) optimizer. The DQN-based optimizer can generate join search policies that optimize the join order selection for datasets with a given cost model. Based on the DQN-based optimizer, we implement a federated query system Coral which can provide optimization for join order selection of federated queries. With the optimized join order, Coral can transform a federated query into a set of subqueries which will be assigned to and executed on different data engines. We also propose a subquery cache optimization to optimize data migration during the query execution. The extensive experimental evaluation demonstrates that Coral can significantly reduce the query latency of federated queries and achieve a speedup of up to 5.03 × compared to the cutting-edge federated query systems. [ABSTRACT FROM AUTHOR]

Directed Acyclic Graph (DAG) workflows are widely used for large-scale data analytics in cluster-based distributed computing systems. The performance model for a DAG on data-parallel frameworks (e.g., MapReduce) is a research challenge because the allocation of preemptable system resources among parallel jobs may dynamically vary during execution. This resource allocation variation during execution makes it difficult to accurately estimate the execution time. In this paper, we tackle this challenge by proposing a new cost model, called Bottleneck Oriented Estimation (BOE), to estimate the allocation of preemptable resources by identifying the bottleneck to accurately predict task execution time. For a DAG workflow, we propose a state-based approach to iteratively use the resource allocation property among stages to estimate the overall execution plan. Furthermore, to handle the skewness of various jobs, we refine the model with the order statistics theory to improve estimation accuracy. Extensive experiments were performed to validate these cost models with HiBench and TPC-H workloads. The BOE model outperforms the state-of-the-art models by a factor of five for task execution time estimation. For the refined skew-aware model, the average prediction error is under 3 % when estimating the execution time of 51 hybrid analytics (HiBench) and query (TPC-H) DAG workflows. [ABSTRACT FROM AUTHOR]

Copyright of Journal of Data Acquisition & Processing / Shu Ju Cai Ji Yu Chu Li is the property of Editorial Department of Journal of Nanjing University of Aeronautics & Astronautics and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Improving data processing in big data is a delicate procedure in our current digital era due to the massive amounts of data created by humans and machines in daily life. Handling this data, creating a repository for storage, and retrieving photos from internet platforms is a difficult issue for businesses and industries. Currently, clusters have been constructed for many types of data, such as text, documents, audio, and video files, but the extraction time and accuracy during data processing remain stressful. Hadoop Distributed File System (HDFS) is a system that provides a large storage area in big data for managing large datasets, although the accuracy level is not as high as desired. Furthermore, query optimization was used to produce low latency and high throughput outcomes. To address these concerns, this study proposes a novel technique for query optimization termed the Enhanced Salp Swarm Algorithm (ESSA) in conjunction with the Modified K-Means Algorithm (MKM) for cluster construction. The process is separated into two stages: data collection and organization, followed by data extraction from the repository. Finally, numerous experiments with assessments were carried out, and the outcomes were compared. This strategy provides a more efficient method for enhancing data processing speed in a big data environment while maintaining an accuracy level of 98% while processing large amounts of data. [ABSTRACT FROM AUTHOR]

Copyright of Journal of Computer Engineering & Applications is the property of Beijing Journal of Computer Engineering & Applications Journal Co Ltd. and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Query, expressing the user need and requirement, has an important role, in an information retrieval system, for reaching a high accuracy search. In this paper, we present an overview of the different refinement operations that the query may undergo, in the sake to enhance performance of an information retrieval system, such as: automatic query formulation through words prevision, query reformulation, query expansion, and query optimization. [ABSTRACT FROM AUTHOR]

In this study, we propose three k-nearest neighbor (k-NN) optimization techniques for a distributed, in-memory-based, high-dimensional indexing method to speed up content-based image retrieval. The proposed techniques perform distributed, in-memory, high-dimensional indexing-based k-NN query optimization: a density-based optimization technique that performs k-NN optimization using data distribution; a cost-based optimization technique using query processing cost statistics; and a learning-based optimization technique using a deep learning model, based on query logs. The proposed techniques were implemented on Spark, which supports a master/slave model for large-scale distributed processing. We showed the superiority and validity of the proposed techniques through various performance evaluations, based on high-dimensional data. [ABSTRACT FROM AUTHOR]