Search

Hive can store massive data through extended clusters, far exceeding the expansion and storage capabilities of traditional databases, and has become a major tool for building data warehouses in the era of big data. Based on Hive technology, this paper studies the enterprise-level data warehouse architecture composed of data storage layer, Hive data warehouse layer and application layer. Then, the paper studies data warehouse tools, uses HDFS for underlying storage and MapReduce for computing engine, and compares Hive data warehouse with relational database. Then, the ETL process is studied. The data goes through three stages: extraction, conversion and loading, and finally the data flows from the source to the target end. Finally, the system function and test are studied, the subsystems including data processing, data management, data migration and data analysis are constructed, and the core functions are tested. The test results were consistent with the expected results and met the delivery standards. [ABSTRACT FROM AUTHOR]

Over the past two decades, the way computing resources are been developed, deployed, upgraded, and applied changed dramatically, with more and more software and hardware solutions being transferred to cloud technologies. Data Warehouses (DW), defined as a way of organizing corporate data in an integrated manner over (sequential) time periods, "structured & disposed" in order to generate a "single data source", were also affected by the evolution, thus giving rise to the concept of Cloud Data Warehouse (CDW). This technology allows users to be more technologically free, as they do not need to spend time investing in software and hardware, they only pay for the resources they used and the infrastructure itself has greater flexibility and scalability. However, selecting the most suitable platform or technology for a CDW can be a complex task due to the large number of factors that can influence the decision and due to the existing offer in the market. The objective of this paper is to describe the process of benchmarking a set of CDW platforms, with the goal of analyzing and exposing each one's performance results. These platforms are Snowflake, Google BigQuery, Amazon Redshift, and Azure Synapse. The metrics to be measured are data loading and query running time, and alias running times. For this benchmark, the dataset used was Star Schema Benchmark (SSB), a dataset based on the well-known TPC Benchmark™ H (TPC-H). [ABSTRACT FROM AUTHOR]

Healthcare facilities use huge quantities of real-time and analytical data to discover meaningful information from patient clinical lab results. Advanced analytics and machine learning algorithms help doctors identify and treat patients more accurately. Accurate models must be trained, tested, and validated with enough data. New real-time data allows healthcare practitioners to quickly and accurately analyse patient demands. Healthcare organizations can improve patient care and outcomes through knowledge discovery. The goal of this effort is to develop a real-time data repository based on patient clinical exams. This collection feeds real-time monitoring panels and machine or deep learning algorithms that forecast patient progression from clinical lab results. Integrate HL7 messages from diverse sources, preprocess them, and add them to an API-accessible data warehouse. In conclusion, the proposed method creates an international-standard data warehouse. This data warehouse can increase healthcare decision-making accuracy and efficacy when utilised with machine learning models, improving patient care and outcomes through more personalised treatment options. [ABSTRACT FROM AUTHOR]

The digital age is responsible for the continuous growth of data volume. Everything related to Information and Communication Technologies contributes to the exponential growth of data, such as social networks, financial technology, and scientific data. Data Mining is a method of extracting data from systems such as Business Intelligence, Big Data, Data Warehouse, so that the patterns extracted during the process are considered information, knowledge, in relation to the theoretical concepts that support them. In this sense, these patterns should be presented as representations of knowledge. In the test case presented, we observed the various phases of the knowledge discovery process and analysed data using regression and classification models, to support companies in the knowledge management. [ABSTRACT FROM AUTHOR]

New technologies and the use of the Internet of Things (IoT) play a remarkable role in the design, construction, and maintenance of Smart Cities. With the exponential increase of data generated in the areas of health, commerce, financial services, science, education, public administration, and society, the challenge of ensuring data security and quality is increasing. IoT and Smart Cities, are in permanent mutation and evolution, towards a new paradigm of digitalization and digital transformation. The existing literature review exposes much of the attention that is given to IoT, but a broad discussion on the characteristics, use, applications, and challenges of these technologies in Smart Cities is still needed. The main contribution of this paper is that it provides a systematic assessment and synthesis of the critical applications and challenges of IoT, in the development of Smart Cities. [ABSTRACT FROM AUTHOR]

The concept of data lake was proposed more than a decade ago. Although progress has been made in data lake research and applications, there are also numerous issues and challenges need to be addressed. In this paper, we survey some recent developments, provide our observations, as well as our expectations on future research and practice in this area. We start with a discussion on general terminology in data lakes, then review various aspects of data lakes, including an examination on metadata and related issues, information granularities involved in data lakes, unique features such as mixed lazy and eager approach in data lake lifecycle, relationship between data lakes and data mining, interplay between data lakes and data warehousing techniques, as well as some notable developments related to applications of data lakes. We present our observations while we examine these aspects, and wrap up this paper by expressing our hope of developing a unified framework for data lake lifecycle evaluation. [ABSTRACT FROM AUTHOR]

The Portuguese Foundation for Science and Technology (FCT) has endorsed Plan-S since early 2021 and it is working towards implementing its obligations. To ensure an effective fulfillment of its open science policies, FCT is implementing a new monitoring and compliance tool, that will enforce the compliance with Plan-S as well as the new FCT's open access (OA) policy. This paper details FCT's approach to implement this tool and discusses the challenges of integrating information that is spread across multiple systems. The new tool will be built on top of infrastructures and services developed by FCT over the last years under the umbrella of the Portuguese Current Research Information System (PTCRIS) program. This tool comprises an operational component, integrated with the grant management system, and an analytical component, based on a Data Warehouse and Business Intelligence (DW/BI) system. The DW/BI system for the ecosystem of Science and Technology in Portugal will be developed using Kimball's Dimensional DW architecture, which advocates the incremental design and development of dimensional data marts linked by conforming dimensions. In this approach, data marts contain atomic and summarized data, corresponding to the development of elementary and derived dimensional models. Three data marts will be developed for the subject areas of funding, scientific results and for monitoring and compliance. The last data mart will calculate the compliance indicators for Plan-S as well as for FCT's OA policy. The operational component of the new tool then receives, from the analytical component, the list of scientific results per funding registry along with data regarding compliance. A list of non-compliances for each scientific result will be the basis of a notification module that will inform the principal investigator (PI) and the FCT compliance officer. [ABSTRACT FROM AUTHOR]

Consolidation of the research information improves the quality of data integration, reducing duplicates between systems and enabling the required flexibility and scalability when processing various data sources. We assume that the combination of a data lake as a data repository and a data wrangling process should allow low-quality or "bad" data to be identified and eliminated, leaving only high-quality data, referred to as "research information" in the Research Information System (RIS) domain, allowing for the most accurate insights gained on their basis. This, however, would lead to increased value of both the data themselves and data-driven actions contributing to more accurate and aware decision-making. This cleansed research information is then entered into the appropriate target Current Research Information System (CRIS) so that it can be used for further data processing steps. In order to minimize the effort for the analysis, the proliferation and enrichment of large amounts of data and metadata, as well as to achieve far-reaching added value in information retrieval for CRIS employees, developers and end users, this paper outlines the concept of a curated data lake with the data wrangling process, showing how it can be used in CRIS to clean up data from heterogeneous data sources during their collection and integration. [ABSTRACT FROM AUTHOR]

Complementary Diagnostic Requests (CDRs) are required for disease identification, monitoring, and prognosis. Diagnostic tests misuse, on the other hand, can lead to negative health outcomes as well as additional costs. Inappropriate diagnostic test requests are primarily the result of a lack of interoperability between Healthcare Information Systems (HIS). On one hand, clinicians can be mislead into which test is the best option for each clinical case, on the other hand missing previous results, leads to duplication or unnecessary tests. HIS is increasingly relying on standards based on dual architecture to promote interoperability as well as the structuring and consistency of clinical and demographic data. The OpenEHR standard's duo-based architecture allows for concise modelling of archetypes and templates for a given clinical case, which was used in this study. As a result, the purpose of this research was to build an openEHR template for the CDR registration as well as the architecture of a Data Warehouse (DW) system capable of storing all of the information needed for the diagnostic test request process. Afterwards, Business Intelligence (BI) indicators was developed in order to answers the needs for test registration and execution. [ABSTRACT FROM AUTHOR]

"Three fit" problems in the Bid Process Information System (BPIS) are still an open issue. The BPIS must be: integrated, flexible and interoperable in order to deal with the urbanization approach. Hence, operational, organizational, decision-making, and cooperative dimensions are four dimensions that have been identified to deal with such failures. Therefore, in this work, we are particularly interested in handling this failure by proposing an approach that focuses on the decision dimension of BPIS. This improves at the end of the various stages of the data warehouse lifecycle and takes into consideration the user's knowledge by focusing on their individual needs too. [ABSTRACT FROM AUTHOR]

The practice of evidence-based medicine has been gaining ground in the most diverse health institutions and consequently in the work of their health professionals. Thus, the Clinical Decision Support Systems appear, which aim to ensure that health professionals are supported with the best evidence in order to improve the clinical decision-making process, reducing the occurrence of adverse events through the availability and sharing of quality information by health professionals. Business Intelligence (BI) and Business Analytics (BA) technologies are increasingly being used to extract knowledge and turn it into useful information and analytical tools for healthcare professionals. This article is part of a dissertation that has as main objective the extraction of knowledge from the hospital data of the obstetrics service in Centro Materno-Infantil do Norte (CMIN) from Centro Hospitalar Universitário do Porto (CHUP) with focus on the area of clinical decision. [ABSTRACT FROM AUTHOR]

The diagnosis and treatment of breast cancer are very much studied in the field of medical research. Indeed, the early detection of this pathology allows for the best possible treatment and minimization of the after-effects. The business process "Diagnosis of breast cancer in women" is perfectly mastered by the practitioner and the medical profession and is widely documented in the literature. We propose a vision of "analysis of the existing" which allows to discover this process to the computer engineer in charge of the implementation of a system for aid in diagnosis and prognosis of the disease. This step is indispensable to discover the data generated during the business process and to discover in a later step all the characteristics that govern the disease. We define a system of diagnosis and prognosis of the disease Diaprog which uses the data of the classified and archived care records. We say that we use the historical data generated during the main process. We distinguish for this purpose between a production database and a data warehouse. The latter is a database fed from the various production databases accessible and made available in our system. In the case of our work, we call this data warehouse DDA (Diagnosis Data Archive). The data warehouse thus defined is a component of the medical diagnosis aid system Diaprog whose functional architecture also implements a learning module that allows the discovery of relationships between the characteristics that govern the disease. [ABSTRACT FROM AUTHOR]

Even though several systems for Air Quality (AQ) monitoring have been in existence for over a decade, a research model for Knowledge Management (KM) of AQ data has to be created in order to enhance the decision-making and organize the air quality data collected from the Internet of Things (IoT) consumer devices. This model should be made more performant by ensuring greater flexibility and interoperability between devices and emerging technologies. In this context, we propose an approach for representing Data WareHouse (DWH) schema based on an ontology that captures the multidimensional knowledge of tools, techniques, and technologies used for novel AQ systems. This enhances decision-making by coping with potential problems such as data sources heterogeneity and covering the various phases of the decision-making life cycle. [ABSTRACT FROM AUTHOR]

"Three fit" problems in the Bid Process Information System (BPIS) are still an open issue. The BPIS must be: integrated, flexible and interoperable in order to deal with the urbanization approach. Hence, operational, organizational, decision-making, and coop-erative dimensions are four dimensions that have been identified to deal with such failures. Accordingly, we propose an approach that focuses on the decision-making dimension of the BPIS. In addition, it represents data warehouse schema based on an ontology in order to capture the multidimensional bid-knowledge. This improves the resulting of data warehouse schema. [ABSTRACT FROM AUTHOR]

In the age of big data, entreprises' information systems are ingested with data generated from social media which raises the need to integrate it in their business intelligence process for better decision making. However, these new data, streaming, voluminous, unstructured and variant, bring existing data warehousing systems and integration tools to their knees which motivated us to conduct this research work. In this paper, we propose a large scale system based on distributed storage and parallel processing to succeed social media data warehousing. In fact, we combine Storm and Hadoop for structured events extraction from social media data and their integration in the data warehouse. We take the advantage of real time analysis of streaming data offered by Storm and batch processing of large volumes of data of Hadoop which facilitated streaming social media data analysis task. For conceptual representation, we propose a customized multidimensional model in which we add an intermediate table to connect the social media data warehouse with the enterprise data warehouse. We implement it using Oracle 12c and we fed it with events extracted from 1000 000 tweets using Pentaho data integration tool. [ABSTRACT FROM AUTHOR]

Data tipically evolve according to specific processes, with the consequent possibility to identify a profile of evolution: the values it may assume, the frequencies at which it changes, the temporal variation in relation to other data, or other constraints that are directly connected to the reference domain. A violation of these conditions could be the signal of different menaces that threat the system, as well as: attempts of a tampering or a cyber attack, a failure in the system operation, a bug in the applications which manage the life cycle of data. To detect such violations is not straightforward as processes could be unknown or hard to extract. In this paper we propose an approach to detect data anomalies. We represent data user behaviours in terms of labelled transition systems and through the model checking techniques we demonstrate the proposed modeling can be exploited to successfully detect data anomalies. [ABSTRACT FROM AUTHOR]

In order to enhance the business process, the Bid Process Information System (BPIS) should be made more performant by ensuring greater flexibility and interoperability between devices. Moreover, the specication of this system has to deal with "three fit" problems. To this end, four dimensions have been identified in order to cope with potential failures: operational, organizational, decision-making, and cooperative dimensions. In this paper, we focus on the decision-making dimension of the BPIS and propose an approach for representing data warehouse schema based on an ontology that captures the multidimensional bid-knowledge. [ABSTRACT FROM AUTHOR]

Traditional data warehouses are unable to meet the growing needs of the modern enterprise to integrate and analyze a wide variety of data generated by social, mobile and sensor sources. What is remarkable is that many companies have changed their data storage in NoSQL databases. Given the important role played by this type of databases, it is necessary to study NoSQL databases as a data source for the modeling and the implementation of data warehouses. This paper proposes a method to design the data warehouse schema from schema free databases known as NoSQL databases. Our proposal starts with the extraction of schemes from document-oriented databases as an example of NoSQL database. This extraction is performed based on the MapReduce paradigm. Then, it defines the structure identification graph from the extracted schemes. This graph will help the designer to identify the multidimensional concepts for each schema extracted in order to design the global data warehouse schema. [ABSTRACT FROM AUTHOR]

The problem with telecommunications companies today is that transactional data is more extensive than existing source tables. This makes business reporting less efficient and overwhelms query processing results in data warehouses so that they do not meet business requirements. The fast and complex evolution of the digital world must be scalable to the data warehouse process, so that the authors implement it in the data warehouse using massive parallel processing (MPP) with the Greenplum database, so that business users can get reports faster and more optimally. This case study explains how the MPP system implements and measures the performance of the Greenplum database by performing complex queries in the data warehouse with parallel processing. Therefore, this case study analyzes whether the use of MPP systems can measure the scalability of throughput and the response time in the data warehouse so that system performance in the Greenplum database remains stable for daily, weekly, and monthly operations. [ABSTRACT FROM AUTHOR]

Nowadays, data warehouse tools and technologies cannot handle the load and analytic process of data into meaningful information for top management. Big data technology should be implemented to extend the existing data warehouse solutions. Universities already collect vast amounts of data so the academic data of university has been growing significantly and become a big academic data. These datasets are rich and growing. University’s top-level management needs tools to produce information from the records. The generated information is expected to support the decision-making process of top-level management. This paper explores how big data technology could be implemented with data warehouse to support decision making process. In this framework, we propose Hadoop as big data analytic tools to be implemented for data ingestion/staging. The paper concludes by outlining future directions relating to the development and implementation of an institutional project on Big Data. [ABSTRACT FROM AUTHOR]

In the healthcare industry it is imperative the need to increase the efficiency of resource management and services. The increasing of Business Intelligence (BI) use in organizations and the demonstrated effectiveness of this type of solution, arises the desire to use BI in healthcare as in Misericórdias . So, in this work some concepts associated to the use of BI in Misericórdias were addressed and a BI architecture was designed. Furthermore, a survey was made in order to understand what are the tools used by Misericórdias every day and which ones have the BI components. Finally, a BI architecture was developed based on the organization's mission and their stakeholders. Through this work it was possible to identify the critical processes and designing the Entity Relationship Diagram as well as a set of indicators to meet the needs of a sustainable decision-making in Portuguese Misericórdias . [ABSTRACT FROM AUTHOR]

Combining powerful parallel frameworks and on-demand commodity hardware, cloud computing has made both analytics and decision support systems canonical to enterprises of all sizes. Associated with unprecedented volumes of data stacked by such companies, filtering and retrieving them are pressing challenges. This data is often organized in star schemas, in which Star Joins are ubiquitous and expensive operations. In particular, excessive disk spill and network communication are tight bottlenecks for all current MapReduce or Spark solutions. Here, we propose two efficient solutions that drop the computation time by at least 60%: the Spark Bloom-Filtered Cascade Join (SBFCJ) and the Spark Broadcast Join (SBJ). Conversely, a direct Spark implementation of a sequence of joins renders poor performance, showcasing the importance of further filtering for minimal disk spill and network communication. Finally, while SBJ is twice faster when memory per executor is large enough, SBFCJ is remarkably resilient to low memory scenarios. Both algorithms pose very competitive solutions to Star Joins in the cloud. [ABSTRACT FROM AUTHOR]

Storage of pre-computed views in data warehouse can essentially reduce query processing cost for decision support queries. The problem is to choose an optimal set of materialized views. Various frameworks such as lattice, MVPP and AND-OR graphs and algorithms like heuristic based, greedy, stochastic algorithm have been proposed in the literature for materialized view selection. Heuristic and greedy algorithms become slower in high dimensional search space while stochastic algorithms do not guarantee global optimal solution but reach to the optimum most solution in a fast and efficient way. In this paper we have implemented Particle Swarm Optimization (PSO) algorithm, one of the stochastic algorithm, on lattice framework to select an optimal set of views for materialization in data warehouse by minimizing query processing cost. We have compared our results with Genetic algorithm to prove the effectiveness of PSO algorithm over genetic algorithm. [ABSTRACT FROM AUTHOR]

A data warehouse is an information system used for data reporting and analysis. It results from the collection, pretreatment and storage of massive integrated data from disparate and heterogeneous sources. Data warehouses contain historical information and constitute a decision support system. When the number of dimensions constituting the data warehouse is too important, it may be difficult for the analyst to explore the OLAP cubes looking for knowledge and well-hidden patterns. It is then useful to provide him with convenient tools to identify and ignore superfluous dimensions that can be deleted without loosing important information. This solution allows creating new parsimonious OLAP cubes instead of the sparse original ones. In this context, we propose in this paper an approach based on Genetic Algorithms (GAs) associated with the Multiple Correspondence Analysis (MCA) method. Our approach aims to identify a subset of n superfluous dimensions from the initial set of p dimensions where n

This study aims to understand the assessment of basic education in the perspective of the State Reviewer as a mechanism that generates information regarding the positivity and weaknesses of a school or an educational system to provide improvements. For this reason, a Data Warehouse was created and later some analysis of the indicators were performed through clustering. [ABSTRACT FROM AUTHOR]

Data warehousing is the crucial part of business intelligence applications. The data warehouse physical design is a hard task due to a large number of possible choices involved. The bitmap join indexes selection problem is crucial in the data warehouse physical design. All proposed approaches to solve the bitmap join indexes selection problem are based on statistics such as data mining or meta-heuristics such as genetic algorithm and particle swarm optimization. In the present work, we propose a new approach based on mixed-integer linear programming for solving the bitmap join indexes selection problem. Several experiments are performed to demonstrate the effectiveness of the proposed approach and the results are compared to the well known approaches that are best so far: the data mining, the genetic algorithm and particle swarm optimization based approaches. The mixed-integer linear programming is found to be faster and more effective than the genetic algorithm, particle swarm optimization and data mining approaches for solving the bitmap join indexes selection problem. [ABSTRACT FROM AUTHOR]

This paper presents an approach of operational control of the state of technosphere and environment objects based on the integration of Data Warehouse, OLAP and Expert systems technologies. It looks at the structure of data warehouse that provides consolidation and storage of monitoring data. There is a description of OLAP-models that provide a multidimensional analysis of monitoring data and dynamic analysis of principal parameters of controlled objects. The authors suggest some criteria of emergency risk assessment using expert knowledge about danger levels. It is demonstrated now some of the proposed solutions could be adopted in territorial decision making support systems. On-line control allows authorities to detect threat, prevent emergencies and ensure a comprehensive safety. [ABSTRACT FROM AUTHOR]

Abstract: This work introduces Ophidia, a big data analytics research effort aiming at supporting the access, analysis and mining of scientific (n-dimensional array based) data. The Ophidia platform extends, in terms of both primitives and data types, current relational database system implementations (in particular MySQL) to enable efficient data analysis tasks on scientific array-based data. To enable big data analytics it exploits well-known scientific numerical libraries, a distributed and hierarchical storage model and a parallel software framework based on the Message Passing Interface to run from single tasks to more complex dataflows. The current version of the Ophidia platform is being tested on NetCDF data produced by CMCC climate scientists in the context of the international Coupled Model Intercomparison Project Phase 5 (CMIP5). [Copyright &y& Elsevier]

Abstract: Currently Hive uses the size of tables on disk to determine if a common-join should be converted into a map-join. Our experiments demonstrate that this is a conservative decision criteria which will fail to identify optimization opportunities close to the decision frontier. Our implementation differs from the current implementation in the way we identify optimization candidates. By pre-computing hashtable sizes and adjusting them with per-query selectivity factors, we are able to choose optimization candidates directly as a function of expected hashtable size, factoring out the conservative file size criteria. In this paper we show that this approach results in more common-join to map-join optimizations and can provide average speedups of up to 1.30 in certain scenarios. [Copyright &y& Elsevier]

Abstract: Extraction, Transformation and Loading (ETL) represent a big portion of a data warehouse project. Complexity of components extensibility is a main problem in the ETL area, because ETL components are tightly-coupled to each others in the current ETL framework. The missing extensibility feature causes impediments to add new components to the current ETL framework; to meet special business needs. This paper shows how to restructure the current ETL framework based on Service Oriented Architecture (SOA) to be easier to extend. This restructuring solution distributes the ETL into interoperable components. The distribution of Extraction, Transformation and Loading components while keeping interoperability amongst them; can be achieved by SOA. A Classified-Fragmentation component to enhance the report generation speed is added to the new framework; as a proof of the extensibility concept. The result of this work is an extensible ETL framework including Classified-Fragmentation component as an extension. [Copyright &y& Elsevier]