Search

Big data and geographic information systems (GIS) are two technologies that have increasingly influenced many areas in the last 10 years and will continue to improve and help solve serious global problems, such as consequences of climate change or global pandemics. A wide spectrum of GIS applications interacts with the continuous growth of geospatial big data sources to drive precise and informed decisions. Geospatial big data integration is designed to accomplish the compatibility of distinct geospatial datasets regardless of their spatial coverage. The large number of geospatial big data sources demand effective data integration for storing and handling such datasets, which will be used for geospatial data analysis and visualization. For instance, risk management datasets related to healthcare and the environment are heterogeneous and disparate. Obtaining a unified view of such geospatial big datasets is complicated and challenging, especially if we consider problems related to healthcare pandemics and environmental disasters. Hence, before we can attempt to predict and mitigate processes occurring in these domains, we must realize that geospatial big data integration is crucial in consolidating datasets. We explore and discuss issues involved in integrating geospatial big datasets in this study. We then classify big data integration processes into three categories, namely, data warehousing, data transformation and integration methods. Furthermore, several research challenges focused on geospatial big data, big earth data, data warehousing, data transformation and linked data are presented. Lastly, open research issues and emerging trends that require in-depth investigations in the near future are highlighted in this study.

Contribution: Recently, real-time data warehousing (DWH) and big data streaming have become ubiquitous due to the fact that a number of business organizations are gearing up to gain competitive advantage. The capability of organizing big data in efficient manner to reach a business decision empowers data warehousing in terms of real-time stream processing. A systematic literature review for real-time stream processing systems is presented in this paper which rigorously look at the recent developments and challenges of real-time stream processing systems and can serve as a guide for the implementation of real-time stream processing framework for all shapes of data streams. Background: Published surveys and reviews either cover papers focusing on stream analysis in applications other than real-time DWH or focusing on extraction, transformation, loading (ETL) challenges for traditional DWH. This systematic review attempts to answer four specific research questions. Research Questions: 1)Which are the relevant publication channels for real-time stream processing research? 2) Which challenges have been faced during implementation of real-time stream processing? 3) Which approaches/tools have been reported to address challenges introduced at ETL stage while processing real-time stream for real-time DWH? 4) What evidence have been reported while addressing different challenges for processing real-time stream? Methodology: A systematic literature was conducted to compile studies related to publication channels targeting real-time stream processing/joins challenges and developments. Following a formal protocol, semi-automatic and manual searches were performed for work from 2011 to 2020 excluding research in traditional data warehousing. Of 679,547 papers selected for data extraction, 74 were retained after quality assessment. Findings: This systematic literature highlights implementation challenges along with developed approaches for real-time DWH and big data stream processing systems and provides their comparisons. This study found that there exists various algorithms for implementing real-time join processing at ETL stage for structured data whereas less work for un-structured data is found in this subject matter.

A Discussion Forum is one component of the learning media in a Learning Management System. There is much information and knowledge that are discussed among a lecturer and students in forums as a part of the learning process. The motivation of this study is to evaluate the learning process in a discussion forum through a dashboard, namely, the Dashboard of Knowledge (DoK). The DoK is formed using an Information Retrieval Architecture and Entity Relationship Diagram that is designed using four category interfaces. Furthermore, the effectiveness of the learning evaluation through the DoK is measured by a survey. The survey is conducted using three out of four dimensions of multi-criteria of the Web-based E-Learning System (WELS), namely, the Learner Interface, the System Context, and Personalization. The respondents of the survey are the heads of department that are DoK users. The results show that most respondents strongly agree that the DoK is useful for evaluating the learning in a discussion forum. This is exhibited by the survey results that show an average score of 3.53 out of 4.00 and 84.32% satisfaction. These results reveal that the DoK can effectively assist heads of department in monitoring the effectiveness of the learning process in a discussion forum.

Data Lakes, as a modern concept of raw analytical data storage, were presented as a next step that will take over Data Warehouses. In the course of time after the Data Lakes emergence, it is clear that those two architectures complement each other with their actual usage. A new architecture concept, Data Lakehouse, has recently been introduced as a next step in the analytical data storage. The Data Lakehouse architecture should be able to deal with currently insurmountable challenges that both Data Warehouses and Data Lakes cannot overcome. In this article, a current state of the Data Lakehouse architecture and vendor implementation will be given. Furthermore, a comparison of the Data Lakehouse will be done between both Data Warehouses and Data Lakes with a focus on tackling their known challenges. Finally, an approach to combine existing Data Warehouses and Data Lakes into a unified Data Lakehouse will be presented, with a special focus on the high- level architecture.

Data mining technology can accurately extract useful information from a large amount of information data, and also has the function of removing falsehood and preserving truth, so it has a very important application in intelligent system set. The essential condition of intelligent building is building intelligence, and the core of intelligent building system design is system integration. The main goal of intelligent building system integration is to realize information integration, and data warehouse technology is an effective solution to solve the problem of intelligent building information integration. Data mining technology is used to analyze and process various kinds of massive data and information, so as to realize the conversion from data to information and provide effective support for leaders' decision-making. The new model of building intelligent system integration based on data mining technology proposed in this paper has solved the compatibility and openness problems in the process of building intelligent system integration, and achieved the goal of safe and efficient building intelligent system.

The successful implementation of various computerized information systems has led to high data growth. Automation works well, but some of the desired reports are not achieved. This is because the data scattered in various systems have not been integrated. This paper discusses the design of data warehouse which is the result of integration and data abstraction from the Academic Information System (AIS) and other systems at UIN Syarif Hidayatullah Jakarta campus that accommodates the need for data history and archiving to support reports in the academic field. These reports are used to fulfill the Higher education accreditation carried out by BAN-PT (Higher Education National Accreditation Agency) to assess the quality of university institutions and study programs in Indonesia. Therefore, the identification of information needs using the reference format of executive reports derived from the accreditation forms of study programs. The methodology in this study includes the project planning phase, defining needs, design, construction, deployment, and maintenance. In these stages, it starts from planning the data subject, designing the star schema, designing the data warehouse and then implementing it by means of Extract, Transform, Load (ETL) to form the cube using Pentaho Data Integration (PDI). The results of this study are academic fact tables that will be combined with fact tables from other standard forms as the basis for making an Academic Business Intelligent Application in University.

A data warehouse aids in the management of large amounts of data that may be stored in order to handle user input during the computer process. The major issue with a data warehouse is to maintain the data that the user stores in good quality. Some traditional techniques can improve data quality while also increasing efficiency. Each unit of data has a unique feature that has been researched by many researchers and has an influence on data quality. This research article has enhanced the K-Means method by utilizing the Euclidean Distance metric to detect missing values from the gathered sources and replace them with closest values while maintaining the data's consistency, exactness, and quality. yThe improved data will assist developers in analysing data quality prior to data integration by allowing them to make informed decisions quickly in accordance with business requirements. Improved K-Means achieves better accuracy and requires less computational time for clustering data objects when compared to other related approaches.

There are many methods to measure the equilibrium exchange rate of RMB, among which the traditional purchasing power parity method is very influential, but the conclusions are quite different. Data mining is a potentially powerful new technology, which is a process of discovering hidden patterns and relationships in large-scale data. It can help enterprises find the most important information in their data warehouse. In this paper, the data mining method based on SEM is used to discuss the dynamic change law of RMB real exchange rate and its influencing factors from the way of qualitative and quantitative integration. It is found that the influence path of RMB exchange rate is characterized by lag or immediate linkage, and the transmission path varies from country to country. There are heteroscedasticity, persistence and asymmetry shocks in RMB exchange rate fluctuations, in which negative shocks will increase volatility while positive shocks have a slight impact on volatility.

With the rapid development of big data, user data information is increasingly perfect, data warehouse integration is more reasonable, and data quality is constantly improving, so the value of data is increasing. Based on parallel processing of K-means clustering algorithm, this paper extracts ability constraint information, integrates K-means clustering algorithm, clusters and integrates various index parameters of post competency. From the final experimental results, this method improves the execution efficiency and accuracy compared with other methods, and can be used in practice.

Intellectual property service agencies mainly provide services such as intellectual property agency or management for enterprises or individuals. There are a large number of intellectual property service agencies in the Beibu Gulf City Cluster, but the research and analysis on them are still relatively few. Based on the Beibu Gulf City Cluster, this paper analyzes the information of the provinces, cities, establishment time, staff size, registered capital and other information of the intellectual property rights service agencies. The first step of big data analysis is to pre-process and store the data of intellectual property service agencies to the data warehouse, and then conduct visual analysis on them through R language. Through visual analysis, enterprises or individuals can have a relatively comprehensive grasp of information related to intellectual property service agencies, and then choose appropriate agencies to provide intellectual property services for them.

A large amount of data information is generated in the informatization construction of the application and installation in power business expanding of the power system. The traditional data analysis method of the application and installation in power business expanding only establishes a single analysis model for the data, and does not clarify the deep relationship of the data, which leads to the ineffective use of the archival data. For this reason, the mining analysis of the application and installation in power business expanding based on big data technology is proposed. Based on the establishment of the data warehouse of the application and installation in power business expanding, the data of the application and installation in power business expanding are processed by using the combined prediction model. After improving k-means clustering by genetic algorithm, data mining was performed to obtain the relationship between the archive data. The experimental results show that the studied analysis method not only has high data processing efficiency, but also can effectively shorten the application and installation in power business expanding process and improve the economic efficiency of enterprises when applied to actual power operation.

Modern start-up, and growing organizations often experience increasing growth and expansion of their client base, resulting in exponential growth in data. Business intelligence (BI) is used by organizations as an analytical method to extract production data, analyze it and present business stakeholders with meaningful insight into operations with regard to decision making. This paper describes how a small, start-up organization developed business intelligence to support its growing business endeavor after also buying out divisions of another company. A descripto-explanatory case study of a single phenomenon was performed to learn from this scenario, with the purpose to inform growing organizations of how to migrate data into their data warehouse. Furthermore, this study aimed to provide means of inserting large amounts of data into an organization's business intelligence architecture with minimal impact on business operations.

Waste management has always been a challenge to the development of urban cities. The existing waste management system in India involves human intervention at each stage. This causes serious health issues to sanitation workers. In 2017, 12 sanitation workers in Delhi lost their life due to improper waste management practices. Poor waste management practices also affect people living around the dustbin or dump yards. They may also get exposed to toxic wastes leading to cancer. In this paper, we propose a concept of IoT enabled zero touch waste management platform for smart cities. The platform consists of a smart dustbin, autonomous waste collection truck, smart dump yard and waste management center. This paper also presents an architecture for the proposed smart waste management platform. A routing algorithm for the autonomous waste collection is also discussed. The algorithm aims to provide a shortest route covering all garbage bins. The proposed platform aims to address the United Nations Sustainable Development Goal 3, 6 and 11.

In order to overcome the problems of low precision and poor stability of financial risk data mining in traditional enterprise financial risk control system, this paper proposes a new enterprise financial risk intelligent control system based on artificial intelligence algorithm. Combined with B/S three-tier structure and MVC architecture, the overall structure of the system is designed. According to the overall structure of the system, the system hardware is designed to meet the operation and use needs of the system. The system hardware module mainly includes user registration module, financial risk degree judgment module and data warehouse module. Through the coordination of each module, the overall operation performance of the system can be improved. In the software part of the system, the artificial intelligence algorithm is used to fully mine the enterprise financial risk data, and the financial risk management and control model is constructed to realize the intelligent management and control of financial risk. The experimental results show that, compared with the traditional risk management and control system, the designed system has higher accuracy of financial risk data mining, and can improve the stability of the system.

Big Data filed is an unsettled standard comparing with a traditional database, data mining, or data warehouse. Stability measure aims to acquire the quality dataset which encourages to use of preprocessing data method to handle instability that miniaturization missing data. Therefore, to increase the data quality in order to achieve an accurate prediction, significant rules are used to provide value and meaningful data. Through, three measures by support, confidence, and the lift to acquire frequently rules. These rules are used to conduct the objective extracting pattern, to estimate each browsing customer's likelihood of making a purchase, and to choose meaningful patterns from the discovered association rules.

The application of the SCRUM method can be most effectively mastered in a practice-oriented manner with students proving the solution of problems such as communication skills, motivation or even attitudinal deficiencies in teamwork.Based on this, we decided to try to use the SCRUM project method in the laboratory work of the course "Data Warehouses and Business Intelligence" with the lecturers and students of the John von Neumann Faculty of the University of Obuda. We introduced the project method as an educational methodology tool during the last semester and in the present study, we would like to report the results of the experimental education.

As the data increase keeps on getting more extensive due to technology evolvement from the rational database, online transaction, cloud computing, data warehouse to big data analytics. This changes influences organizations to advance from data mining support to machine learning-enabled software platform. Seemingly, the study summarised secondary data from non-grey and grey academic literature as the research field recently started getting attention. Consequently, the work identifies, analyzes, and synthesizes the challenges of ML-enabled software development, which differs from traditional software development. But, with the adoption of the SE technique to engineer ML-enabled software development, the study was able to identify advancement for ML-enabled software likes automation of mismatch detection, which occurs due to the nature of different perspectives of stakeholders involved. Another one is integrating ML and SE data end-to-end pipeline to allow Systematic test mechanism and test automation where necessary when ML is complex in format to enable standard SE test logs. Then, education, training, and cooperation between the stakeholders, especially SE and ML, to gain more experience, knowledge, put rifts aside to join hands, and work together to ascertain user requirements. Finally, the work reframed the traditional SE development process to engineer the ML software development process. Therefore, the study can benefit stakeholders in the ML and SE communities in handling ML development challenges and may benefits academicians in conduction future research on software engineering for artificial intelligence.

The ETL (Extract, Transform and Load) process is part of the Intelligent Business (BI) tools. It is used in organizations, public and private, to compile and process data, as well as perform extraction, cleaning, optimization and unification of various sources and loads of data in a Data Warehouse. BI makes it possible to achieve more accurate results, using a multidimensional database, oriented by subject, non-volatile, historical, decision-making and variable in relation to time. The ETL process proved to be an effective tool in the management of retired employees of a Federal Educational Institution, since it enabled the development of a base of indicators, through the BI, quantitative, qualitative and geographic tools of these employees. With the unification of the various databases, it was possible to identify the spatial location of the servers, allowing faster contact, their concentration by age group, the level and intensity of the links between the parties, as well as the evolution of health-related issues, evolution of retirement, mainly in legal matters. Thus, this article describes the ETL process designed to extract information about retired employees from different data sources and the creation of a Data Warehouse capable of providing individual and group information about these retired employees.

The Data Warehouse context is in transformation in private and public organizations. New proposals for managing large volumes of data have been defined in the literature, such as the storage and analysis of data by non-relational DBMS, or NoSQL (Not-only SQL) Databases, under a Big Data scenario. Thus, Data Warehouses, which originally relied on relational databases, have been using NoSQL databases to analyze this massive data in recent years. Besides, there is the rise of hybrid transactional and analytical processing, in real-time, known as HTAP, which provides simplicity and quick support to decision making. In this context, this article has the objective to accomplish a literature mapping of DW based on NoSQL databases and advances in HTAP. A methodology for a systematic literature search protocol was used to answer the research questions. We found 72 papers in the last five years, which we analyzed in this paper.

Business Intelligence (BI) is a term that gained popularity in recent years, with other concepts coming along, such as Data Warehouse, Data Mart, Data Deluge, Data Quality, and ETL (Extract, Transform, Load). The increase of the competitiveness and wildness among organizations in the current market and business intensifies the need of those organizations to have and hold key information about their issues. This gathering and analysis will keep the stakeholders always informed about their products. This paper fills the gap of a formal Business Intelligence with principles of Design Thinking, Rational Unified Process, using ISO 25012 to evaluate and categorize the BI indicators, also the functional and non-functional requirements.

The development of future technologies for the National Airspace System (NAS) will be reliant on a new communications infrastructure capable of managing the limited available spectrum for communications among aircraft and ground systems. Emerging approaches to autonomous allocation of aviation spectrum mostly rely on machine learning techniques, where 4D (longitude, latitude, altitude, time) trajectory prediction is an important data input to enable real-time resource allocation. This study explores and evaluates effective data sources and deep recurrent neural network techniques when determining flight trajectories. Specifically, data are collected and evaluated in a 14-day period. Sources of data include NASA Sherlock Data Warehouse, MIT Lincoln Labs Corridor Integrated Weather Service (CIWS), and assorted NOAA weather datasets. Deep learning models for 4D predictions all utilize a hybrid-recurrent technique. A baseline model is considered via the convolutional-recurrent design from the existing literature. The modified design considers Gated Recurrent Units (GRU), Independently Recurrent Neural Networks (IndRNN), and stand-alone self-attention layers. Results indicate the effectiveness of Long-Short Term Memory (LSTM) and GRU cells for state-of-the-art data processing (interpolation). Attention mechanisms provide notable performance improvements to convolutional layers and may extend dimensional capabilities of a learning model. Finally, NOAA measurements provide only a supplemental value by improving predictions along a flight altitude; however, tailored products such as CIWS Echo Top remain the most holistically accurate.

The article discusses the application of data cleaning algorithms focused on data collection for data stores, and their adaptation for the task of data reengineering. A classification of reengineering problems depending on the number of databases is given, various levels of the algorithm's operation are analyzed, and a description of its main stages is given. The selection and modification of the cleaning algorithm for database reengineering is performed, its time complexity and convergence are assessed, and the prospects for its use are assessed. An example of the operation of the algorithm is given.

With the diversification of applications and the diversity of users, the era of big data is always producing various new types of data, such as graph data, time series data, and spatial data. When predicting the future trend of the system, we always need to learn from the historical working state, and time series data plays a huge value in this aspect. Hardware resources are becoming cheaper, computing power has made major breakthroughs, and multi-machine and multi-core scenarios have become commonplace. At present, the focus of large-scale time series data processing is either focused on data collection and storage, or on single-machine processing, which has been unable to meet the existing needs. In response to this problem, this paper designs and implements a distributed large-scale time series processing and analysis system based on the Spark platform. The system framework is mainly divided into storage layer, operator layer and algorithm layer. At the storage layer, the system completes the organization and indexing of large-scale time series data based on HDFS and Hive. At the operator layer, the system provides users with basic operations commonly used for time series data on the Spark platform, and allows users to directly use these operators to implement custom time series related processing algorithms. At the algorithm level, the system implements some commonly used time series analysis algorithms in the Spark platform, including time series similarity query, clustering and prediction, so that users can directly use these algorithms to analyze time series. Finally, the feasibility and practicability of this system are verified by testing the performance and function of the system.

Unraveling human mobility patterns is critical for understanding disease spread and implementing effective controls during large-scale disease outbreaks such as the COVID-19 pandemic. Given the urgency associated with such situations, it is important to leverage on the common existing digital infrastructures that can be readily activated for disease outbreak analytics. We introduce an integrated system for disease outbreak investigation using data from Wi-Fi sessions. The system offers outbreak analytics, simulation, and visualization capabilities to assist in the identification of infection hot-spots and in contacttracing exercises. The system has been developed and experimentally deployed for research purposes on a large local university campus in Singapore.

Exploratory analysis is an important way to gain understanding and find unknown relationships from various data sources, especially in the era of big data. Traditional paradigms of social science data analysis follow the steps of feature selection, modeling, and prediction. In this paper, we propose a new paradigm that does not require feature selection so that data can speak for itself without manually picking out features. Besides, we propose using the deep network as a methodology to explore previously unknown relationships and capture complexity and non-linearity between target variables and a large number of input features for big social data. The new paradigm tends to be a relatively generic approach that can be widely used in different scenarios. In order to validate the feasibility of the paradigm, we use country-level indicators forecasting as a case study. The process includes: 1) data collection and preparation and 2) modeling and experiment. The data collection and preparation part builds a data warehouse and conducts the extract-transform-load process to eliminate data format inconsistency. The modeling and experiment part includes model setup and model structures change to achieve relatively high accuracy on prediction results at both model level and case level. We find some patterns about network capacity modification and the influence of time interval difference on the test results, whereas both of them deserve further research.

In recent years, the radical advancement of technologies has given rise to an abundance of software applications, social media, and smart devices such as smartphone, sensors, and so on. More extensive use of these applications and tools in various industrial domains has led to data deluge, which has fostered enormous challenges and opportunities. However, it is not only the volume of the data but also the speed, variety, and uncertainty, which are promoting a massive challenge for traditional technologies such as data warehouse. These diverse and unprecedented characteristics have engendered the notion of “Big Data.” The data-intensive industries have been experiencing a wide variety of challenges in terms of processing, managing, and analysis of data. For instance, the healthcare sector is confronting difficulties in respect of integration or fusion of diverse medical data stemming from multiple heterogeneous sources. Data integration is critically important within the healthcare sector because it enriches data, enhances its value, and more importantly paves a solid foundation for highly efficient and effective healthcare analytics such as predicting diseases or an outbreak. Several data integration technologies and tools have been developed over the last two decades. This paper aims at studying data integration technologies, tools, and applications within the healthcare domain. Furthermore, this paper discusses future research directions in the integration of Big healthcare data.

This study proposes an issue-oriented data integration and system-building model to promote institutional research. In direction of the issues, we advance the idea of grouping university stakeholders into the university, students, and the public. In the aspect of data integration and system construction, five steps are proposed: issue confirmation, information system and data inventory, data model establishment, data warehouse and data dictionary establishment, and system construction and data analysis. Finally, this study proposes several preliminary conclusions and suggestions on the promotion of institutional research for universities or researchers.

Address matching is an important task for many businesses especially delivery and take out companies which help them to take out a certain address from their data warehouse. Existing solution uses similarity of strings, and edit distance algorithms to find out the similar addresses from the address database, but these algorithms could not work effectively with redundant, unstructured, or incomplete address data. This paper discuss semantic Address matching technique, by which we can find out a particular address from a list of possible addresses. We have also reviewed existing practices and their shortcoming. Semantic address matching is an essentially NLP task in the field of deep learning. Through this technique We have the ability to triumph the drawbacks of existing methods like redundant or abbreviated data problems. The solution uses the OCR on invoices to extract the address and create the data pool of addresses. Then this data is fed to the algorithm BM-25 for scoring the best matching entries. Then to observe the best result, this will pass through BERT for giving the best possible result from the similar queries. Our investigation exhibits that our methodology enormously improves both accuracy and review of cutting-edge technology existing techniques.

Data warehousing and analytics are going through a metamorphosis. Corporations are transforming their data warehousing capabilities, centralizing data analytics, and leveraging cloud technologies. Eight years ago, at ASMC 2013, we presented our approach to big data solutions and asked the industry leaders to help unify technologies to enable our big data transformations more seamlessly for the industry. This paper will cover what does, what did not, and what will not in the future work, where data and analytics are heading, and what the semiconductor industry needs to do to address big data challenges. The material will present clear reasons for leveraging big data solutions including up to 20× performance improvements analysis, cloud to cloud sharing options, ways to easily scale and handle data growth and centralizing information. There will also be a few areas where it will be shown how better integration from the factory floor to the modern data analysis platforms could be possible but cannot be fully addressed without the help from vendors.

Under the background of the continuous expansion of the scale and capacity of power grid and the sharp increase of information measurement points, the amount of data that needs to be analyzed and processed is growing geometrically. It is urgent to carry out the research of massive information intelligent collection and efficient computing processing scheme and storage scheme. In this paper, combined with the actual demand of monitoring data access and lossless compression method in engineering practice, we build Hadoop cloud cluster platform and hive distributed data warehouse, take analog measurement data set as cluster compression object, realize the effective compression and storage of measurement information, which has important theoretical significance and practical value for solving power grid large data processing.

Currently, supermarket run-centres, Big Marts keep track of each individual item's sales data in order to anticipate potential consumer demand and update inventory management. Anomalies and general trends are often discovered by mining the data warehouse's data store. For retailers like Big Mart, the resulting data can be used to forecast future sales volume using various machine learning techniques like big mart. A predictive model was developed using Xgboost, Linear regression, Polynomial regression, and Ridge regression techniques for forecasting the sales of a business such as Big-Mart, and it was discovered that the model outperforms existing models.

Data warehouse maintains historical information, which is used for taking important decisions in an enterprise. Hence, it becomes necessary to maintain information quality of data warehouse. Many researchers suggested various metrics for the evaluation of data model quality for maintaining information quality of data warehouse. Data models are at different level of abstraction namely; requirements, conceptual, logical and physical model. Quite a few investigations were witnessed in the past for evaluating quality of requirements model in formal and empirical manner. In this study, formal investigation of requirements traceability metrics of requirements model for data warehouse is carried out by employing Briand's property-based framework to ensure whether they are valid and are defined in the approved manner which in-turn build the quality based data warehouse. Results showed that out of seven traceability metrics, five metrics are found as cohesion measure and rest two metrics are found as coupling measure; which concluded that these metrics are valid and defined correctly. Thus, these requirements traceability metrics can be used successfully to assess the quality of requirements model of data warehouse.

A core work of the science and technology management system is to support the integration and utilization of massive data from distributed systems using data warehouse technology. In this paper, we focus on this work. First, we introduce the background of science and technology management by illustrating the scheme of project management business flows. Then, to define the science and technology data structure, we propose a customized star model in which we add various dimensional tables to connect the different attributes with science and technology projects. Particularly, a 4-D data cube is introduced to satisfy complicated query conditions in science and technology domain. Furthermore, to integrate data from distributed systems, we propose an architecture of data warehouse of science and technology projects (STPDW)—a five layered system that gathers raw data from the science and technology management systems, and processes the data in a standard workflow by using Extract Transform and Load (ETL) tools with predefined rules. Finally, a prototype system has also been designed to achieve the functionality of the STPDW.

This article discusses the features of distributed computing in systems based on MapReduce, analyzes join strategies in Apache Spark, gives basic recommendations for optimizing join operation, and proposes an algorithm of the join operation for distributed data warehouses, which increases the processing speed. Experimental data represents that the method becomes more effective when the volume of initial data increases. So, for 2 TB data, the join operation was performed ~ 37% faster, for 7 TB data already by ~ 47%.

Based on the functional requirements of the oil and gas exploration and development data warehouse and the characteristics of the multidimensional analysis language, this paper establishes a universal and reliable data cube model, and tries to design a set of easy-to-implement algebraic logic to support OLAP operations on the cube. This is a data warehouse and OLAP formal model centered on the multi-dimensional analysis function requirements of oil and gas exploration. Like all data and computing models, the purpose of this article is to make this model a basic service to effectively design and serve multidimensional query languages.

In any organization, the leadership is responsible for taking decisions that will lift the said organization to a better place. The problem-solving abilities of the management are mostly depending on the ability to grasp all the required information in a clean and actionable format. Building a well-designed data warehouse leads to answer that problem. When data sourced from different sourcing systems, it's very important that the aggregated data is relevant and supporting the decision-making by the leadership. This study aims at mitigating the issues that are hindering such organizations to make correct decisions.

In order to meet the development of OCN business, it is key to establish a perfect business support system. The designer of OCN enterprise data warehouse platform is the core part of the whole support system. This paper studies the framework mode and integration structure of OCN customer data integration platform, analyzes the functions of each layer in the logic architecture of the data platform, and analyzes the function of each layer in the data warehouse architecture. This paper analyzes the data integration process in detail, and describes the data acquisition method, data extraction strategy and data loading in detail.

Fashion industry and market represents a paramount domain touching people all over the world; however, it lacks transactional and analytical tools allowing ordinary people and fashion actors to get either real time information about their fashion life trajectory and/or analytical ones about the fashion market and industry evolving. Inspired from the Snow-White fairy tale, the objective of this work is to design a fashion pervasive system related to multiple participating persons providing their multimedia data captured through smart mirrors. Captured data, related essentially to fashion images and video augmented by users' annotation, is then organized and stored locally for users' transactional querying and secured and centralized into a data warehouse and/or data mart for enhanced analysis and mining. Transactional querying permits to ordinary users to get information about their clothing habits and history. Analysis and mining allow fashion enterprises to get insights and predictions about the fashion market and industry evolution.

The Gaza Governorate Electricity Distribution Company (GEDCO) is an economic service company operating with a special financial liability and owned by the Palestinian National Authority and local authorities.The unpaid bills are one of the company's biggest operating obstacles that obstructs money processing. The ever-increasing number of unpaid invoices has caused constant problems with the company's cash flow.In this paper, Data mining techniques and neural networks techniques were implemented over the data warehouses of GEDCO. Decision tree machine learning algorithm is used to classify the customers according to their commitment to payment, as well as to extract the categories for each class of customers. The neural networks are used to train the machine to extract a threshold for consumption and amperes. The threshold extracted for each subscriber according to his history of consumption and payment, to help him stay committed to payment.The results concluded that consumption should not exceed some value, based on the address of the subscribers, the number of amperes and the history of payment.The study that will be recommended to GEDCO, should improve their data warehouses to include more features and categories of their databases especially the subscribers’ occupations and level of income.

Cloud data migration concept deals with data warehouses used for data analysis and data validation working with the flat file from the multiple sources A record is a text file that contains data. Comma-separated fields delimits the record in the column. Flat files DB does not accommodate various record reports. ETL is a process in which a data repository can be dealing with extracts information from a variety of sources and stores it in an archive. we are happy to reveal about ETL method used for this data validation using the Talend tool.