Search

With the rapid development of AI and big data mining technologies, computerized medical decision-making has become increasingly prominent. The aim of high-utility pattern mining (HUPM) is to discover meaningful patterns in medical databases that contribute to maximizing the utility from the perspective of diagnosis. However, HUPM pays less attention to the interpretability and explainability of these patterns in medical decision-making scenarios. This paper proposes a novel algorithm called the Improved fuzzy high-utility pattern mining (IF-HUPM) to address this problem. First, the paper applies a fuzzy preprocessing method to divide the fuzzy intervals of a medical quantitative data set, which enhances the fuzziness and interpretability of the data. Next, in the process of IF-HUPM, both fuzzy tree and list structures are employed to calculate fuzzy high-utility values. By combining the characteristics of the one-stage and two-stage algorithms of HUPM, an adaptive-phase Fuzzy HUPM hybrid frame is proposed. The experimental results demonstrate that the proposed IF-HUPM algorithm enhances both accuracy and efficiency and the mining process requires less time and space on average. [ABSTRACT FROM AUTHOR]

The large amount of data generated every day makes necessary the re-implementation of new methods capable of handle with massive data efficiently. This is the case of Association Rules, an unsupervised data mining tool capable of extracting information in the form of IF-THEN patterns. Although several methods have been proposed for the extraction of frequent itemsets (previous phase before mining association rules) in very large databases, the high computational cost and lack of memory remains a major problem to be solved when processing large data. Therefore, the aim of this paper is three fold: (1) to review existent algorithms for frequent itemset and association rule mining, (2)to develop new efficient frequent itemset Big Data algorithms using distributive computation, as well as a new association rule mining algorithm in Spark, and (3) to compare the proposed algorithms with the existent proposals varying the number of transactions and the number of items. To this purpose, we have used the Spark platform which has been demonstrated to outperform existing distributive algorithmic implementations. [ABSTRACT FROM AUTHOR]

Data mining has reached a state that is difficult to break through, while the scale of data is growing rapidly, due to the lack of traditional computing power and limited data storage space. Efficient and accurate extraction of valuable information from massive data has become a challenge. Researchers have combined quantum computing with data mining to address this problem, hence the concept of quantum data mining has emerged. The fundamental tenets of quantum physics are adhered to for information transmission and computing operations in quantum data mining, which use the states of minuscule particles to represent and process information. Quantum data mining are based on the characteristics of quantum computing, such as superposition and entanglement, which make the ability of computational and information extraction effectively improved. The paper discusses and summarizes the relevant literature on quantum data mining in recent 3 years. After introducing relevant basic concepts of quantum computing, quantum data mining is presented in five aspects: quantum data classification, quantum data clustering, quantum dimensionality reduction, quantum association rules, quantum linear regression, and quantum causal analysis. These approaches, based on quantum computing, offer new perspectives and tools for handling complex data mining tasks. In conclusion, the development of quantum data mining is promising and crucial to overcome the difficulties associated with large-scale data mining. [ABSTRACT FROM AUTHOR]

Data mining algorithms can process target data and extract useful hidden information, which is helpful for decision making. However, current mining algorithms have some shortcomings such as time-consuming processing of big data or inability to process massive data. Since data mining technology cannot be used in the traditional cloud platform environment, it is necessary to improve the algorithm to make it more suitable for the cloud platform environment. By analyzing the actual application process of BP classification algorithm, this paper expounds the practicability of BP classification algorithm, analyzes the data mining process based on Hadoop cloud platform, and expounds the development ideas of BP classification algorithm. The source of data mining algorithm supported by cloud computing is discussed. Finally, Hadoop cloud platform. This paper designed the corresponding system architecture and data interface, and established a suitable test environment for this system, and completed the simulation experiment test through the design. The calculation time of this algorithm is proportional to the amount of data, showing a linear relationship. In data mining, the optimized BP algorithm in this paper can significantly save resources in terms of spatial features. This paper designs an optimized operating system based on Hadoop platform through comprehensive analysis and improvement algorithm. [ABSTRACT FROM AUTHOR]

In order to explore potential value of explosively growing data in thermal power unit, this paper proposes a big data mining method based on Hadoop-based Spark cluster-computing framework and algorithms. Firstly, positive and negative balance methods are used to accurately obtain actual net coal consumption, and maximum information coefficient method is used to select all parameters related to optimization objectives. Then, Spark-based Mini-Batch K-means algorithm and Elbow method are constructed to divide whole operating modes. After that, all data are discretized and mapped to corresponding intervals by using Spark-based Elbow method and Mini-Batch K-means algorithm. Finally, Spark-based parallel FP-growth algorithm is used to deeply mine the potential relationships and laws. To verify the proposed method, a 350-MW thermal power unit is taken as a study case. The important conclusions are as follows: (1) the proposed Spark-based Mini-Batch K-means algorithm reduces the calculation time by 57.11% compared with Mini-Batch K-means algorithm, and 85.61% calculation time compared with K-means algorithm. The proposed Spark-based FP-growth algorithm reduces computational time by 32.8% compared with FP-growth algorithm. (2) Strong association rules of whole operating modes are mined, and operating optimization guidance schemes for important parameters are obtained. Take operating mode 1 as an example: if the optimal result can be reasonably applied, it can save 2.942 g coal per kilowatt hour. (3) Besides, we have found out some other potential relationships among parameters, which have important reference value for on-site operators to analyze economy of the thermal power unit. [ABSTRACT FROM AUTHOR]

In order to overcome the problems of the traditional power big data mining methods, such as the low integrity of data mining and the long time-consuming of data mining, this paper realizes multi-dimensional power big data mining by improving the grey clustering algorithm. Firstly, a relay multi hop network is established to collect power big data through the collector; Secondly, Lagrange interpolation method is used to fill the missing data of power data mining; Standardized processing of power consumption data; Finally, according to the grey theory and FCM clustering algorithm, the multi-dimensional power big data mining is realized. The experimental results show that the integrity of power big data mining in this method is up to 0.996, the mining time is not more than 3.05 s, and the mining integrity is up to 0.992, which indicates that this method can effectively improve the effect of power big data mining. [ABSTRACT FROM AUTHOR]

Outlier detection is an important branch of data mining. This paper proposes an advanced fast density peak outlier detection algorithm based on the characteristics of big data. The algorithm is an outlier detection method based on the improved density peak clustering algorithm. This paper improves the original algorithm. From the perspective of outlier detection, although it is a clustering idea, it avoids the clustering process, reduces the time complexity of the cluster-based outlier detection algorithm, and absorbs. The outlier detection based on neighbors is not sensitive to data dimensions and other advantages. In the power industry, outlier detection can be used in areas such as grid fault detection, equipment fault detection, and power abnormality detection. The simulation experiment of outlier detection based on the daily load curve of single and multiple transformers in a certain province shows that the improved algorithm can effectively detect outliers in the data. [ABSTRACT FROM AUTHOR]

In the big data era, traditional data mining technology cannot meet the requirements of massive data processing with the background of intelligent manufacturing. Aiming at insufficient computing power and low efficiency in mining process, this paper proposes a improved K-means clustering algorithm based on the concept of distributed clustering in cloud computing environment. The improved algorithm (T.K-means) is combined with MapReduce computing framework of Hadoop platform to realize parallel computing, so as to perform processing tasks of massive data. In order to verify the practical performance of T.K-means algorithm, taking machining data of milling Ti-6Al-4V alloy as the mining object. The mapping relationship among cutting parameters, surface roughness, and material removal rate is mined, and the optimized value for cutting parameters is obtained. The results show that T.K-means algorithm can be used to mine the optimal cutting parameters, so that the best surface roughness can be obtained in milling Ti-6Al-4V titanium alloy. [ABSTRACT FROM AUTHOR]

A crucial step in improving data quality is to discover semantic relationships between data. Functional dependencies are rules that describe semantic relationships between data in relational databases and have been applied to improve data quality recently. However, traditional functional discovery algorithms applied to distributed data may lead to errors and the inability to scale to large-scale data. To solve the above problems, we propose a novel distributed functional dependency discovery algorithm based on Apache Spark, which can effectively discover functional dependencies in large-scale data. The basic idea is to use data redistribution to discover functional dependencies in parallel on multiple nodes. In this algorithm, we take a sampling approach to quickly remove invalid functional dependencies and propose a greedy-based task assignment strategy to balance the load. In addition, the prefix tree is used to store intermediate computation results during the validation process to avoid repeated computation of equivalence classes. Experimental results on real and synthetic datasets show that the proposed algorithm in this paper is more efficient than existing methods while ensuring accuracy. [ABSTRACT FROM AUTHOR]

Clustering is a traditional data mining problem that has attracted researchers from different disciplines because its solution can be applied to many useful problems in our daily life. Since the era of big data is coming, how to "reduce the computing time" of an "effective clustering algorithm" has been a promising research issue in recent years. Thus, this paper presents an effective clustering algorithm, by using the so-called searched information to determine later search directions, and then has it implemented on Spark to accelerate its response time for analyzing large-scale datasets. Simulation results show that the proposed algorithm provides a better result than the other clustering algorithms compared in this paper because it is less sensitive to the initial solutions. The simulation results further show that cloud computing platform is capable of enhancing the performance of the proposed algorithm. • This study presents a high-performance clustering algorithm based on searched experiences. • The results show that it is able to find a better result than traditional clustering algorithms. • We apply it to Spark to show its possibility to accelerate the response time on a cloud system. [ABSTRACT FROM AUTHOR]

One of the main goals of Big Data research, is to find new data mining methods that are able to process large amounts of data in acceptable times. In Big Data classification, as in traditional classification, class imbalance is a common problem that must be addressed, in the case of Big Data also looking for a solution that can be applied in an acceptable execution time. In this paper we present Approx-SMOTE, a parallel implementation of the SMOTE algorithm for the Apache Spark framework. The key difference with the original SMOTE, besides parallelism, is that it uses an approximated version of k -Nearest Neighbor which makes it highly scalable. Although an implementation of SMOTE for Big Data already exists (SMOTE-BD), it uses an exact Nearest Neighbor search, which does not make it entirely scalable. Approx-SMOTE on the other hand is able to achieve up to 30 times faster run times without sacrificing the improved classification performance offered by the original SMOTE. [ABSTRACT FROM AUTHOR]

• A new method based on Locality-Sensitive Hashing (LSH) for cluster initialization in the k-means like clustering was proposed. • A new clustering algorithm called LSH-k-representatives for handling big categorical data was developed. • Extensive experiments conducted on multiple real-world and synthetic datasets have demonstrated the effectiveness of the proposed clustering method. • The newly developed algorithm achieves comparable or better clustering performance to the existing well-known k-means like algorithms, while it is significantly more efficient by a factor of between 2× and 32×. Clustering categorical data remains a challenging problem in the era of big data, due to the difficulty in measuring dis/similarity meaningfully for categorical data and the high computational complexity of existing clustering algorithms that makes it difficult to be applied in practical use for big data mining applications. In this paper, we propose an integrated approach that incorporates the Locality-Sensitive Hashing (LSH) technique into the k -means-like clustering so as to make it capable of predicting the better initial clusters for boosting clustering effectiveness. To this end, we first utilize a data-driven dissimilarity measure for categorical data to construct a family of binary hash functions that are then used to generate the initial clusters. We also propose to use a nearest neighbor search at each iteration for cluster reassignment of data objects to improve the clustering complexity. These solutions are incorporated into the k -representatives algorithm resulting in the so-called LSH- k -representatives algorithm. Extensive experiments conducted on multiple real-world and synthetic datasets have demonstrated the effectiveness of the proposed method. It is shown that the newly developed algorithm yields comparable or better clustering results in comparison to the existing closely related works, yet it is significantly more efficient by a factor of between 2 × and 32 ×. [ABSTRACT FROM AUTHOR]

With the development of medical informatization, the data related to medical field are growing at an amazing speed, and medical big data appears. The mining and analysis of these data plays an important role in the prediction, monitoring, diagnosis, and treatment of tumor diseases. Therefore, this paper proposes a clustering algorithm of the high-order simulated annealing neural network algorithm and uses this algorithm to extract tumor disease-related big data, constructs training set according to the relevant information mined, designs a kind of dimension reduction model, aiming at the problem of excessive and wrong diagnosis and treatment in the diagnosis and treatment module of tumor disease monitoring mode, and establishes the corresponding control mechanism, so as to optimize the tumor disease monitoring mode. The results show that the clustering accuracy of the high-order simulated annealing neural network algorithm on different data sets (iris, wine, and Pima India diabetes) is 97.33%, 82.11%, and 70.56% and the execution time is 0.75 s, 0.562 s, and 1.092 s, which are better than those of the fast k-medoids algorithm and improved k-medoids clustering algorithm. To sum up, the high-order simulated annealing neural network algorithm can achieve good clustering effect in medical big data mining. The establishment of model M1 can reduce the probability of excessive and wrong medical treatment and improve the effectiveness of diagnosis and treatment module monitoring in tumor disease monitoring mode. [ABSTRACT FROM AUTHOR]

The shuffling algorithm based on folding technique has ideal data scrambling effect and can meet the prerequisite of big data sampling. In order to prove that the internal rules of the scrambled data set are not destroyed after sampling, this paper analyzed the association rules of the data before and after sampling by data mining, and compared the support and confidence of the association rules as well as the frequency of the transactions, and found that the association rules of the data after the collapsed shuffle algorithm were stable before and after sampling. And by comparing with the time efficiency of the existing algorithm and the overall sampling error, it is further concluded that big data sampling is effective. Which means that the overall situation of the data can be inferred from the sampled sample. [ABSTRACT FROM AUTHOR]

In recent years, the application of the gradient boosting-back propagation (GB-BP) neural network algorithm in many industries has brought huge benefits, so how to combine the GB-BP neural network algorithm with sports has become a research hotspot. Based on this, this paper studies the application of the GB-BP neural network algorithm in wrestling, designs the sports athletes action recognition and classification model based on the GB-BP neural network algorithm, first analyzes the research status of wrestling action recognition, and then optimizes and improves the shortcomings of action recognition and big data analysis technology. The GB-BP neural network algorithm can realize the accurate recognition and classification of wrestlers' training actions and carry out big data mining analysis with known action recognition, so as to achieve accurate classification. The experimental results show that the model can play a good role in wrestling and effectively improve the efficiency of wrestlers in training. [ABSTRACT FROM AUTHOR]

The ability to exploit students' sentiments using different machine learning techniques is considered an important strategy for planning and manoeuvring in a collaborative educational environment. The advancement of machine learning technology is energised by the healthy growth of big data technologies. This helps the applications based on Sentiment Mining (SM) using big data to become a common platform for data mining activities. However, very little has been studied on the sentiment application using a huge amount of available educational data. Therefore, this paper has made an attempt to mine the academic data using different efficient machine learning algorithms. The contribution of this paper is two-fold: (i) studying the sentiment polarity (positive, negative and neutral) from students' data using machine learning techniques, and (ii) modelling and predicting students' emotions (Amused, Anxiety, Bored, Confused, Enthused, Excited, Frustrated, etc.) using the big data frameworks. The developed SM techniques using big data frameworks can be scaled and made adaptable for source variation, velocity and veracity to maximise value mining for the benefit of students, faculties and other stakeholders. [ABSTRACT FROM AUTHOR]

In this paper, we propose a method for predicting the distributions of people's trajectories on the road network throughout a city. Specifically, we predict the number of people who will move from one area to another, their probable trajectories, and the corresponding likelihoods of those trajectories in the near future, such as within an hour. With this prediction, we will identify the hot road segments where potential traffic jams might occur and reveal the formation of those traffic jams. Accurate predictions of human trajectories at a city level in real time is challenging due to the uncertainty of people's spatial and temporal mobility patterns, the complexity of a city level's road network, and the scale of the data. To address these challenges, this paper proposes a method which includes several major components: (1) a model for predicting movements between neighboring areas, which combines both latent and explicit features that may influence the movements; (2) different methods to estimate corresponding flow trajectory distributions in the road network; (3) a MapReduce-based distributed algorithm to simulate large-scale trajectory distributions under real-time constraints. We conducted two case studies with taxi data collected from Beijing and New York City and systematically evaluated our method. [ABSTRACT FROM AUTHOR]

In the big data age, the traditional parallel collaborative filtering algorithm cannot meet the needs of data analysis in the efficiency and accuracy of data processing. Therefore, this paper improves the traditional parallel collaborative filtering algorithm, analyzes the execution flow of the collaborative filtering algorithm, discusses the shortcomings of the traditional parallel collaborative filtering algorithm, and then describes in detail the steps of improved the collaborative filtering algorithm from generating nodes scoring vectors, obtaining neighboring nodes and forming recommendation information. Finally, the improved parallel collaborative filtering algorithm is verified through three aspects of running time, speedup and recommended accuracy. Experimental results show that the improved parallel collaborative filtering algorithm proposed in this paper has better running efficiency and higher recommendation accuracy than traditional parallel algorithm based on co-occurrence matrix. [ABSTRACT FROM AUTHOR]

Clustering as a major task in data mining is responsible for discovering hidden patterns in unlabeled datasets. Finding the best clustering is also considered as one of the most challenging problems in data mining. Due to the problem complexity and the weaknesses of primary clustering algorithm, a large part of research has been directed toward ensemble clustering methods. Ensemble clustering aggregates a pool of base clusterings and produces an output clustering that is also named consensus clustering. The consensus clustering is usually better clustering than the output clusterings of the basic clustering algorithms. However, lack of quality in base clusterings makes their consensus clustering weak. In spite of some researches in selection of a subset of high quality base clusterings based on a clustering assessment metric, cluster-level selection has been always ignored. In this paper, a new clustering ensemble framework has been proposed based on cluster-level weighting. The certainty amount that the given ensemble has about a cluster is considered as the reliability of that cluster. The certainty amount that the given ensemble has about a cluster is computed by the accretion amount of that cluster by the ensemble. Then by selecting the best clusters and assigning a weight to each selected cluster based on its reliability, the final ensemble is created. After that, the paper proposes cluster-level weighting co-association matrix instead of traditional co-association matrix. Then, two consensus functions have been introduced and used for production of the consensus partition. The proposed framework completely overshadows the state-of-the-art clustering ensemble methods experimentally. [ABSTRACT FROM AUTHOR]

With the development of information technology, the rise of surveying and mapping big data and artificial intelligence, the lack of data is no longer a problem. However, the existing surveying and mapping data processing technology has been pursuing the accuracy of data (micro), and big data research just allows the data to be mixed and uncertain (macro). Therefore, although the traditional surveying and mapping data processing theory has accumulated a large number of technical advantages in micro data processing, the large‑scale and complexity of big data has become increasingly prominent, in which traditional calculation model and analysis algorithm cannot effectively support the efficient analysis and processing of big data. As the key to the intelligent era, data processing theory and method, how to adapt to the challenges and opportunities of new technology is worth our deep thinking. Driven by big data, new ideas and methods such as large‑scale data mining, machine learning and deep learning are booming, which greatly promote the fusion of multi‑source heterogeneous big data inside and outside the scene, effectively extract surface feature information from a variety of sensor data, and constantly improve the ability of surveying and mapping information acquisition and analysis. We think that the theory of surveying and mapping data also needs to be followed up, and the existing data processing methods need to be intelligent. Combined with the frontier hot spots, development trends and existing challenges of intelligent surveying and mapping, this paper explores the expansion direction of data processing theory. One is to promote the further development of surveying data processing theory, and the other is to provide reference for graduate students who are interested in entering the field of surveying and mapping big data. [ABSTRACT FROM AUTHOR]

With the rapid development of the Internet and the rapid development of big data analysis technology, data mining has played a positive role in promoting industry and academia. Classification is an important problem in data mining. This paper explores the background and theory of support vector machines (SVM) in data mining classification algorithms and analyzes and summarizes the research status of various improved methods of SVM. According to the scale and characteristics of the data, different solution spaces are selected, and the solution of the dual problem is transformed into the classification surface of the original space to improve the algorithm speed. Research Process. Incorporating fuzzy membership into multicore learning, it is found that the time complexity of the original problem is determined by the dimension, and the time complexity of the dual problem is determined by the quantity, and the dimension and quantity constitute the scale of the data, so it can be based on the scale of the data Features Choose different solution spaces. The algorithm speed can be improved by transforming the solution of the dual problem into the classification surface of the original space. Conclusion. By improving the calculation rate of traditional machine learning algorithms, it is concluded that the accuracy of the fitting prediction between the predicted data and the actual value is as high as 98%, which can make the traditional machine learning algorithm meet the requirements of the big data era. It can be widely used in the context of big data. [ABSTRACT FROM AUTHOR]

Box-constrained optimization problems in the real m × n matrix space have been widely applied in big data mining. However, efficient solution of them is still a challenge. In this paper, a new nonmonotone line search rule is first proposed by extending the well-known ones and inheriting their advantages. Then, by analyzing and exploiting properties of this rule, a new nonmonotone spectral projected gradient algorithm is developed to solve the box-constrained optimization problems in the matrix space. Global convergence of the developed algorithm is also established. Numerical tests are conducted on a series of randomly generated test problems and those in the set of benchmark test problems. Compared with other existing nonmonotone line search rules, our rule shows its advantages in terms of the significantly reduced number of function evaluations and significantly reduced number of iterations. To further validate applicability of this research, we apply the studied optimization problem and the developed algorithm to solve the problems of image clustering. Numerical results demonstrate that the proposed method can generate better clustering results and is more robust than the similar ones available in the literature. [ABSTRACT FROM AUTHOR]

In many modern applications, the generated data is a dynamic network. These networks are graphs that change over time by a sequence of update operations (node addition, node deletion, edge addition, edge deletion, and edge weight change). In such networks, it is inefficient to compute from scratch the solution of a data mining/machine learning task, after any update operation. Therefore in recent years, several so‐called dynamical algorithms have been proposed that update the solution, instead of computing it from scratch. In this paper, first we formulate this emerging setting and discuss its high‐level algorithmic aspects. Then, we review state of the art dynamical algorithms proposed for several data mining and machine learning tasks, including frequent pattern discovery, betweenness/closeness/PageRank centralities, clustering, classification, and regression. This article is categorized under:Technologies > Structure Discovery and ClusteringTechnologies > Machine LearningFundamental Concepts of Data and Knowledge > Big Data Mining [ABSTRACT FROM AUTHOR]

Many important applications continuously generate data, such as financial transaction administration, satellite monitoring, network flow monitoring, and web information processing. The data mining results are always evolving with the newly generated data. Obviously, for the clustering task, it is better to incrementally update the new clustering results based on the old data rather than to recluster all of the data from scratch. The incremental clustering approach is an essential way to solve the problem of clustering with growing Big Data. This paper proposes a boundary-profile-based incremental clustering (BPIC) method to find arbitrarily shaped clusters with dynamically growing datasets. This method represents the existing clustering results with a collection of boundary profiles and discards the inner points of clusters rather than keep all data. It greatly saves both time and space storage costs. To identify the boundary profile, this paper presents a boundary-vector-based boundary point detection (BV-BPD) algorithm that summarizes the structure of the existing clusters. The BPIC method processes each new point in an online fashion and updates the clustering results in a batch mode. When a new point arrives, the BPIC method either immediately labels it or temporarily puts it into a bucket according to the relationship between the new data and the boundary profiles. A bucket is employed to distinguish the noise from the potential seeds of new clusters and alleviate the effects of data order. When the bucket is full, the BPIC method will cluster the data within it and update the clustering results. Thus, the BPIC method is insensitive to noise and the order of new data, which is critical for the robustness of the incremental clustering process. In the experiments, the performance of the boundary point detection algorithm BV-BPD is compared with the state-of-the-art method. The results show that the BV-BPD is better than the state-of-the-art method. Additionally, the performance of BPIC and other two incremental clustering methods are investigated in terms of clustering quality, time and space efficiency. The experimental results indicate that the BPIC method is able to get a qualified clustering result on a large dataset with higher time and space efficiency. [ABSTRACT FROM AUTHOR]

A schedule-based system is a system that operates on or contains within a schedule of events and breaks at particular time intervals. Entities within the system show presence or absence in these events by entering or exiting the locations of the events. Given radio frequency identification (RFID) data from a schedule-based system, what can we learn about the system (the events and entities) through data mining? Which data mining methods can be applied so that one can obtain rich actionable insights regarding the system and the domain? The research goal of this paper is to answer these posed research questions, through the development of a framework that systematically produces actionable insights for a given schedule-based system. We show that through integrating appropriate data mining methodologies as a unified framework, one can obtain many insights from even a very simple RFID dataset, which contains only very few fields. The developed framework is general, and is applicable to any schedule-based system, as long as it operates under certain basic assumptions. The types of insights are also general, and are formulated in this paper in the most abstract way. The applicability of the developed framework is illustrated through a case study, where real world data from a schedule-based system is analyzed using the introduced framework. Insights obtained include the profiling of entities and events, the interactions between entity and events, and the relations between events. [ABSTRACT FROM AUTHOR]

Aiming at the problem of extensibility of big data analysis on massively parallel distributed systems and software platforms, this paper proposed a big data mining technique based on parameter-free CLUBS algorithm around centroids. The technique worked in a completely unsupervised manner, splitting clusters based on minimum quadratic distance criteria to separate data from noise, identified blocks containing only outliers by intermediate refinement and generated complete clusters for the remaining blocks, it designed a parallelized version of CLUBS to enable fast and efficient clustering of big data. Experiments show that the CLUBS parallel algorithm is not affected by data dimension and noise, and is better than the existing algorithms in terms of scalability and execution time. [ABSTRACT FROM AUTHOR]

Finding frequent itemsets in a continuous streaming data is an important data mining task which is widely used in network monitoring, Internet of Things data analysis and so on. In the era of big data, it is necessary to develop a distributed frequent itemset mining algorithm to meet the needs of massive streaming data processing. Apache Spark is a unified analytic engine for massive data processing which has been successfully used in many data mining fields. In this paper, we propose a distributed algorithm for mining frequent itemsets over massive streaming data named SWEclat. The algorithm uses sliding window to process streaming data and uses vertical data structure to store the dataset in the sliding window. This algorithm is implemented by Apache Spark and uses Spark RDD to store streaming data and dataset in vertical data format, so as to divide these RDDs into partitions for distributed processing. Experimental results show that SWEclat algorithm has good acceleration, parallel scalability and load balancing. [ABSTRACT FROM AUTHOR]

The k-nearest neighbors outlier detection is a simple yet effective widely renowned method in data mining. The actual application of this model in the big data domain is not feasible due to time and memory restrictions. Several distributed alternatives based on MapReduce have been proposed to enable this method to handle large-scale data. However, their performance can be further improved with new designs that fit with newly arising technologies. Furthermore, it gives to each attribute the same importance to outlier. There are several approaches to enhance its precision, with the entropy-based outlier detection being among the most successful ones. Entropy-based outlier detection computes attribute entropy of the data set to weighted distance formula for the outlier detection. Apart from the existing the k-nearest neighbors outlier detection to handle big datasets, there is not an entropy-based outlier detection to manage that volume of data. In this paper, we propose an entropy-based outlier detection based on Spark. It presents three separately stages. The first stage computes attribute entropy. The second stage finds the k nearest neighbors and calculates the degrees of outliers using the attribute entropy computed previously. The third stage ranks each point on the degrees of outliers and declares the top n points in this ranking to be outliers. Extensive experimental results show the advantages of the proposed method. This algorithm can improve the outlier detection precision, reduce the runtime and realize the effective large scale dataset outlier detection. [ABSTRACT FROM AUTHOR]

Because any MapReduce job requires a series of complex operations such as task scheduling and resource allocation independently, there are a lot of redundant disk I/O and resource duplicate application operations among multiple MapReduce jobs coordinated by the same algorithm, causing inefficient resource utilization in job computing process. Big data mining algorithms are usually divided into several MapReduce Jobs, taking ItemBased algorithm as an example, this paper analyzed the resource efficiency of mining algorithm with multi-MapReduce job collaboration scenario. It proposed an ItemBased algorithm based on Distributed Cache, which used Distributed Cache to cache I/O data between multiple MapReduce Jobs, broke the defect of independence between jobs, and reduced the waiting delay between Map and Reduce tasks. The experimental results show that, DistributedCache can improve the data reading speed of MapReduce jobs. The algorithm reconstructed by Distributed Cache greatly reduces the waiting delay between Map and Reduce tasks, and improves the resource efficiency by more than three times. [ABSTRACT FROM AUTHOR]

We prove in this paper that the expected value of the objective function of the k-means++ algorithm for samples converges to population expected value. As k-means++, for samples, provides with constant factor approximation for k-means objectives, such an approximation can be achieved for the population with increase of the sample size. This result is of potential practical relevance when one is considering using subsampling when clustering large data sets (large data bases). [ABSTRACT FROM AUTHOR]

This paper draws on regulatory governance scholarship to argue that the analytic phenomenon currently known as ‘Big Data’ can be understood as a mode of ‘design-based’ regulation. Although Big Data decision-making technologies can take the form of automated decision-making systems, this paper focuses on algorithmic decision-guidance techniques. By highlighting correlations between data items that would not otherwise be observable, these techniques are being used to shape the informational choice context in which individual decision-making occurs, with the aim of channelling attention and decision-making in directions preferred by the ‘choice architect’. By relying upon the use of ‘nudge’ – a particular form of choice architecture that alters people’s behaviour in a predictable way without forbidding any options or significantly changing their economic incentives, these techniques constitute a ‘soft’ form of design-based control. But, unlike the static Nudges popularised by Thaler and Sunstein [(2008).Nudge. London: Penguin Books] such as placing the salad in front of the lasagne to encourage healthy eating, Big Data analytic nudges are extremely powerful and potent due to their networked, continuously updated, dynamic and pervasive nature (hence ‘hypernudge’). I adopt a liberal, rights-based critique of these techniques, contrasting liberal theoretical accounts with selective insights from science and technology studies (STS) and surveillance studies on the other. I argue that concerns about the legitimacy of these techniques are not satisfactorily resolved through reliance on individual notice and consent, touching upon the troubling implications for democracy and human flourishing if Big Data analytic techniques driven by commercial self-interest continue their onward march unchecked by effective and legitimate constraints. [ABSTRACT FROM PUBLISHER]

With the rapid development of information technology, the era of big data has come. This paper focuses on the analysis and processing of data. How to mine valuable information in large data has been widely concerned by mining frequent itemsets in the data to derive. Association rules are an important part of data mining technology. The analysis of complex network structure based on improved botnet algorithm is the basis of network science. The study of complex network structure is helpful to understand and predict large data of complex network function and behavior. The conclusion of the system in the botnet algorithm process analysis provides some directional suggestions for the improvement of search engine functions and services. In theory, a set of simple and easy botnet algorithm for analyzing the satisfaction factors of clustering is proposed. The practice proves that the big data intelligent search engine based on differential evolution botnet algorithm has significant improvement in retrieval time, recall rate and precision, and can effectively realize personalized and accurate search. A complete big data mining system is implemented, which provides an efficient and easy-to-use tool for data mining botnet algorithms on large data sets. [ABSTRACT FROM AUTHOR]

Today, big data processing has become a challenging task due to the amount of data collected using various sensors increasingly significantly. To build knowledge and predict the data, traditional data mining methods calculate all numerical attributes into the memory simultaneously. The data stream method is a solution for processing and calculating data. The method streams incrementally in batch form; therefore, infrastructure memory is sufficient to develop knowledge. The existing method for data stream prediction is FIMT-DD (Fast Incremental Model Tree-Drift Detection). Using this method, knowledge is developed in tree form for every instance. In this paper, enhanced FIMT-DD is proposed using ARDEV (Average Restrain Divider of Evaluation Value). ARDEV utilizes the Chernoff bound approach with error evaluation, improvement in learning rate, modification of perceptron rule calculation, and utilization of activation function. Standard FIMT-DD separates the tree formation process and perceptron prediction. The proposed method evaluates and connects the development of the tree for knowledge formation and the perceptron rule for prediction. The prediction accuracy of the proposed method is measured using MAE, RMSE and MAPE. From the experiment performed, the utilization of ARDEV enhancement shows significant improvement in terms of accuracy prediction. Statistically, the overall accuracy prediction improvement is approximately 6.99 % compared to standard FIMT-DD with a traffic dataset. [ABSTRACT FROM AUTHOR]

In order to achieve the research from individual data to data system and from passive verification of data to active discovery, taking high dimensional data oriented data mining technology as the research object, an association rule redundancy processing algorithm based on hypergraph in data mining technology is studied according to the project requirements. The concepts of hypergraph and system are introduced to explore the construction of hypergraph on 3D matrix model. In view of the characteristics of big data, a new method of super edge definition is adopted, which improves the ability of dealing with problems. In the association rules redundancy and loop detection based on directed hypergraph, the association rules are transformed into directed hypergraph, and the adjacency matrix is redefined. The detection of redundancy and loop is transformed into the processing of connected blocks and circles in hypergraph, which provides a new idea and method for the redundant processing of association rules. The new method is applied to the data processing of practical projects. The experimental results show that the 3D matrix mathematical model and related data mining algorithms in this paper can find new high-quality knowledge from high-dimensional data. [ABSTRACT FROM AUTHOR]

Bearing is an important machine detail that appears in almost all mechanical systems. Estimating its operating condition online in order to hold the initiative in exploiting the systems, therefore, is one of the most urgent requirements. In this paper, we propose an online bearing damage identifying method named ASSBDIM based on adaptive neuro-fuzzy inference system (ANFIS), singular spectrum analysis (SSA), and sparse filtering (SF). It is an online process with offline and online phases. In the offline phase, by applying SSA and SF to the measured data stream typed big data with noise, both preprocessing data and extracting valuable information are implemented to build two offline databases signed Off_DaB and Off_testDaB. The ANFIS identifies the dynamic response of the mechanical system via Off_DaB. Based on Off_testDaB, the parameters of the ASSBDIM are then optimized. In the online phase, at each time, another database called On_DaB is built in the way similar to the one used for building the input space of the two offline databases. On_DaB participates as inputs of the ANFIS to estimate its outputs, which are then compared with the corresponding encoded outputs to specify bearing real status at this time. Survey results based on different data sources showed the effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]

The arrival of the big data era in the new century has made the traditional data mining algorithms unable to meet the requirements of big data mining in accuracy and efficiency. Therefore, a data mining algorithm based on efficient incremental kernel fuzzy clustering for big data was optimized—in this paper. First of all, the methods of big data mining and fuzzy clustering technique for data mining were summarized. Then, the data mining algorithm based on the incremental kernel fuzzy clustering was optimized. Finally, the method was validated by comparing with the stKFCM algorithm. The verification results showed that the improved algorithm was superior in performance and accuracy, but only a slight gap in running time. [ABSTRACT FROM AUTHOR]

Spatial co-location mining is a useful tool for discovering spatial association patterns of feature sets which are frequently observed together in nearby geographic space. Most of co-location mining techniques aim to find all prevalent co-located feature sets which satisfy a given prevalence threshold. However the result is often large, especially when the prevalence threshold is set low, or long co-location patterns present. Moreover the output has many redundant information which makes it difficult for users to filter useful patterns. This work introduces the problem of mining reduced sets of co-location patterns in order to concisely represent interesting spatial relationship patterns. With aiming two such outputs in the form of maximal and closed co-locations, this paper proposes an algorithmic framework to discover maximal co-location patterns and closed co-location patterns as well as all prevalent co-location patterns, and presents the algorithm details for each pattern discovery. The developed algorithms are correct and complete in finding maximal co-locations and closed co-locations. The experiment result shows that the framework reduces candidate feature sets effectively and finds co-location patterns efficiently. [ABSTRACT FROM AUTHOR]

Record Matching refers to identifying pairs of records that relate to the same entities across different data sources. In many applications of data mining, record matching is usually associated to quadratic complexity. In practice, the number of non-matching record pairs always far exceeds the number of matching pairs, and this is called imbalance problem. Blocking is a technique of data reduction, which can filter unlikely matching pairs before record matching. However, for big data there is no fast and effective blocking algorithm yet. In this paper, we report on big data infrastructure to improve efficiency of blocking. Our approach runs blocking process independently and distributedly on the partitions of whole data. To improve efficiency, we adopt a probabilistic technique to balance the speed and the effect of the algorithm that we proposed for distributed blocking. Our experimental analysis endorses the superiority of our technique and shows its novel scalability. [ABSTRACT FROM AUTHOR]

A machine learning approach for the recent detection of crossing faults is presented in the paper. The basis for the research are the data of the axle box inertial measurements on operational trains with the system ESAH-F. Within the machine learning approach the signal processing methods, as well as data reduction classification methods, are used. The wavelet analysis is applied to detect the spectral features at measured signals. The simple filter approach and sequential feature selection is used to find the most significant features and train the classification model. The validation and error estimates are presented and its relation to the number of selected features is analysed, as well. [ABSTRACT FROM AUTHOR]

Abstract The present society has been sculpted into a continuous data generator. In fact, the massive automatic data collection has generated a new genre of dataset, termed as 'high-dimensional data', which is characterized by a relatively small number of rows, in comparison to that of large number of columns (or dimensions). Among the vast data mining tasks, association rules have been extensively employed so as to describe the correlations between the variables found in a dataset. The task of mining association rules highly relies on the efficiency of the algorithms to extract all frequent itemsets that exist in the database. The focus towards improving run time and memory consumption of algorithms is strongly influenced by search strategies, effective pruning strategies, and the method of closure checking. Neither depth- nor breadth-first search may exert any variance without these techniques, mainly because the search space appears similar. With that, this paper investigated the strategies implemented in both row and column enumeration-based algorithms, hence proposing the RARE; a breadth-first bottom-up row-enumeration algorithm, in mining colossal closed itemsets in high-dimensional data. [ABSTRACT FROM AUTHOR]

Big data classification is very slow when using traditional machine learning classifiers, particularly when using a lazy and slow-by-nature classifier such as the k-nearest neighbors algorithm (KNN). This paper proposes a new approach which is based on sorting the feature vectors of training data in a binary search tree to accelerate big data classification using the KNN approach. This is done using two methods, both of which utilize two local points to sort the examples based on their similarity to these local points. The first method chooses the local points based on their similarity to the global extreme points, while the second method chooses the local points randomly. The results of various experiments conducted on different big datasets show reasonable accuracy rates compared to state-of-the-art methods and the KNN classifier itself. More importantly, they show the high classification speed of both methods. This strong trait can be used to further improve the accuracy of the proposed methods. [ABSTRACT FROM AUTHOR]

As a combinatorial optimization problem, feature selection has been widely used in machine learning and data mining. In this paper, a feature selection method using forest optimization algorithm based on contribution degree is proposed. The proposed method uses a contribution degree strategy which is embedded in forest optimization algorithm. The goal of the contribution degree is to guide the search process of the forest optimization algorithm to select features according to high class correlation and low redundancy between features. The proposed algorithm is verified on some data sets from the UCI repository and the experiments show that the proposed method improves the classification accuracy compared with some other methods. [ABSTRACT FROM AUTHOR]

Ensemble clustering has attracted much attention for its robustness, stability, and accuracy in academic and industry communities. In order to yield base clusterings with high quality and diversity simultaneously in ensemble clustering, many efforts have been done by exploiting different clustering models and data information. However, these methods neglect correlation between different base clusterings during the process of base clusterings generation, which is important to obtain a quality and diverse clustering decision. To overcome this deficiency, a sequential ensemble clusterings generation algorithm for mixed data is developed in this paper based on information entropy. The first high quality base clustering is yield by maximizing the entropy-based criterion. Afterward, a sequential paradigm is utilized to incrementally find more base clusterings, in which the diversity between a new base clustering and the former base partitions is measured by the normalized mutual information. Extensive experiments conducted on various data sets have demonstrated the superiority of our proposal as compared to several existing base clusterings generation algorithms. [ABSTRACT FROM AUTHOR]

Sparse modeling representative selection (SMRS) has been recently introduced for selecting the most relevant examples in data sets. SMRS exploits data self-representativeness coding in order to infer a coding matrix with block sparsity constraint. The relevance scores of samples are then derived from the estimated matrix of coefficients. Since SMRS is based on a linear model for data self-representation, it cannot always provide good relevant samples. Besides, most of its selected samples can be found in dense areas in input space. In this paper, we propose to overcome the SMRS method’s shortcomings that are related to the coding matrix estimation. We introduce two nonlinear data self-representativeness coding schemes that are based on Hilbert space and column generation. Experimental evaluation is carried out on summarizing a video movie and on summarizing training image data sets used for classification tasks. These experiments demonstrated that the proposed nonlinear methods can outperform state-of-the art selection methods. [ABSTRACT FROM AUTHOR]

Many techniques have been proposed to implement the Apriori algorithm on MapReduce framework but only a few have focused on performance improvement. FPC (Fixed Passes Combined-counting) and DPC (Dynamic Passes Combined-counting) algorithms combine multiple passes of Apriori in a single MapReduce phase to reduce the execution time. In this paper, we propose improved MapReduce based Apriori algorithms VFPC (Variable Size based Fixed Passes Combined-counting) and ETDPC (Elapsed Time based Dynamic Passes Combined-counting) over FPC and DPC. Further, we optimize the multi-pass phases of these algorithms by skipping pruning step in some passes, and propose Optimized-VFPC and Optimized-ETDPC algorithms. Quantitative analysis reveals that counting cost of additional un-pruned candidates produced due to skipped-pruning is less significant than reduction in computation cost due to the same. Experimental results show that VFPC and ETDPC are more robust and flexible than FPC and DPC whereas their optimized versions are more efficient in terms of execution time. [ABSTRACT FROM AUTHOR]

The data are rapidly expanding nowadays, which makes it very difficult to analyze valuable information from big data. Most of the existing data mining algorithms deal with big data problems at large time and space costs. This paper focuses on the sampling problem of big data and puts forward an efficient heuristic Cluster Sampling Arithmetic, called CSA. Many of the former researchers adopted random method to extract early sample set from the original data and then made a variety of different processing of the sample in order to obtain the corresponding minimum sample set, which is regarded as a representation of the original big data set. However, the final processing results of big data will be severely affected by the random sampling process at the beginning, resulting in lower comprehensiveness and quality of the final data results and longer processing time. Based on this view, CSA introduces the idea of clustering to obtain minimum sample set of big data, which is in contrast to the random sampling method in the current literature. CSA makes cluster analysis of the original data set and selects the center of each class as centralized members of the minimum sample set. It aims at ensuring that the sample distribution accords with the characteristics of the original data, guarantees the data integrity and reduces the processing time. The max-min distance means that the pattern recognition has been integrated into the clustering process in order to get the clustering center and prevent algorithm from local optimum. The final experimental results show that, compared with the existing work, CSA algorithm can efficiently reflect the characteristics of the original data and reduce the time of data processing. The obtained minimum sample set has also achieved good effects in the classification algorithm. [ABSTRACT FROM AUTHOR]

Density-based spatial clustering of applications with noise (DBSCAN) is a density-based clustering algorithm that has the characteristics of being able to discover clusters of any shape, effectively distinguishing noise points and naturally supporting spatial databases. DBSCAN has been widely used in the field of spatial data mining. This paper studies the parallelization design and realization of the DBSCAN algorithm based on the Spark platform, and solves the following problems that arise when computing macro data: the requirement of a great deal of calculation using the single-node algorithm; the low level of resource-utilization with the multi-node algorithm; the large time consumption; and the lack of instantaneity. The experimental results indicate that the proposed parallel algorithm design is able to achieve more stable speedup at an increased involved spatial data scale. [ABSTRACT FROM AUTHOR]

Short-term building cooling load prediction is the essential foundation for many building energy management tasks, such as fault detection and diagnosis, demand-side management and control optimization. Conventional methods, which heavily rely on physical principles, have limited power in practice as their performance is subject to many physical assumptions. By contrast, data-driven methods have gained huge interests due to their flexibility in model development and the rich data available in modern buildings. The rapid development in data science has provided advanced data analytics to tackle prediction problems in a more convenient, efficient and effective way. This paper investigates the potential of one of the most promising techniques in advanced data analytics, i.e., deep learning, in predicting 24-h ahead building cooling load profiles. Deep learning refers to a collection of machine learning algorithms which are powerful in revealing nonlinear and complex patterns in big data. Deep learning can be used either in a supervised manner to develop prediction models with given inputs and output (i.e., cooling load), or in an unsupervised manner to extract meaningful features from raw data as model inputs. This study exploits the potential of deep learning in both manners, and compares its performance in cooling load prediction with typical feature extraction methods and popular prediction techniques in the building field. The results show that deep learning can enhance the performance of building cooling load prediction, especially when used in an unsupervised manner for constructing high-level features as model inputs. Using the features extracted by unsupervised deep learning as inputs for cooling load prediction can evidently enhance the prediction performance. The findings are enlightening and could bring more flexible and effective solutions for building energy predictions. [ABSTRACT FROM AUTHOR]