Search

Clustering algorithms are part of algorithms of unsupervised-learning that are commonly used in different fields. Two major impacts on the data clustering algorithm have been the rapid developments and creation of electronic data. This includes the view of how the big data is stored as well as how it is processed. A cluster has a high level of resemblance to the same cluster and a low level of resemblance to other clusters. Clustering algorithms are commonly used in all fields, such as retail, banking, development, etc.. Even though different methodologies are proposed with different scenarios, but none of the existing methodologies are proven to be a better approaches. Here, in our work we have adopted an improved Map- Reduce programming model to combine the Canopy clustering and K-means clustering algorithms to process the Microarray data with the available commodity hardware with an aim to achieve better performance. The results obtained from different scenarios shown that the proposed method was capable of improving computational speed significantly by increasing the nodes as required. In this research, this paper explored the efficiency by evaluating and implementing the proposed Improved k-means with Improved Map Reduce algorithm which runs on Hadoop Frame work using Microarray dataset along with different datasets. [ABSTRACT FROM AUTHOR]

In order to explore potential value of explosively growing data in thermal power unit, this paper proposes a big data mining method based on Hadoop-based Spark cluster-computing framework and algorithms. Firstly, positive and negative balance methods are used to accurately obtain actual net coal consumption, and maximum information coefficient method is used to select all parameters related to optimization objectives. Then, Spark-based Mini-Batch K-means algorithm and Elbow method are constructed to divide whole operating modes. After that, all data are discretized and mapped to corresponding intervals by using Spark-based Elbow method and Mini-Batch K-means algorithm. Finally, Spark-based parallel FP-growth algorithm is used to deeply mine the potential relationships and laws. To verify the proposed method, a 350-MW thermal power unit is taken as a study case. The important conclusions are as follows: (1) the proposed Spark-based Mini-Batch K-means algorithm reduces the calculation time by 57.11% compared with Mini-Batch K-means algorithm, and 85.61% calculation time compared with K-means algorithm. The proposed Spark-based FP-growth algorithm reduces computational time by 32.8% compared with FP-growth algorithm. (2) Strong association rules of whole operating modes are mined, and operating optimization guidance schemes for important parameters are obtained. Take operating mode 1 as an example: if the optimal result can be reasonably applied, it can save 2.942 g coal per kilowatt hour. (3) Besides, we have found out some other potential relationships among parameters, which have important reference value for on-site operators to analyze economy of the thermal power unit. [ABSTRACT FROM AUTHOR]

In this paper, the clustering algorithm of big data fuzzy K-means is used to research and analyze the teaching management system. Since the values of the indicator variables of bad units are usually significantly different in smooth areas and discontinuous areas, we cluster the indicators of local areas through K-means clustering, so that only good units or bad units are included in each class. The server is deployed on the campus and is networked with the teaching computers of each classroom. Teachers log in to the class rate management system before or during class, log in to their respective users, select the corresponding class, and make corresponding attendance records according to the attendance of students in the class. The system automatically counts the number of expected arrivals, attendance, late arrivals, leave requests, and absenteeism for the class. Teachers can also fill in information feedback for this class: a list of absenteeism, classroom discipline, student learning status, teaching suggestions, and teaching testimonials. The development environment built by this system is a combination of PHP + MYSQL, which reasonably planned according to the overall needs. The development mode adopts the top-down model, which runs through the whole process from system development to testing to application. The most used floating point number encoding is the decimal floating point number encoding. The relationship of each module of the virtual reality teaching management system is sorted out, the overall framework of the virtual reality teaching management system is established, and the design of the business flow diagram of the management system is completed. This paper also uses Oracle database technology to further explain the design process of designing the system database. Finally, combined with the system design and implementation, the system test is completed from two aspects: system function test and performance test. [ABSTRACT FROM AUTHOR]

With the advancement of science and technology as well as the continual improvement of big data analysis technology, the accuracy of traditional data information classification has declined, making it impossible to assess English ability effectively. A competency evaluation model for college English teaching vacancies is built using this information and the big data architecture. The ability of the big data information model is evaluated and feature information of ability constraints is extracted using the predefined constraint parameter index analysis model. Simultaneously, the K-means clustering algorithm is used to cluster and integrate a series of index parameters of English ability using big data, and the English teaching resource allocation plan is completed in accordance with this, allowing for the scientific evaluation of English teaching ability. The results of the studies show that the clustering method utilized in the context of big data can aid in the evaluation of English competence. In the experiment, four test cycles of English teaching skills were set up, and the effectiveness of the English evaluation techniques described in this paper and two classical cluster evaluation methods were compared and tested. The research shows that using the method described in this paper to evaluate English teaching skills can significantly improve the full utilization of data. [ABSTRACT FROM AUTHOR]

One of the problems in implementing DRPs is the lack of sufficient understanding of consumers' behaviour when implementing DRPs. This paper compares consumers' load patterns annually by the improved Weighted Fuzzy Average (WFA) K‐means clustering method. According to the collected data, DRPs are discussed annually using a combined Convolutional Neural Network (CNN) and Long Short‐Term Memory (LSTM). To make CNN‐LSTM a practical algorithm for executing DRPs, Time Series Prediction (TSP) operations must be performed, and input data must be memorized. Finally, according to the TSP done for the obtained data, the implementation of DRPs in practice is facilitated. Then, the DRPs executed with the Deep Learning (DL) model and the power model will be prioritized by Technique for Order of Preference by Similarity to the Ideal Solution (TOPSIS) method, and decision indicators will be determined and weighted using the Shannon entropy method. The numerical studies section shows that the CNN‐LSTM algorithm is able to simulate DRPs with the Mean Absolute Percentage Error (MAPE) almost below 1% in the residential cluster and the highest MAPE in the commercial cluster of 15%. Nevertheless, this algorithm could easily give the same answer as the DRPs power model in the optimal selection of programs. [ABSTRACT FROM AUTHOR]

In the big data era, traditional data mining technology cannot meet the requirements of massive data processing with the background of intelligent manufacturing. Aiming at insufficient computing power and low efficiency in mining process, this paper proposes a improved K-means clustering algorithm based on the concept of distributed clustering in cloud computing environment. The improved algorithm (T.K-means) is combined with MapReduce computing framework of Hadoop platform to realize parallel computing, so as to perform processing tasks of massive data. In order to verify the practical performance of T.K-means algorithm, taking machining data of milling Ti-6Al-4V alloy as the mining object. The mapping relationship among cutting parameters, surface roughness, and material removal rate is mined, and the optimized value for cutting parameters is obtained. The results show that T.K-means algorithm can be used to mine the optimal cutting parameters, so that the best surface roughness can be obtained in milling Ti-6Al-4V titanium alloy. [ABSTRACT FROM AUTHOR]

In the era of big data, text as an information reserve database is very important, in all walks of life. From humanities research to government decision-making, from precision medicine to quantitative finance, from customer management to marketing, massive text, as one of the most important information carriers, plays an important role everywhere. The text data generated in these practical problems of humanities research, financial industry, marketing, and other fields often has obvious domain characteristics, often containing the professional vocabulary and unique language patterns in these fields and often accompanied by a variety of "noise." Dealing with such texts is a great challenge for the current technical conditions, especially for Chinese texts. A clustering algorithm provides a better solution for text big data information processing. Clustering algorithm is the main body of cluster analysis, K-means algorithm with its implementation principle is simple, low time complexity is widely used in the field of cluster analysis, but its K value needs to be preset, initial clustering center random selection into local optimal solution, other clustering algorithm, such as mean drift clustering, K-means clustering in mining text big data. In view of the problems of the above algorithm, this paper first extracts and analyzes the text big data and then does experiments with the clustering algorithm. Experimental conclusion: by analyzing large-scale text data limited to large-scale and simple data set, the traditional K-means algorithm has low efficiency and reduced accuracy, and the K-means algorithm is susceptible to the influence of initial center and abnormal data. According to the above problems, the K-means cluster analysis algorithm for data sets with large data volumes is analyzed and improved to improve its execution efficiency and accuracy on data sets with large data volume set. Mean shift clustering can be regarded as making many random centers move towards the direction of maximum density gradually, that is, moving their mean centroid continuously according to the probability density of data and finally obtaining multiple maximum density centers. It can also be said that mean shift clustering is a kernel density estimation algorithm. [ABSTRACT FROM AUTHOR]

With the evolution of the Internet and information technology, the era of big data is a new digital one. Accordingly, animation IP has been more and more widely welcomed and concerned with the continuous development of the domestic and international animation industry. Hence, animation video analysis will be a good landing application for computers. This paper proposes an algorithm based on clustering and cascaded SSD for object detection of animation characters in the big data environment. In the training process, the improved classification Loss function based on Focal Loss and Truncated Gradient was used to enhance the initial detection effect. In the detection phase, this algorithm designs a small target enhanced detection module cascaded with an SSD network. In this way, the high-level features corresponding to the small target region can be extracted separately to detect small targets, which can effectively enhance the detection effect of small targets. In order to further improve the effect of small target detection, the regional candidate box is reconstructed by a k-means clustering algorithm to improve the detection accuracy of the algorithm. Experimental results demonstrate that this method can effectively detect animation characters, and performance indicators are better than other existing algorithms. [ABSTRACT FROM AUTHOR]

Aim/Purpose The clustering techniques are normally considered to determine the significant and meaningful subclasses purposed in datasets. It is an unsupervised type of Machine Learning (ML) where the objective is to form groups from objects based on their similarity and used to determine the implicit relationships between the different features of the data. Cluster Analysis is considered a significant problem area in data exploration when dealing with arbitrary shape problems in different datasets. Clustering on large data sets has the following challenges: (1) clusters with arbitrary shapes; (2) less knowledge discovery process to decide the possible input features; (3) scalability for large data sizes. Density-based clustering has been known as a dominant method for determining the arbitrary-shape clusters. Background Existing density-based clustering methods commonly cited in the literature have been examined in terms of their behavior with data sets that contain nested clusters of varying density. The existing methods are not enough or ideal for such data sets, because they typically partition the data into clusters that cannot be nested. Methodology A density-based approach on traditional center-based clustering is introduced that assigns a weight to each cluster. The weights are then utilized in calculating the distances from data vectors to centroids by multiplying the distance by the centroid weight. Contribution In this paper, we have examined different density-based clustering methods for data sets with nested clusters of varying density. Two such data sets were used to evaluate some of the commonly cited algorithms found in the literature. Nested clusters were found to be challenging for the existing algorithms. In utmost cases, the targeted algorithms either did not detect the largest clusters or simply divided large clusters into non-overlapping regions. But, it may be possible to detect all clusters by doing multiple runs of the algorithm with different inputs and then combining the results. This work considered three challenges of clustering methods. Findings As a result, a center with a low weight will attract objects from further away than a centroid with higher weight. This allows dense clusters inside larger clusters to be recognized. The methods are tested experimentally using the K-means, DBSCAN, TURN*, and IDCUP algorithms. The experimental results with different data sets showed that IDCUP is more robust and produces better clusters than DBSCAN, TURN*, and K-means. Finally, we compare K-means, DBSCAN, TURN*, and to deal with arbitrary shapes problems at different datasets. IDCUP shows better scalability compared to TURN*. Future Research As future recommendations of this research, we are concerned with the exploration of further available challenges of the knowledge discovery process in clustering along with complex data sets with more time. A hybrid approach based on density-based and model-based clustering algorithms needs to compare to achieve maximum performance accuracy and avoid the arbitrary shapes related problems including optimization. It is anticipated that the comparable kind of the future suggested process will attain improved performance with analogous precision in identification of clustering shapes. [ABSTRACT FROM AUTHOR]

Unmanned aerial vehicles (UAVs) with auto-pilot capabilities are often used for surveillance and patrol. Pilots set the flight points on a map in order to navigate to the imaging point where surveillance or patrolling is required. However, there is the limit denoting the information such as absolute altitudes and angles. Therefore, it is required to set the information accurately. This paper hereby proposes a method to construct environmental symmetric big data using an unmanned aerial vehicle (UAV) during flight by designating the imaging and non-imaging points for surveillance and patrols. The K-Means-based algorithm proposed in this paper is then employed to divide the imaging points, which is set by the pilot, into K clusters, and K imaging points are determined using these clusters. Flight data are then used to set the points to which the UAV will fly. In our experiment, flight records were gathered through an UAV in order to monitor a stadium and the imaging and non-imaging points were set using the proposed method and compared with the points determined by a traditional K-Means algorithm. Through the proposed method, the cluster centroids and cumulative distance of its members were reduced by 87.57% more than with the traditional K-Means algorithm. With the traditional K-Means algorithm, imaging points were not created in the five points desired by the pilot, and two incorrect points were obtained. However, with the proposed method, two incorrect imaging points were obtained. Due to these two incorrect imaging points, the two points desired by the pilot were not generated. [ABSTRACT FROM AUTHOR]

In this paper we propose a criterion to balance the processing time and the solution quality of k-means cluster algorithms when applied to instances where the number n of objects is big. The majority of the known strategies aimed to improve the performance of k-means algorithms are related to the initialization or classification steps. In contrast, our criterion applies in the convergence step, namely, the process stops whenever the number of objects that change their assigned cluster at any iteration is lower than a given threshold. Through computer experimentation with synthetic and real instances, we found that a threshold close to 0.03n involves a decrease in computing time of about a factor 4/100, yielding solutions whose quality reduces by less than two percent. These findings naturally suggest the usefulness of our criterion in Big Data realms. [ABSTRACT FROM AUTHOR]

Traditional data analytics tools are designed to deal with the asymmetrical type of data i.e., structured, semi-structured, and unstructured. The diverse behavior of data produced by different sources requires the selection of suitable tools. The restriction of recourses to deal with a huge volume of data is a challenge for these tools, which affects the performances of the tool's execution time. Therefore, in the present paper, we proposed a time optimization model, shares common HDFS (Hadoop Distributed File System) between three Name-node (Master Node), three Data-node, and one Client-node. These nodes work under the DeMilitarized zone (DMZ) to maintain symmetry. Machine learning jobs are explored from an independent platform to realize this model. In the first node (Name-node 1), Mahout is installed with all machine learning libraries through the maven repositories. The second node (Name-node 2), R connected to Hadoop, is running through the shiny-server. Splunk is configured in the third node (Name-node 3) and is used to analyze the logs. Experiments are performed between the proposed and legacy model to evaluate the response time, execution time, and throughput. K-means clustering, Navies Bayes, and recommender algorithms are run on three different data sets, i.e., movie rating, newsgroup, and Spam SMS data set, representing structured, semi-structured, and unstructured data, respectively. The selection of tools defines data independence, e.g., Newsgroup data set to run on Mahout as others cannot be compatible with this data. It is evident from the outcome of the data that the performance of the proposed model establishes the hypothesis that our model overcomes the limitation of the resources of the legacy model. In addition, the proposed model can process any kind of algorithm on different sets of data, which resides in its native formats. [ABSTRACT FROM AUTHOR]

The study was conducted by applying machine learning and data mining methods to treatment personalization. This allows individual patient characteristics to be investigated. The personalization method was built on the clustering method and associative rules. It was suggested to determine the average distance between instances in order to find the optimal performance metrics. The formalization of the medical data preprocessing stage was proposed in order to find personalized solutions based on current standards and pharmaceutical protocols. The patient data model was built using time-dependent and time-independent parameters. Personalized treatment is usually based on the decision tree method. This approach requires significant computation time and cannot be parallelized. Therefore, it was proposed to group people by conditions and to determine deviations of parameters from the normative parameters of the group, as well as the average parameters. The novelty of the paper is the new clustering method, which was built from an ensemble of cluster algorithms, and the usage of the new distance measure with Hopkins metrics, which were 0.13 less than for the k-means method. The Dunn index was 0.03 higher than for the BIRCH (balanced iterative reducing and clustering using hierarchies) algorithm. The next stage was the mining of associative rules provided separately for each cluster. This allows a personalized approach to treatment to be created for each patient based on long-term monitoring. The correctness level of the proposed medical decisions is 86%, which was approved by experts. [ABSTRACT FROM AUTHOR]