Search

The commonly used method for network intrusion is based on pattern matching systems, but these systems do not have high detection accuracy when facing large-scale high-speed network traffic environments. In addition, when facing a wide variety of network attacks, relying solely on a certain network security technology is difficult to ensure network security. Therefore, a distributed network intrusion prevention system based on the Spark framework and dynamic information security theory model was innovatively proposed to improve the detection efficiency of network intrusion and solve these issues. The architecture and functional structure of the network intrusion prevention system were constructed by improving the random forest algorithm and Spark. In addition, the dynamic network intrusion prevention system was designed on the basis of the Policy Protection Detection Response (P2DR) model and the Protection Detection Reaction Recovery (PDRR) model to cope with the diverse network attacks and make up for the lack of dynamic defense based on improved forest algorithm and Spark. The test results showed that the network intrusion prevention system based on the improved random forest algorithm and Spark had a maximum detection time of 13,593 ms, a minimum detection time of 13,318 ms, and an average detection time of 13,468 ms when the data were 6000 pieces. The average success rate of the dynamic network intrusion prevention system based on the dynamic information security theory model was 87.72%, the average detection rate was 71.68%, and the average false alarm rate was 17.23%. The F1 values corresponding to the improved random forest algorithm under 7 different attack types of data were 0.985, 0.983, 0.876, 0.843, 0.797, 0.983, and 0.890, respectively, which were significantly better than the comparison algorithm. It can be seen that the dynamic network intrusion prevention system based on the improved random forest algorithm, Spark, and the dynamic information security theoretical model, has good performance and can effectively detect network intrusions, providing technical support for solving network intrusion problems in reality. The contribution of the research is reflected in the improvement of the detection performance of network intrusion detection systems and the reduction of detection time and false alarm rates. [ABSTRACT FROM AUTHOR]

The performance optimization of Apache Spark, a widely used distributed computing framework, is crucial for the efficient execution of data-intensive workloads. However, the automatic tuning of around 180 internal configuration features of Spark for getting optimum performance can be a complex, expensive, and time-consuming task. This research paper aims to address these challenges along with the task of reducing execution time and tuning cost for benchmark Spark MLlib workloads on a benchmark dataset by automating the tuning process. A grid search based novel approach Reduced Internal Configuration Grid (RICG) is proposed that reduces the Spark's configuration feature values grid levaraging exponential projective sampling. A significant reduction in Spark workload execution time is obtained at a low cost without compromising the performance of the CPU, memory and network resources of the system. Extensive experimentation and evaluation demonstrate the effectiveness of the proposed approach, showcasing notable reductions of around 26–30% in execution time for benchmark workloads. This reduction is achieved by cutting the features tuning cost by approximately 58 to 60% compared to the prior best grid search based approach. The proposed RICG model outperforms existing statistical trial-test approaches, striking a favorable balance between performance optimization and cost optimization. Lastly, the logs generated by the RICG model are used to predict the performance of Spark application workloads by using the boosting algorithms with around 90% accuracy. The findings of this research highlight the potential of automated tuning techniques to enhance the performance of Apache Spark-based applications with cost efficiency. [ABSTRACT FROM AUTHOR]

Wavelet neural networks (WNN) have been applied in many fields to solve regression as well as classification problems. After the advent of big data, as data gets generated at a brisk pace, it is imperative to analyze it as soon as it is generated owing to the fact that the nature of the data may change dramatically in short time intervals. This is necessitated by the fact that big data is all pervasive and throws computational challenges for data scientists. Therefore, in this paper, we built an efficient Scalable, Parallelized Wavelet Neural Network (SPWNN) which employs the parallel stochastic gradient algorithm (SGD) algorithm. SPWNN is designed and developed under both static and streaming environments in the horizontal parallelization framework. SPWNN is implemented by using Morlet and Gaussian functions as activation functions. This study is conducted on big datasets like gas sensor data which has more than 4 million samples and medical research data which has more than 10,000 features, which are high dimensional in nature. The experimental analysis indicates that in the static environment, SPWNN with Morlet activation function outperformed SPWNN with Gaussian on the classification datasets. However, in the case of regression, there is no clear trend was observed. In contrast, in the streaming environment i.e., Gaussian outperformed Morlet on the classification and Morlet outperformed Gaussian on the regression datasets. Overall, the proposed SPWNN architecture achieved a speedup of 1.22 - 1.78. [ABSTRACT FROM AUTHOR]

The cooperative coevolution framework was introduced to address the shortcomings of metaheuristic algorithms in solving continuous large-scale global optimization problems. By dividing the problem into subcomponents which can be optimized separately, the framework can improve on both the solution's quality as well as the computational speed by exposing a degree of parallelism. Distributed computing platforms, such as Apache Spark, have long been used to improve the speed of different algorithms in solving computational problems. This work proposes a distributed implementation of the cooperative coevolution framework for solving large-scale global optimization problems on the Apache Spark distributed computing platform. By using a formerly outlined distributed variant of the cooperative coevolution framework, features of the Spark platform are utilized to enhance the computational speed of the algorithm while maintaining comparable search quality to other works in the literature. To test for the proposed implementation's improvement in computational speed, the CEC 2010 large-scale global optimization benchmark functions are used due to the diversity they offer in terms of complexity, separability and modality. Results of the proposed distributed implementation suggest that a speedup of up to ×3.36 is possible on large-scale global optimization benchmarks using the Apache Spark platform. [ABSTRACT FROM AUTHOR]

The large amount of data generated every day makes necessary the re-implementation of new methods capable of handle with massive data efficiently. This is the case of Association Rules, an unsupervised data mining tool capable of extracting information in the form of IF-THEN patterns. Although several methods have been proposed for the extraction of frequent itemsets (previous phase before mining association rules) in very large databases, the high computational cost and lack of memory remains a major problem to be solved when processing large data. Therefore, the aim of this paper is three fold: (1) to review existent algorithms for frequent itemset and association rule mining, (2)to develop new efficient frequent itemset Big Data algorithms using distributive computation, as well as a new association rule mining algorithm in Spark, and (3) to compare the proposed algorithms with the existent proposals varying the number of transactions and the number of items. To this purpose, we have used the Spark platform which has been demonstrated to outperform existing distributive algorithmic implementations. [ABSTRACT FROM AUTHOR]

Conventional micro-batch streaming systems introduce high tail latency for the following two reasons. First, they trigger data processing on a fixed time interval, without taking into account the data ingestion rate and data analysis processing rate. Second, in a multi-query environment, their scheduling methods do not consider the real-time characteristics of the application and each query. This paper presents zStream, a low latency micro-batch streaming system implemented on Apache Spark. The main idea of zStream's design is to use a reference value called deadline for each query to constrain latency around a specific time. Considering the deadline information of each query, zStream controls the admission of every micro-batch dynamically and schedules them by their priorities, significantly reducing the average and tail latency while maintaining throughput. Experimental results show that the tail latency of zStream is reduced by a maximum of 34% compared to the original Spark. Also, zStream reduces the average latency per query by up to 48% while stably maintaining throughput compared to the original Spark. [ABSTRACT FROM AUTHOR]

The rapidly increasing complexity and scale of optimization problems pose challenges to search ability and performance of traditional evolutionary algorithms which could be only executed sequentially without scalability and are difficult to obtain an ideal solution in a reasonable time. In this paper, Spark-ITGO, a parallel and scalable invasive tumor growth optimization algorithm on Spark, is presented based on the serial invasive tumor growth optimization (ITGO). In Spark-ITGO, a parallel multiple-tumor evolution model is proposed to search the optimal solution of problems. A balanced multi-island optimal migration strategy is designed to increase diversity of population and prevent converging into a local optimum. Additionally, a universal parallel evolutionary algorithm framework is implemented based on resilient distributed dataset (RDD) and central broadcast mechanism. Spark-ITGO is evaluated on benchmark experiments of CEC2013 and CEC2010 LSGO and the results show that it achieves great scalability and performs better than other evolutionary algorithms. [ABSTRACT FROM AUTHOR]

Improper data replacement and inappropriate selection of job scheduling policy are important reasons for the degradation of Spark system operation speed, which directly causes the performance degradation of Spark parallel computing. In this paper, we analyze the existing caching mechanism of Spark and find that there is still more room for optimization of the existing caching policy. For the task structure analysis, the key information of Spark tasks is taken out to obtain the data and memory usage during the task runtime, and based on this, an RDD weight calculation method is proposed, which integrates various factors affecting the RDD usage and establishes an RDD weight model. Based on this model, a minimum weight replacement algorithm based on RDD structure analyzing is proposed. The algorithm ensure that the relatively more valuable data in the data replacement process can be cached into memory. In addition, the default job scheduling algorithm of the Spark framework considers a single factor, which cannot form effective scheduling for jobs and causes a waste of cluster resources. In this paper, an adaptive job scheduling policy based on job classification is proposed to solve the above problem. The policy can classify job types and schedule resources more effectively for different types of jobs. The experimental results show that the proposed dynamic data replacement algorithm effectively improves Spark's memory utilization. The proposed job classification-based adaptive job scheduling algorithm effectively improves the system resource utilization and shortens the job completion time. [ABSTRACT FROM AUTHOR]

Over the past few years, cloud computing model (e.g., Spark) has aroused huge attention. Differential evolution (DE) has been applied to cloud computing models by a number of researchers for its merits in solving large-scale global optimization problems (LSGO), and remarkable results have been achieved. Moreover, we noticed that a combination of better topology and migration strategy is critical to solve the mentioned problems when DE algorithm acts as an internal optimizer for Spark cloud computing model. However, rare studies have been conducted to combine the combination to enhance the performance of DE algorithm for solving large-scale global optimization problems. Thus, inspired by the mentioned insights, we propose a novel grouping topology model that uses DE variants as internal optimizers to solve LSGO problems, called SgtDE. In SgtDE, population is split into subgroups evenly, and various topology structures are introduced to migrate individuals between and within subgroups. In this paper, five types of DE are adopted as the internal optimizers. By comparing the 20 benchmark functions presented on CEC2010, the results demonstrate that the SgtDE, especially a combination of better topology and migration strategy, exhibits significant performance in applying various DE variants. Thus, the SgtDE can act as the next generation optimizer of the cloud computing platform. [ABSTRACT FROM AUTHOR]

The k-nearest neighbors outlier detection is a simple yet effective widely renowned method in data mining. The actual application of this model in the big data domain is not feasible due to time and memory restrictions. Several distributed alternatives based on MapReduce have been proposed to enable this method to handle large-scale data. However, their performance can be further improved with new designs that fit with newly arising technologies. Furthermore, it gives to each attribute the same importance to outlier. There are several approaches to enhance its precision, with the entropy-based outlier detection being among the most successful ones. Entropy-based outlier detection computes attribute entropy of the data set to weighted distance formula for the outlier detection. Apart from the existing the k-nearest neighbors outlier detection to handle big datasets, there is not an entropy-based outlier detection to manage that volume of data. In this paper, we propose an entropy-based outlier detection based on Spark. It presents three separately stages. The first stage computes attribute entropy. The second stage finds the k nearest neighbors and calculates the degrees of outliers using the attribute entropy computed previously. The third stage ranks each point on the degrees of outliers and declares the top n points in this ranking to be outliers. Extensive experimental results show the advantages of the proposed method. This algorithm can improve the outlier detection precision, reduce the runtime and realize the effective large scale dataset outlier detection. [ABSTRACT FROM AUTHOR]

Nowadays, video cameras are increasingly used for surveillance, monitoring, and activity recording. These cameras generate high resolution image and video data at large scale. Processing such large scale video streams to extract useful information with time constraints is challenging. Traditional methods do not offer scalability to process large scale data. In this paper, we propose and evaluate cloud services for high resolution video streams in order to perform line detection using Canny edge detection followed by Hough transform. These algorithms are often used as preprocessing steps for various high level tasks including object, anomaly, and activity recognition. We implement and evaluate both Canny edge detector and Hough transform algorithms in Hadoop and Spark. Our experimental evaluation using Spark shows an excellent scalability and performance compared to Hadoop and standalone implementations for both Canny edge detection and Hough transform. We obtained a speedup of 10.8 × and 9.3 × for Canny edge detection and Hough transform respectively using Spark. These results demonstrate the effectiveness of parallel implementation of computer vision algorithms to achieve good scalability for real-world applications. [ABSTRACT FROM AUTHOR]

With the rapid development of mobile devices, global positioning system and Web2.0 technology, more and more users are choosing to share rich information in location-based social networks. Point of interest (POI) recommendation is a hot research direction in recent years, this paper first based on the existed recommending research results on the POI, using methods of linear weighting and cascading combination to integrate social factors, geographical factors and time factors to recommend POI, for the sake of improving effectiveness of results. Then, in the Spark cluster environment, it was extended to improve the efficiency and scalability of POI recommendation. Experimental results showed that the implementation of comprehensive POI recommendation methods in Spark cluster environment was more efficient, as the amount of data increased, efficiency advantage was increasingly obvious. [ABSTRACT FROM AUTHOR]

In the digital revolution era the work style of the people changed into real and live communicative systems. Smart cities are evolved from the smart ecosystem with various domains like healthcare, information exchange, transportation, buildings etc., the presence of multiple information system makes these smart cities into a heterogeneous which includes multiple data transfer based on the large no of interconnected sub systems. Since the systems provide data from various terminals to the host the amount of information generates is same as the chances of increasing the challenges and issues in the smart city. These flow data from various sources are handled using cluster computing and this proposed research provides the data flow in the smart city using cluster computing. Apache spark is the tool used to handle the issues in the smart city which is much better than the Hadoop. The results discuss the comparison between the applications in same environment on both the tools. [ABSTRACT FROM AUTHOR]

In the big data age, the traditional parallel collaborative filtering algorithm cannot meet the needs of data analysis in the efficiency and accuracy of data processing. Therefore, this paper improves the traditional parallel collaborative filtering algorithm, analyzes the execution flow of the collaborative filtering algorithm, discusses the shortcomings of the traditional parallel collaborative filtering algorithm, and then describes in detail the steps of improved the collaborative filtering algorithm from generating nodes scoring vectors, obtaining neighboring nodes and forming recommendation information. Finally, the improved parallel collaborative filtering algorithm is verified through three aspects of running time, speedup and recommended accuracy. Experimental results show that the improved parallel collaborative filtering algorithm proposed in this paper has better running efficiency and higher recommendation accuracy than traditional parallel algorithm based on co-occurrence matrix. [ABSTRACT FROM AUTHOR]

MapReduce is the most popular framework for distributed processing. Recently, the scalability of data mining and machine learning algorithms has significantly improved with help from MapReduce. However, MapReduce does not handle iterative algorithms very efficiently. The problem is that many data mining and machine learning algorithms are iterative by nature. In order to overcome the limitations of MapReduce, many advanced distributed systems have been developed, including HaLoop, iMapReduce, Twister, and Spark. In this paper, we identify and categorize the limitations of MapReduce in handling iterative algorithms, and then, experimentally investigate the consequences of these limitations by using the most flexible and stable distributed system, Spark. According to our experiment results, the network I/O overhead was the primary factor that affected system performance the most. The disk I/O overhead also affected system performance, but it was less significant than the network I/O overhead. For the synchronization overhead, it affected system performance only when the static data was not cached. [ABSTRACT FROM AUTHOR]

Spark has been increasingly employed by industries for big data analytics recently, due to its resilience, scalability and efficient in-memory distributed programming model. Meanwhile, the rapid growing community is also actively incubating a rich ecosystem around Spark to tackle various big data challenges. The current benchmarks fall short in providing guidance of development, optimization, configuration and deployment of Spark. To this end, we introduce SparkBench, a Spark specific benchmarking suite. It selectively embraces a set of representative applications to identify various performance bottlenecks and reveals the resource consumption behaviors across execution phases. Overall, SparkBench covers four critical usage patterns of Spark, including machine learning, graph processing, stream computations and SQL query processing. We present comprehensive characterization of resource consumptions, data flows and timing information under different execution patterns and demonstrate that SparkBench can effectively guide the optimization of data analytic platforms to better suit for various workloads. [ABSTRACT FROM AUTHOR]

Metaheuristics are gaining increasing recognition in many research areas, computational systems biology among them. Recent advances in metaheuristics can be helpful in locating the vicinity of the global solution in reasonable computation times, with Differential Evolution (DE) being one of the most popular methods. However, for most realistic applications, DE still requires excessive computation times. With the advent of Cloud Computing effortless access to large number of distributed resources has become more feasible, and new distributed frameworks, like Spark, have been developed to deal with large scale computations on commodity clusters and cloud resources. In this paper we propose a parallel implementation of an enhanced DE using Spark. The proposal drastically reduces the execution time, by means of including a selected local search and exploiting the available distributed resources. The performance of the proposal has been thoroughly assessed using challenging parameter estimation problems from the domain of computational systems biology. Two different platforms have been used for the evaluation, a local cluster and the Microsoft Azure public cloud. Additionally, it has been also compared with other parallel approaches, another cloud-based solution (a MapReduce implementation) and a traditional HPC solution (a MPI implementation) [ABSTRACT FROM AUTHOR]

Water pollution incidents have occurred frequently in recent years, causing severe damages, economic loss and long-lasting society impact. A viable solution is to install water quality monitoring sensors in water supply networks (WSNs) for real-time pollution detection, thereby mitigating the risk of catastrophic contamination incidents. Given the significant cost of placing sensors at all locations in a network, a critical issue is where to deploy sensors within WSNs, while achieving rapid detection of contaminant events. Existing studies have mainly focused on sensor placement in water distribution systems (WDSs). However, the problem is still not adequately addressed, especially for large scale WSNs. In this paper, we investigate the sensor placement problem in large scale WDSs with the objective of minimizing the impact of contamination events. Specifically, we propose a two-phase Spark-based genetic algorithm (SGA). Experimental results show that SGA outperforms other traditional algorithms in both accuracy and efficiency, which validates the feasibility and effectiveness of our proposed approach. [ABSTRACT FROM AUTHOR]

In order to store and analyze the increasing data in recent years, big data techniques are applied to many fields such as healthcare, manufacturing, telecommunications, retail, energy, transportation, automotive, security, environment, etc. This work implements a city traffic state assessment system in cloud using a novel big data architecture. The proposed system provides the real-time busses location and real-time traffic state, especially the real-time traffic state nearby, through open data, cloud computing, bid data technology, clustering methods, and irregular moving average. With the high-scalability cloud technologies, Hadoop and Spark, the proposed system architecture is first implemented successfully and efficiently. Next, we utilize irregular moving average and clustering methods to find the area of traffic jam. Finally, three important experiments are performed. The first experiment indicates that the computing ability of Spark is better than that of Hadoop. The second experiment applies Spark to process bus location data under different number of executors. In the last experiment, we apply irregular moving average and clustering methods to efficiently find the area of traffic jam in Taiwan Boulevard which is the main road in Taichung city. Based on these experimental results, the provided system services are present via an advanced web technology. [ABSTRACT FROM AUTHOR]

Frequent itemset mining is an essential step in the process of association rule mining. Conventional approaches for mining frequent itemsets in big data era encounter significant challenges when computing power and memory space are limited. This paper proposes an efficient distributed frequent itemset mining algorithm (DFIMA) which can significantly reduce the amount of candidate itemsets by applying a matrix-based pruning approach. The proposed algorithm has been implemented using Spark to further improve the efficiency of iterative computation. Numeric experiment results using standard benchmark datasets by comparing the proposed algorithm with the existing algorithm, parallel FP-growth, show that DFIMA has better efficiency and scalability. In addition, a case study has been carried out to validate the feasibility of DFIMA. [ABSTRACT FROM AUTHOR]