Search

Aiming at the problem that the Spark scalable distributed platform does not consider the computing capabilities of heterogeneous cluster nodes and load balance during job task scheduling, which affects the system performance, this paper constructs heterogeneous cluster nodes load balance scheduling policy under the Spark environment, Heterogeneous cluster node predicts the data distribution characteristics according to the sampling algorithm. divides the data into balancing partitions. According to the static load and dynamic load weight distribution, heterogeneous cluster node obtains the real-time load, and dynamically schedules job tasks. Finally. Wordcount, TeraSort, and K-means three benchmark tests were used to compare and analyze during heterogeneous cluster operation. Experimental results show that this algorithm can reduce the execution time significantly, and improve the performance of heterogeneous cluster. [ABSTRACT FROM AUTHOR]

The commonly used method for network intrusion is based on pattern matching systems, but these systems do not have high detection accuracy when facing large-scale high-speed network traffic environments. In addition, when facing a wide variety of network attacks, relying solely on a certain network security technology is difficult to ensure network security. Therefore, a distributed network intrusion prevention system based on the Spark framework and dynamic information security theory model was innovatively proposed to improve the detection efficiency of network intrusion and solve these issues. The architecture and functional structure of the network intrusion prevention system were constructed by improving the random forest algorithm and Spark. In addition, the dynamic network intrusion prevention system was designed on the basis of the Policy Protection Detection Response (P2DR) model and the Protection Detection Reaction Recovery (PDRR) model to cope with the diverse network attacks and make up for the lack of dynamic defense based on improved forest algorithm and Spark. The test results showed that the network intrusion prevention system based on the improved random forest algorithm and Spark had a maximum detection time of 13,593 ms, a minimum detection time of 13,318 ms, and an average detection time of 13,468 ms when the data were 6000 pieces. The average success rate of the dynamic network intrusion prevention system based on the dynamic information security theory model was 87.72%, the average detection rate was 71.68%, and the average false alarm rate was 17.23%. The F1 values corresponding to the improved random forest algorithm under 7 different attack types of data were 0.985, 0.983, 0.876, 0.843, 0.797, 0.983, and 0.890, respectively, which were significantly better than the comparison algorithm. It can be seen that the dynamic network intrusion prevention system based on the improved random forest algorithm, Spark, and the dynamic information security theoretical model, has good performance and can effectively detect network intrusions, providing technical support for solving network intrusion problems in reality. The contribution of the research is reflected in the improvement of the detection performance of network intrusion detection systems and the reduction of detection time and false alarm rates. [ABSTRACT FROM AUTHOR]

The performance optimization of Apache Spark, a widely used distributed computing framework, is crucial for the efficient execution of data-intensive workloads. However, the automatic tuning of around 180 internal configuration features of Spark for getting optimum performance can be a complex, expensive, and time-consuming task. This research paper aims to address these challenges along with the task of reducing execution time and tuning cost for benchmark Spark MLlib workloads on a benchmark dataset by automating the tuning process. A grid search based novel approach Reduced Internal Configuration Grid (RICG) is proposed that reduces the Spark's configuration feature values grid levaraging exponential projective sampling. A significant reduction in Spark workload execution time is obtained at a low cost without compromising the performance of the CPU, memory and network resources of the system. Extensive experimentation and evaluation demonstrate the effectiveness of the proposed approach, showcasing notable reductions of around 26–30% in execution time for benchmark workloads. This reduction is achieved by cutting the features tuning cost by approximately 58 to 60% compared to the prior best grid search based approach. The proposed RICG model outperforms existing statistical trial-test approaches, striking a favorable balance between performance optimization and cost optimization. Lastly, the logs generated by the RICG model are used to predict the performance of Spark application workloads by using the boosting algorithms with around 90% accuracy. The findings of this research highlight the potential of automated tuning techniques to enhance the performance of Apache Spark-based applications with cost efficiency. [ABSTRACT FROM AUTHOR]

Credit fraud is a common financial crime that causes significant economic losses to financial institutions. To address this issue, researchers have proposed various fraud detection methods. Recently, research on deep forests has opened up a new path for exploring deep models beyond neural networks. It combines the features of neural networks and ensemble learning, and has achieved good results in various fields. This paper mainly studies the application of deep forests to the field of fraud detection and proposes a distributed dense rotation deep forest algorithm (DRDF-spark) based on the improved RotBoost. The model has three main characteristics: firstly, it solves the problem of multi-granularity scanning due to the lack of spatial correlation in the data by introducing RotBoost. Secondly, Spark is used for parallel construction to improve the processing speed and efficiency of data. Thirdly, a pre-aggregation mechanism is added to the distributed algorithm to locally aggregate the statistical results of sub-forests in the same node in advance to improve communication efficiency. The experiments show that DRDF-spark performs better than deep forests and some mainstream ensemble learning algorithms on the fraud dataset in this paper, and the training speed is up to 3.53 times faster. Furthermore, if the number of nodes is further increased, the speedup ratio will continue to increase. [ABSTRACT FROM AUTHOR]

A very promising alternative fuel that can take the place of present fossil fuels is ammonia. Given that it is a hydrogen carrier with no carbon-based emissions and can be manufactured using renewable sources, it is a green fuel for Internal Compression Engine (ICE) with promising future applications. The notion of using pure ammonia in fueling combustion engines is not new, but despite numerous studies that looked into the possibility, Combustion engine fueling techniques based on ammonia are not yet ready for commercialization. This is because using this toxic molecule still presents significant challenges due to its supply, safety concerns, and physical differences from conventional fossil fuels. The objective of this review is to provide readers with a broad overview of the green ammonia research as a potential replacement fuel for ICE, as well as a look at potential future uses and workable solutions to related issues. The physicochemical characteristics of ammonia are discussed to determine the key benefits and drawbacks of its application in combustion engines, with an emphasis on the difficulties, advantages, and potentials for the future as well as recent technological advancements to get around the obstacles related to its production, storage, and utilization. Additionally, this work delves into sustainable approaches for the production of green ammonia and its application in spark ignition (SI) and compression ignition (CI) engines. Within this context, this study elucidates the foundational principles, research findings, practical considerations, economic implications, and prospective outlook towards a paradigm shift wherein ammonia assumes a significant role as a primary energy source in combustion engines. More study of ammonia combustion phenomena is necessary to propose solutions to these problems. This overview clarifies recent developments in ammonia technology from the point of manufacture to the point of consumption. [Display omitted] • A comprehensive review on NH 3 as a potential fuel in internal combustion engine (ICE). • Assessed physiochemical properties and different sustainable NH 3 production methods. • Spark ignition and compression ignition engines can work with NH 3 only or NH 3 dual-fuel. • Studies are required to ensured complete NH 3 combustion and to address NO x formation issues. [ABSTRACT FROM AUTHOR]

Wavelet neural networks (WNN) have been applied in many fields to solve regression as well as classification problems. After the advent of big data, as data gets generated at a brisk pace, it is imperative to analyze it as soon as it is generated owing to the fact that the nature of the data may change dramatically in short time intervals. This is necessitated by the fact that big data is all pervasive and throws computational challenges for data scientists. Therefore, in this paper, we built an efficient Scalable, Parallelized Wavelet Neural Network (SPWNN) which employs the parallel stochastic gradient algorithm (SGD) algorithm. SPWNN is designed and developed under both static and streaming environments in the horizontal parallelization framework. SPWNN is implemented by using Morlet and Gaussian functions as activation functions. This study is conducted on big datasets like gas sensor data which has more than 4 million samples and medical research data which has more than 10,000 features, which are high dimensional in nature. The experimental analysis indicates that in the static environment, SPWNN with Morlet activation function outperformed SPWNN with Gaussian on the classification datasets. However, in the case of regression, there is no clear trend was observed. In contrast, in the streaming environment i.e., Gaussian outperformed Morlet on the classification and Morlet outperformed Gaussian on the regression datasets. Overall, the proposed SPWNN architecture achieved a speedup of 1.22 - 1.78. [ABSTRACT FROM AUTHOR]

Developing and executing large-scale data analysis applications in parallel and distributed environments can be a complex and time-consuming task. Developers often find themselves diverted from their application logic to handle technical details about the underlying runtime and related issues. To simplify this process, ParSoDA, a Java library, has been proposed to facilitate the development of parallel data mining applications executed on HPC systems. It simplifies the process by providing built-in scalability mechanisms relying on the Hadoop and Spark frameworks. This paper presents ParSoDA-Py, the Python version of the ParSoDA library, which allows for further support of commonly used runtimes and libraries for big data analysis. After a complete library redesign, ParSoDA can be now easily integrated with other Python-based distributed runtimes for HPC systems, such as COMPSs and Apache Spark, and with the large ecosystem of Python-based data processing libraries. The paper discusses the adaptation process, which takes into consideration the new technical requirements, and evaluates both usability and scalability through some case study applications. [ABSTRACT FROM AUTHOR]

Typhoon trajectory related data involve many factors, such as atmospheric factors, oceanic factors, and physical factors. It has the characteristics of high dimension, strong spatio-temporal correlation, and nonlinear correlation, which increases the difficulty of typhoon trajectory prediction. Using feature selection approaches to select appropriate prediction factors becomes an important means to reduce the dimension of typhoon trajectory related data and improve the performance and accuracy of typhoon trajectory prediction methods. However, the existing feature selection methods based on linear correlation analysis cannot well depict the nonlinear correlation between data features, which results in low accuracy of feature selection. The feature selection methods based on nonlinear correlation analysis are computationally expensive, which affects the timeliness of feature selection. To solve the problem, we propose a parallel feature selection method NX-Spark-DC based on the Spark platform for typhoon trajectory related data. The method firstly filters out the redundant features of typhoon related data by normalized mutual information (NMI) method, subsequently eliminates the useless features by XGBoost machine learning model, and thus reducing the dimension of typhoon related data. On this basis, an improved Spark-based parallel distance correlation algorithm (Spark-DC) is proposed to select the feature combinations with strong correlation. A series of experimental results show that NX-Spark-DC method has high execution efficiency and accuracy, which is significantly better than the existing methods. [ABSTRACT FROM AUTHOR]

A sequential implementation of Bootstrap sampling and Bagging ensemble learning is computationally inefficient and not scalable to build large Bagging ensemble models with a large number of component models. Inspired by distributed big data computing, a new Bootstrap sample partition (BSP) big data model and a distributed ensemble learning method for large-scale distributed ensemble learning were proposed. The BSP data model extended a dataset as a set of Bootstrap samples stored in Hadoop distributed file system. Our distributed ensemble learning method randomly selected a subset of samples from the BSP data model and read them into Java virtual machines of the cluster. Following this, a serial algorithm was executed in each virtual machine to process each sample data and build a machine learningmodel on each sample data independently and in parallel with other virtual machines. Eventually, allsub-results were collected and processed in the master node to produce the ensemble result, optionally adding a sample preferences trategy for the BSP data blocks. The BSP data model generation and the component model building were computed using a non-MapReduce computing paradigm. All component models were computed in parallel without data communication among the nodes. The algorithms proposed in this paper were implemented in spark as internal operators that can be utilized in Spark applications. Experiments have demonstrated that BSP data model of a dataset can be generated efficiently through the new distributed algorithm. It improves the reusability of data samples and increases computational efficiency by over 50% in large-scale Bagging ensemble learning, while also increasing prediction accuracy by approximately 2%. [ABSTRACT FROM AUTHOR]

Floating car GPS data serves as the foundation for processing traffic information, with the rapid increase in the number of monitored vehicles, a massive amount of GPS data is generated, posing great challenges to map matching. To address the shortcomings of traditional matching methods in terms of matching efficiency and accuracy, this paper proposed a real time parallel map matching algorithm for massive GPS data that ensured both high matching accuracy and computational efficiency. This paper firstly reconstructed an efficient and accurate real-time map matching algorithm for streaming data by introducing a comprehensive weight factor that considered velocity and direction to enhance the offline map matching algorithm that relied on historical trajectories. Then, it introduced the Spark Streaming distributed computing framework to achieve real-time and parallel computation of the map matching algorithm, significantly improving the efficiency of real-time map matching. Experimental results demonstrate that the proposed algorithm achieves more than a 10% increase in matching accuracy compared to conventional topological matching algorithms on complex road sections, with an overall matching accuracy of over 95%. In terms of matching efficiency, it achieves approximately a fourfold improvement compared to an equivalent number of standalone servers. The experimental results show that the proposed algorithm achieves real-time map matching of 80 million GPS data points on a computing cluster composed of 11 machines, proving that the proposed algorithm can achieve real-time vehicle matching in urban areas. [ABSTRACT FROM AUTHOR]

In recent years, various studies have been conducted on SVMs and their applications in different area. They have been developed significantly in many areas. SVM is one of the most robust classification and regression algorithms that plays a significant role in pattern recognition. However, SVM has not been developed significantly in some areas like large-scale datasets, unbalanced datasets, and multiclass classification. Efficient SVM training in large-scale datasets is of great importance in the big data era. However, as the number of samples increases, the time and memory required to train SVM increase, making SVM impractical even for a medium-sized problem. With the emergence of big data, this problem becomes more significant. This paper presents a novel distributed method for SVM training in which a very small subset of training samples is used for classification, which reduces the problem size and thus the required memory and computational resources. The solution of this problem almost converges to standard SVM. This method includes three steps: first, detecting a subset of distributed training samples, second, creating local models of SVM and obtaining partial vectors, and finally combining the partial vectors and obtaining the global vector and the final model. In addition, the datasets which suffer from unbalanced number of samples and tend to the majority class, the proposed method balances the samples of the two classes and it can be used in unbalanced datasets. The empirical results show that using this method is efficient for large-scale problems. [ABSTRACT FROM AUTHOR]

The cooperative coevolution framework was introduced to address the shortcomings of metaheuristic algorithms in solving continuous large-scale global optimization problems. By dividing the problem into subcomponents which can be optimized separately, the framework can improve on both the solution's quality as well as the computational speed by exposing a degree of parallelism. Distributed computing platforms, such as Apache Spark, have long been used to improve the speed of different algorithms in solving computational problems. This work proposes a distributed implementation of the cooperative coevolution framework for solving large-scale global optimization problems on the Apache Spark distributed computing platform. By using a formerly outlined distributed variant of the cooperative coevolution framework, features of the Spark platform are utilized to enhance the computational speed of the algorithm while maintaining comparable search quality to other works in the literature. To test for the proposed implementation's improvement in computational speed, the CEC 2010 large-scale global optimization benchmark functions are used due to the diversity they offer in terms of complexity, separability and modality. Results of the proposed distributed implementation suggest that a speedup of up to ×3.36 is possible on large-scale global optimization benchmarks using the Apache Spark platform. [ABSTRACT FROM AUTHOR]

The large amount of data generated every day makes necessary the re-implementation of new methods capable of handle with massive data efficiently. This is the case of Association Rules, an unsupervised data mining tool capable of extracting information in the form of IF-THEN patterns. Although several methods have been proposed for the extraction of frequent itemsets (previous phase before mining association rules) in very large databases, the high computational cost and lack of memory remains a major problem to be solved when processing large data. Therefore, the aim of this paper is three fold: (1) to review existent algorithms for frequent itemset and association rule mining, (2)to develop new efficient frequent itemset Big Data algorithms using distributive computation, as well as a new association rule mining algorithm in Spark, and (3) to compare the proposed algorithms with the existent proposals varying the number of transactions and the number of items. To this purpose, we have used the Spark platform which has been demonstrated to outperform existing distributive algorithmic implementations. [ABSTRACT FROM AUTHOR]

With the use of cloud computing, which provides the infrastructure necessary for the efficient delivery of smart city services to every citizen over the internet, intelligent systems may be readily integrated into smart cities and communicate with one another. Any smart system at home, in a car, or in the workplace can be remotely controlled and directed by the individual at any time. Continuous cloud service availability is becoming a critical subscriber requirement within smart cities. However, these cost-cutting measures and service improvements will make smart city cloud networks more vulnerable and at risk. The primary function of Intrusion Detection Systems (IDS) has gotten increasingly challenging due to the enormous proliferation of data created in cloud networks of smart cities. To alleviate these concerns, we provide a framework for automatic, reliable, and uninterrupted cloud availability of services for the network data security of intelligent connected devices. This framework enables IDS to defend against security threats and to provide services that meet the users' Quality of Service (QoS) expectations. This study's intrusion detection solution for cloud network data from smart cities employed Spark and Waikato Environment for Knowledge Analysis (WEKA). WEKA and Spark are linked and made scalable and distributed. The Hadoop Distributed File System (HDFS) storage advantages are combined with WEKA's Knowledge flow for processing cloud network data for smart cities. Utilizing HDFS components, WEKA's machine learning algorithms receive cloud network data from smart cities. This research utilizes the wrapper-based Feature Selection (FS) approach for IDS, employing both the Pigeon Inspired Optimizer (PIO) and the Particle Swarm Optimization (PSO). For classifying the cloud network traffic of smart cities, the tree-based Stacking Ensemble Method (SEM) of J48, Random Forest (RF), and eXtreme Gradient Boosting (XGBoost) are applied. Performance evaluations of our system were conducted using the UNSW-NB15 and NSL-KDD datasets. Our technique is superior to previous works in terms of sensitivity, specificity, precision, false positive rate (FPR), accuracy, F1 Score, and Matthews correlation coefficient (MCC). [ABSTRACT FROM AUTHOR]

The Spark computing framework provides an efficient solution to address the major requirements of big data processing, but data partitioning and job scheduling in the Spark framework are the two major bottlenecks that limit Spark's performance. In the Spark Shuffle phase, the data skewing problem caused by unbalanced data partitioning leads to the problem of increased job completion time. In response to the above problems, a balanced partitioning strategy for intermediate data is proposed in this article, which considers the characteristics of intermediate data, establishes a data skewing model and proposes a dynamic partitioning algorithm. In Spark heterogeneous clusters, because of the differences in node performance and task requirements, the default task scheduling algorithm cannot complete scheduling efficiently, which leads to low system task processing efficiency. In order to deal with the above problems, an efficient job scheduling strategy is proposed in this article, which integrates node performance and task requirements, and proposes a task scheduling algorithm using greedy strategy. The experimental results prove that the dynamic partitioning algorithm for intermediate data proposed in this article effectively alleviates the problem that data skew leads to the decrease of system task processing efficiency and shortens the overall task completion time. The efficient job scheduling strategy proposed in this article can efficiently complete the job scheduling tasks under heterogeneous clusters, allocate jobs to nodes in a balanced manner, decrease the overall job completion time and increase the system resource utilization. [ABSTRACT FROM AUTHOR]

In recent years, technological advancements have been replicated in various industries, including sports medicine. Recent developments, such as big data analytics and data mining, which have revolutionized medical services in sports, are apparent in this transformation. This technological shift is motivated by the need to enhance athletic performance, prevent injuries, and offer individualized health advice. Modern lifestyles have simultaneously increased people's attention to their health, creating a demand for better medical services. However, China's ability to provide superior medical care needs to be improved due to a lack of medical resources and an ever-increasing patient population. To address these challenges, this research paper presents an integrated framework that leverages Spark-based big data analytics and the XGBoost algorithm. The framework aims to provide a robust sports medical service encompassing real-time health monitoring and data-driven insights. Powered by the formidable distributed computing platform Spark, it adeptly manages extensive sports data generated during training and events, facilitating instant health evaluations. Incorporating the XGBoost algorithm for data mining amplifies health prediction and recommendation capabilities. Renowned for its predictive prowess, XGBoost excels in discerning intricate sports data patterns and trends. Its proficiency in tackling intricates feature selection and modeling tasks ensures precision and actionable insights. Empirical findings underscore substantial enhancements in sports medical services. When applied to chronic disease datasets, the XGBoost algorithm garnered an impressive 93% trust rate. In contrast to conventional methods like K-Nearest Neighbors (KNN), Random Forest (RF), Decision Trees (DT), Support Vector Machines (SVM), Naïve Bayes (NB), and Logistic Regression (LR), the proposed framework consistently outperforms these established techniques. This remarkable performance underscores the transformative potential of the integrated framework in revolutionizing sports medical services. [ABSTRACT FROM AUTHOR]

The microwave diagnostics of discharges occurring in an artificial cloud of charged water droplets created in an open air simulating the environment of thunderclouds is implemented. An artificial cloud with a droplet size of about 1 µm is opaque in the visible range, so intra-cloud discharges are not available for investigation by traditional methods in the spark discharge physics based on the registration of visible discharge radiation. Microwaves pass through such a cloud without noticeable attenuation, they interact only with the plasma of discharges occurring in the cloud. The probing microwave radiation had a wavelength of 8 mm. The attenuation of microwaves passed through the cloud was measured with temporary resolution of about 10 ns. The temporal characteristics of intracloud discharges were investigated. [ABSTRACT FROM AUTHOR]

High altitude relight is a critical aspect of the aeronautical engine certification and may be addressed with the numerical simulation of two-phase ignition. However, such configurations are stiff and combined with local evaporation may lead to numerical issues. This paper provides several methods to perform two-phase ignition simulations using analytically reduced chemistry in the context of unstructured large Eddy simulation and Euler–Lagrange formalism. Firstly, an exponential formulation combined with a local and dynamic sub-cycling of the stiff chemistry is demonstrated to allow stable integration at the flow time step. Secondly, a particle-bursting method is applied to limit the impact of stiffness induced by the Lagrangian point-source approach in fine meshes. These methods are then applied in the simulation of ignition of a mono-disperse, multi-component kerosene spray in air. The use of the analytically reduced chemistry model enables us to describe in detail the chemical structure of the flame kernel during its formation. Moreover, local increase of fuel concentration is observed as the ignition proceeds which has a large influence on the combustion processes and the flame kernel development. [ABSTRACT FROM AUTHOR]

Apache Spark is a high-speed computing engine for processing massive data. With its widespread adoption, there is a growing need to analyze its correctness and temporal properties. However, there is scarce research focused on the verification of temporal properties in Spark programs. To address this gap, we employ the code-level runtime verification tool UMC4M based on the Modeling, Simulation, and Verification Language (MSVL). To this end, a Spark program S has to be translated into an MSVL program M, and the negation of the property P specified by a Propositional Projection Temporal Logic (PPTL) formula that needs to be verified is also translated to an MSVL program M1; then, a new MSVL program "M and M1" can be compiled and executed. Whether program S violates the property P is determined by the existence of an acceptable execution of "M and M1". Thus, the key issue lies in how to formalize model Spark programs using MSVL programs. We previously proposed a solution to this problem—using the MSVL functions to perform Resilient Distributed Datasets (RDD) operations and converting the Spark program into an MSVL program based on the Directed Acyclic Graph (DAG) of the Spark program. However, we only proposed this idea. Building upon this foundation, we implement the conversion from RDD operations to MSVL functions and propose, as well as implement, the rules for translating Spark programs to MSVL programs based on DAG. We confirm the feasibility of this approach and provide a viable method for verifying the temporal properties of Spark programs. Additionally, an automatic translation tool, S2M, is developed. Finally, a case study is presented to demonstrate this conversion process. [ABSTRACT FROM AUTHOR]

Miller and Urey applied electric sparks to a reducive mixture of CH4, NH3, and water to obtain a complex organic mixture including biomolecules. In this study, we examined the impact of temperature, initial pressure, ammonia concentration, and the spark generator on the chemical profile of a Miller–Urey-type prebiotic broth. We analyzed the broth composition using Gas Chromatography combined with Mass Spectroscopy (GC/MS). The results point towards strong compositional changes with the nature of the spark. Ammonia exhibited catalytic properties even with non-nitrogen-containing compounds. A more elevated temperature led to a higher variety of substances. We conclude that to reproduce such a broth as well as possible, all the studied parameters need to be tightly controlled, the most difficult and important being spark generation. [ABSTRACT FROM AUTHOR]

The stochastic nature of environmental factors that govern the behavior of fire, such as wind and fuel, exposes wildfire modeling to a degree of uncertainty. In order to produce more realistic wildfire predictions, it is, therefore, necessary to incorporate these uncertainties within wildfire models in a way that reflects the influence of environmental stochasticity on wildfire propagation. Otherwise, the risks of the potential danger of a given wildfire may be under-represented. Specifically, environmental stochasticity in the form of wind variability results in considerable uncertainty in the output of fire spread models. Here, we consider two stochastic wind models and their implementation in the spark fire simulator framework to capture the environmental uncertainty related to wind variability. The results are compared with the output from purely deterministic wildfire spread models and are discussed in the context of the potential ramifications for wildfire risk management. [ABSTRACT FROM AUTHOR]

Mosaicking is a crucial step in the application of remote sensing images. The amount of remote sensing image data has grown rapidly, along with the expansion of observed areas and increased image resolution. As a result, traditional serial mosaicking techniques are facing significant challenges. In recent times, various studies have utilized high-performance computing to hasten image mosaicking and attain favorable outcomes. Nevertheless, the current research only accelerates mosaicking through external technology, without optimizing from the perspective of algorithm flow, which introduces unnecessary data I/O and slows down the mosaicking. This paper introduces a rapid parallel remote sensing image mosaicking algorithm utilizing read filtering. To begin with, the target images are divided into blocks and stored in a distributed file system. Subsequently, the image blocks are read and filtered based on a designated input format. Finally, the overlapping and non-overlapping areas are read and processed asynchronously, reducing the data I/O and computing overhead, thereby improving the efficiency of parallel computing. The experiments indicate that the mosaicking algorithm introduced in this paper enhances throughput and speedup by an average of 1.38 MB/S and 0.87 relative to the current techniques, respectively, concerning various datasets and cores. This study provides a theoretical foundation and novel ideas for processing remote sensing images on cluster platforms. [ABSTRACT FROM AUTHOR]

Conventional micro-batch streaming systems introduce high tail latency for the following two reasons. First, they trigger data processing on a fixed time interval, without taking into account the data ingestion rate and data analysis processing rate. Second, in a multi-query environment, their scheduling methods do not consider the real-time characteristics of the application and each query. This paper presents zStream, a low latency micro-batch streaming system implemented on Apache Spark. The main idea of zStream's design is to use a reference value called deadline for each query to constrain latency around a specific time. Considering the deadline information of each query, zStream controls the admission of every micro-batch dynamically and schedules them by their priorities, significantly reducing the average and tail latency while maintaining throughput. Experimental results show that the tail latency of zStream is reduced by a maximum of 34% compared to the original Spark. Also, zStream reduces the average latency per query by up to 48% while stably maintaining throughput compared to the original Spark. [ABSTRACT FROM AUTHOR]

This paper proposed a parallel deep convolutional neural network optimization algorithm based on Spark and AMPSO (PDCNN-SAMPSO), aiming to address several issues encountered by parallel DCNN algorithms in big data environments, such as excessive redundant parameters, slow convergence speed, easy to fall into local optimal, and low parallel efficiency. Firstly, the algorithm designed a kernel pruning strategy based on importance and similarity (KP-IS) to address the problem of excessive redundant parameters by pruning the redundant convolution kernels in the model. Secondly, it proposed a model parallel training strategy based on adaptive mutation particle swarm optimization algorithm (MPT-AMPSO) to solve the slow convergence speed and easy to fall into local optimal issues of parallel DCNN algorithms by initializing the model parameters using adaptive mutation particle swarm optimization algorithm (AMPSO) . Finally, the algorithm proposed a dynamic load balancing strategy based on node performance (DLBNP) to balance the load of each node in the cluster and improve the parallel efficiency. Experiments show that, when using 8 computing nodes to process the CompCars dataset, the runtime of PDCNN-SAMPSO is 22%, 30%, 37% and 27% lower than that of Dis-CNN, DS-DCNN, CLR-Distributed-CNN and RS-DCNN, respectively, the speedup ratio is higher by 1.707, 1.424, 1.859, and 0.922, respectively, and the top-1 accuracy is higher by 4.01%, 4.89%, 2.42%, 5.94%, indicating that PDCNN-AMPSO has good classification performance in the big data environment and is suitable for parallel training of DCNN models in the big data environment. [ABSTRACT FROM AUTHOR]

This study introduces a novel methodology designed to assess the accuracy of data processing in the Lambda Architecture (LA), an advanced big-data framework qualified for processing streaming (data in motion) and batch (data at rest) data. Distinct from prior studies that have focused on hardware performance and scalability evaluations, our research uniquely targets the intricate aspects of data-processing accuracy within the various layers of LA. The salient contribution of this study lies in its empirical approach. For the first time, we provide empirical evidence that validates previously theoretical assertions about LA, which have remained largely unexamined due to LA's intricate design. Our methodology encompasses the evaluation of prospective technologies across all levels of LA, the examination of layer-specific design limitations, and the implementation of a uniform software development framework across multiple layers. Specifically, our methodology employs a unique set of metrics, including data latency and processing accuracy under various conditions, which serve as critical indicators of LA's accurate data-processing performance. Our findings compellingly illustrate LA's "eventual consistency". Despite potential transient inconsistencies during real-time processing in the Speed Layer (SL), the system ultimately converges to deliver precise and reliable results, as informed by the comprehensive computations of the Batch Layer (BL). This empirical validation not only confirms but also quantifies the claims posited by previous theoretical discourse, with our results indicating a 100% accuracy rate under various severe data-ingestion scenarios. We applied this methodology in a practical case study involving air/ground surveillance, a domain where data accuracy is paramount. This application demonstrates the effectiveness of the methodology using real-world data-intake scenarios, therefore distinguishing this study from hardware-centric evaluations. This study not only contributes to the existing body of knowledge on LA but also addresses a significant literature gap. By offering a novel, empirically supported methodology for testing LA, a methodology with potential applicability to other big-data architectures, this study sets a precedent for future research in this area, advancing beyond previous work that lacked empirical validation. [ABSTRACT FROM AUTHOR]

Ultrasonic frequency vibration coupled micro-wire electrical discharge machining (UFV- μ WEDM) has received enormous consideration due to its zero-tolerance machining. Nickel chromium (Ni–Cr) space alloys are a natural choice within the aerospace industry, which are exposed to high temperatures and high pressure, such as turbine seals and exhaust liners. This study reveals the impact of the UFV- μ WEDM influencing machining parameters like ultrasonic frequency vibration (UFV), servo voltage ( V S ), time on ( T on ), cutting angle ( A C ), time off ( T off ), and current (I) on the Ni–Cr space alloy in terms of minimum surface undulation (Ra) with maximum material removal rate ( M RR ). The cutting trials are carried out by central composite design (CCD). Analysis of variance (ANOVA) is used to find out the proportionate contribution of several factors, and it discloses that V S was the significant parameter impacting Ra (64.57%) and M RR (61.86%). The performance sequence of significant influencing parameters is V S > T off > A C > T on > I. According to desirability analysis (DA), optimum parameters for numerous solutions are T on = 8 μ s, V S = 5 0 V, T off = 1 4 μ s, I = 3 A, and A C = 3 0 ∘ . The optimum conditions lead to the highest M RR (5.72 mm3/min) and the lowest Ra (3.42 μ m). Scanning electron, 3D topography, and atomic force microscope images are used to analyze the machined surface. [ABSTRACT FROM AUTHOR]

Solving problems of high dimensionality (and complexity) usually needs the intense use of technologies, like parallelism, advanced computers and new types of algorithms. MapReduce (MR) is a computing paradigm long time existing in computer science that has been proposed in the last years for dealing with big data applications, though it could also be used for many other tasks. In this article, we address big optimization: the solution to large instances of combinatorial optimization problems by using MR as the paradigm to design solvers that allow transparent runs on a varied number of computers that collaborate to find the problem solution. We study and analyze the MR technology, focusing on Hadoop, Spark, and MPI as the middleware platforms to develop genetic algorithms (GAs). From this, MRGA solvers arise using a different programming paradigm from the usual imperative transformational programming. Our objective is to confirm the expected benefits of these systems, namely file, memory, and communication management, over the resulting algorithms. We analyze our MRGA solvers from relevant points of view like scalability, speedup, and communication vs. computation time in big optimization. The results for high-dimensional datasets show that the MRGA over Hadoop outperforms the implementations in Spark and MPI frameworks. For the smallest datasets, the execution of MRGA on MPI is always faster than the executions of the remaining MRGAs. Finally, the MRGA over Spark presents the lowest communication times. Numerical and time insights are given in our work, so as to ease future comparisons of new algorithms over these three popular technologies. [ABSTRACT FROM AUTHOR]

The current main trajectory similarity connection methods are based on GPS trajectories. Optimization methods for GPS trajectories cannot be directly applied to the problem of connecting subway passenger trajectories. By fully utilizing the spatiotemporal characteristics of subway passenger trajectories and leveraging the trajectory􀆳s repetitiveness and symmetry, the trajectory is transformed from a point sequence to an OD sequence to reduce the trajectory length and save storage space. This paper focuses on the design and implementation of the trajectory connection algorithm based on PPJoin+ on the Spark platform. The method is validated on a 13-node Spark cluster and a large-scale dataset containing 5 million passenger trajectories (560 million tap records collected in two consecutive months). The experimental results show that the PPJoin+ algorithm based on OD sequence only takes 14.0 minutes, which saves 62.5% of the execution time compared to the default point sequence trajectory connection algorithm and 78.2% of the execution time compared to the Dima connection algorithm, and exhibits good scalability. [ABSTRACT FROM AUTHOR]

Support Vector Machine (SVM) is a machine learning with excellent classification performance, which has been widely used in various fields such as data mining, text classification, face recognition and etc. However, when data volume scales to a certain level, the computational time becomes too long and the efficiency becomes low. To address this issue, we propose a parallel balanced SVM algorithm based on Spark, named PB-SVM, which is optimized on the basis of the traditional Cascade SVM algorithm. PB-SVM contains three parts, i.e., Clustering Equal Division, Balancing Shuffle and Iteration Termination, which solves the problems of data skew of Cascade SVM and the large difference between local support vector and global support vector. We implement PB-SVM in AliCloud Spark distributed cluster with five kinds of public datasets. Our experimental results show that in the two-classification test on the dataset covtype, compared with MLlib-SVM and Cascade SVM on Spark, PB-SVM improves efficiency by 38.9% and 75.4%, and the accuracy is improved by 7.16% and 8.38%. Moreover, in the multi-classification test, compared with Cascade SVM on Spark on the dataset covtype, PB-SVM improves efficiency and accuracy by 94.8% and 18.26% respectively. [ABSTRACT FROM AUTHOR]

In this paper, we introduce MARVEL, a system designed to simplify the teaching of MapReduce, a popular distributed programming paradigm, through software visualization. At its core, it allows a teacher to describe and recreate a MapReduce application by interactively requesting, through a graphical interface, the execution of a sequence of MapReduce transformations that target an input dataset. Then, the execution of each operation is illustrated on the screen by playing an appropriate graphical animation stage, highlighting aspects related to its distributed nature. The sequence of all animation stages, played back one after the other in a sequential order, results in a visualization of the whole algorithm. The content of the resulting visualization is not simulated or fictitious, but reflects the real behavior of the requested operations, thanks to the adoption of an architecture based on a real instance of a distributed system running on Apache Spark. On the teacher's side, it is expected that by using MARVEL he/she will spend less time preparing materials and will be able to design a more interactive lesson than with electronic slides or a whiteboard. To test the effectiveness of the proposed approach on the learner side, we also conducted a small scientific experiment with a class of volunteer students who formed a control group. The results are encouraging, showing that the use of software visualization guarantees students a learning experience at least equivalent to that of conventional approaches. [ABSTRACT FROM AUTHOR]

In the modern era, digital data processing with a huge volume of data from the repository is challenging due to various data formats and the extraction techniques available. The accuracy levels and speed of the data processing on larger networks using modern tools have limitations for getting quick results. The major problem of data extraction on the repository is finding the data location and the dynamic changes in the existing data. Even though many researchers created different tools with algorithms for processing those data from the warehouse, it has not given accurate results and gives low latency. This output is due to a larger network of batch processing. The performance of the database scalability has to be tuned with the powerful distributed framework and programming languages for the latest real-time applications to process the huge datasets over the network. Data processing has been done in big data analytics using the modern tools HADOOP and SPARK effectively. Moreover, a recent programming language such as Python will provide solutions with the concepts of map reduction and erasure coding. But it has some challenges and limitations on a huge dataset at network clusters. This review paper deals with Hadoop and Spark features also their challenges and limitations over different criteria such as file size, file formats, and scheduling techniques. In this paper, a detailed survey of the challenges and limitations that occurred during the processing phase in big data analytics was discussed and provided solutions to that by selecting the languages and techniques using modern tools. This paper gives solutions to the research people who are working in big data analytics, for improving the speed of data processing with a proper algorithm over digital data in huge repositories. [ABSTRACT FROM AUTHOR]

A consistent theme in trace analytical atomic spectroscopy is the search for methods of efficient atomization and excitation of sample matrices in solid form – so-called direct analysis. The technique reported here – the sliding spark – is a special pulsed plasma specularly discharging on a dielectric surface enforced between a pair of electrodes in air at atmospheric pressure. Although the sliding spark is characterized by complex emission spectra with very few lines that are optically thin and interference-free, the plasma has potential for spectroanalysis of dielectric materials as the excited species in the plasma which are at low concentration compared to those in the ground or metastable states may be quantified via their spectral line intensities. The challenge is how to conceptualize an analytical model from first principles and develop from it a calibration strategy for trace quantitative analysis in a form that can be expressed mathematically. We describe the sliding spark technique and the utility of halide matrix modifiers to increase sensitivity, as well as the feasibility of developing, for selected spectral lines, a conceptual scheme with semi-empirical models for trace quantitative spectroscopy of heavy metals spiked in pelletized dielectric samples. The proposed quantitative models demonstrate that the sliding spark plasma spectra are useful for simple, affordable, and rapid direct trace spectroscopy of dielectrics, a ubiquitous theme in industrial, environmental and medical analyses. [ABSTRACT FROM AUTHOR]

In recent years, with the widespread use of location-aware handheld devices and the development of wireless networks, trajectory data have shown a trend of rapid growth in data volume and coverage, which has led to the prosperous development of location-based services (LBS). Spatio-temporal range query, as the basis of many services, remains a challenge in supporting efficient analysis and calculation of data, especially when large volumes of trajectory data have been accumulated. We propose a Hilbert-GeoSOT spatio-temporal meshing and coding method called HGST to improve the efficiency of spatio-temporal range queries on massive trajectory data. First, the method uses Hilbert to encode the grids obtained based on the GeoSOT space division model, and then constructs a unified time division standard to generate the space–time location identification of trajectory data. Second, this paper builds a novel spatio-temporal index to organize trajectory data, and designs an adaptive spatio-temporal scaling and coding method based on HGST to improve the query performance on indexed records. Finally, we implement a prototype system based on HBase and Spark, and develop a Spark-based algorithm to accelerate the spatio-temporal range query for huge trajectory data. Extensive experiments on a real taxi trajectory dataset demonstrate that HGST improves query efficiency levels by approximately 14.77% and 34.93% compared with GeoSOT-ST and GeoMesa at various spatial scales, respectively, and has better scalability under different data volumes. [ABSTRACT FROM AUTHOR]

The advent of the era of big remote sensing data has transformed traditional data management and analysis models, among which visualization analysis has gradually become an effective method, and map tiles for remote sensing data have always played an important role. However, in high-latitude regions, especially in polar regions, the deformation caused by map projection still exists, which lowers the accuracy of global or large-scale visual analysis, as well as the execution efficiency of big data. To solve the above problems, this paper proposes an algorithm called HexTile, which uses a hexagonal discrete global grid system (DGGS) model to effectively avoid problems caused by map projection and ensure global consistency. At the same time, the algorithm was implemented based on the Spark platform, which also has advantages in efficiency. Based on the DGGS model, hierarchical hexagon map tile construction and a visualization algorithm were designed, including hexagonal slicing, merging, and stitching. The above algorithms were parallelized in Spark to improve the big data execution efficiency. Experiments were carried out with Landsat-8, and the results show that the HexTile algorithm can not only guarantee the quality of global data, but also give full play to the advantages of the cluster in terms of efficiency. Additionally, the visualization was conducted with Cesium and OpenLayers to validate the integration and completeness of hexagon tiles. The scheme proposed in this paper could provide a reference for spatiotemporal big data visualization technology. [ABSTRACT FROM AUTHOR]

Viewshed analysis is a terrain visibility computation method based on the digital elevation model (DEM). With the rapid growth of remote sensing and data collection technologies, the volume of large-scale raster DEM data has reached a great size (ZB). However, the data storage and GIS analysis based on such large-scale digital data volume become extra difficult. The usually distributed approaches based on Apache Hadoop and Spark can efficiently handle the viewshed analysis computation of large-scale DEM data, but there are still bottleneck and precision problems. In this article, we present a multi-level distributed XDraw (ML-XDraw) algorithm with Apache Spark to handle the viewshed analysis of large DEM data. The ML-XDraw algorithm mainly consists of 3 parts: (1) designing the XDraw algorithm into a multi-level distributed computing process, (2) introducing a multi-level data decomposition strategy to solve the calculating bottleneck problem of the cluster's executor, and (3) proposing a boundary approximate calculation strategy to solve the precision loss problem in calculation near the boundary. Experiments show that the ML-XDraw algorithm adequately addresses the above problems and achieves better speed-up and accuracy as the volume of raster DEM data increases drastically. [ABSTRACT FROM AUTHOR]

In response to the rapid growth of many sorts of information, highway data has continued to evolve in the direction of big data in terms of scale, type, and structure, exhibiting characteristics of multi-source heterogeneous data. The k-nearest neighbor (KNN) join has received a lot of interest in recent years due to its wide range of applications. Processing KNN joins is time-consuming and inefficient due to the quadratic structure of the join method. As the number of applications dealing with vast amounts of data develops, KNN joins get more sophisticated. The authors seek to save money on computer resources by leveraging a large number of threads and multiprocessors. Six popular datasets are used to apply the method and evaluate the sequential and parallel performance of the KNN technique. These datasets are used to compare the sequential and parallel performance of the KNN method. When compared to a matching multi-core solution, the final implementation saves computing resources. It has been optimized to utilize as little RAM as possible, allowing it to manage high-resolution photo data without sacrificing efficiency. The authors will use the technique they presented using Spark Radoop. Our performance research validates the supplied method's efficacy and scalability. [ABSTRACT FROM AUTHOR]

Big Data is a phenomenon that affects today's world, with new data being generated every second. Today's enterprises face major challenges from the increasingly diverse data, as well as from indexing, searching, and analyzing such enormous amounts of data. In this context, several frameworks and libraries for processing and analyzing Big Data exist. Among those frameworks Hadoop MapReduce, Mahout, Spark, and MLlib appear to be the most popular, although it is unclear which of them best suits and performs in various data processing and analysis scenarios. This paper proposes EverAnalyzer, a self-adjustable Big Data management platform built to fill this gap by exploiting all of these frameworks. The platform is able to collect data both in a streaming and in a batch manner, utilizing the metadata obtained from its users' processing and analytical processes applied to the collected data. Based on this metadata, the platform recommends the optimum framework for the data processing/analytical activities that the users aim to execute. To verify the platform's efficiency, numerous experiments were carried out using 30 diverse datasets related to various diseases. The results revealed that EverAnalyzer correctly suggested the optimum framework in 80% of the cases, indicating that the platform made the best selections in the majority of the experiments. [ABSTRACT FROM AUTHOR]

Provider referral is one of the most influential factors in research recruitment. To ease referral burden on providers, we adapted the Best Practice Alert (BPA) in the EPIC Electronic Health Record and assessed its utility in recruiting pediatric patients with autism spectrum disorder for the national SPARK study. During a year-long surveillance, 1203 (64.0%) patients were Interested in SPARK and 223 enrolled. Another 754 participants not recruited via the BPA also enrolled; 35.5% of these participants completed their participation compared to 58.3% of BPA-referred participants. Results suggest that (a) a BPA can successfully engage providers in the study-referral process and (b) families who learn about research through their providers may be more engaged and effectively retained. [ABSTRACT FROM AUTHOR]

With the arrival of the current digital era and the advancement of information transmission technologies, there has been an unprecedented rise in data. Efficient extraction of useful information from the volumes of data has garnered growing interest from academics and the industry. Data mining research focuses on finding utility patterns in large datasets. But the inherent complications like frequent scans, creation of substantial candidate sets, etc. plague the mining process for large datasets. Distributive architecture-based approaches also prove inefficacious due to high communication overhead over iterations. High communication cost over data exchange both locally and remotely further aggravates the situation. We propose a Communication Cost Effective Utility-based Pattern Mining (CEUPM) algorithm based on the Spark framework to address this issue. Spark accelerates iterative scanning by storing scanned datasets in a memory abstraction called resilient distributed datasets (RDD). RDD operations need a redistribution of data among cluster nodes during processing. To minimize the communication cost incurred during the shuffle process, we adopt a search space division strategy based on data parallelism for a fair and effective task allocation across cluster nodes. Communication overhead is incurred during this redistribution or shuffle process while minimizing costs. Experimental results in four real datasets demonstrate that CEUPM considerably reduces shuffling overhead and outperforms other existing methods in terms of memory usage, communication cost, execution time, and scalability. [ABSTRACT FROM AUTHOR]

On the set of frames in a finite-dimensional space, we introduce the widest possible equivalence preserving the main characteristics of frames, that is, tightness, equiangularity, and spark (the least number of linearly dependent vectors). This equivalence is known as projective–permutational unitary equivalence. For example, we show that full spark equiangular tight frames in the spaces , , and are unique up to equivalence. A similar uniqueness result is obtained for the general uniform Parseval frame of vectors in the space . Related questions have been raised in the literature several times. Calculating the spark is computationally much harder than calculating the rank of a matrix. Here we present an algorithm that can possibly simplify the spark calculation. The use of Seidel matrices and the Naimark complement technique proves to be very useful in the classification of frames up to equivalence. [ABSTRACT FROM AUTHOR]

As software becomes more complex, it becomes more difficult to verify its correctness. This poses a particular problem for autonomous systems, since they are software-intensive and will also require strong evidence of correctness in order to be allowed to operate in the real world. One way to help address this problem is through the use of formal methods, i.e. mathematicallybased tools for software and hardware verification. In this paper, we perform formal program verification on a service in OpenUxAS, a free and open source software framework for mission-level, multi-vehicle autonomy. More specifically, we apply the SPARK language and verification toolset to a service that sanity-checks and segments long sequences of vehicle waypoints to prove that it is free of runtime errors. [ABSTRACT FROM AUTHOR]

Nowadays, Apache Hadoop and Apache Spark are two of the most prominent distributed solutions for processing big data applications on the market. Since in many cases these frameworks are adopted to support business critical activities, it is often important to predict with fair confidence the execution time of submitted applications, for instance when service-level agreements are established with end-users. In this work, we propose and validate a hybrid approach for the performance prediction of big data applications running on clouds, which exploits both analytical modeling and machine learning (ML) techniques and it is able to achieve a good accuracy without too many time consuming and costly experiments on a real setup. The experimental results show how the proposed approach attains improvement in accuracy, number of experiments to be run on the operational system and cost over applying ML techniques without any support from analytical models. Moreover, we compare our approach with Ernest , an ML-based technique proposed in the literature by the Spark inventors. Experiments show that Ernest can accurately estimate the performance in interpolating scenarios while it fails to predict the performance when configurations with increasing number of cores are considered. Finally, a comparison with a similar hybrid approach proposed in the literature demonstrates how our approach significantly reduce prediction errors especially when few experiments on the real system are performed. [ABSTRACT FROM AUTHOR]

Companies, institutions or governments process large amounts of data for the development of their activities. This knowledge usually comes from devices that collect data from various sources. Processing them in real time is essential to ensure the flow of information about the current state of infrastructure, as this knowledge is the basis for management and decision making in the event of an attack or anomalous situations. Therefore, this article exposes three unsupervised machine learning models based on clustering techniques and threshold definitions to detect anomalies from heterogeneous streaming cybersecurity data sources. After evaluation, this paper presents a case of heterogeneous cybersecurity devices, comparing WSSSE, Silhouette and training time metrics for all models, where K-Means was defined as the optimal algorithm for anomaly detection in streaming data processing. The anomaly detection's accuracy achieved is also significantly high. A comparison with other research studies is also performed, against which the proposed method proved its strong points. [ABSTRACT FROM AUTHOR]

It is hard to forecast waiting time from mobile trajectory big data on the traditional centralized mining platform, and especially the taxi driving direction cannot be clearly distinguished by the GPS trajectories of taxicabs in intelligent transportation systems. To this end, we propose a direction identification method (named CA-D) combined with Coordinate Axis and GPS Direction to distinguish taxi driving direction and then establish a distributed model (named EMDN-GRU) on Spark to forecast passenger waiting time based on an Empirical Mode Decomposition (EMD) algorithm with Normalization and a Gated Recurrent Unit (GRU) model. Specifically, in the process of waiting time forecasting, the CA-D method is used to differentiate directions to reduce the data interference caused by different taxi driving directions. Furthermore, the EMD algorithm with normalization is utilized to process large-scale GPS trajectory data. Finally, the GRU model with adjusted parameters is employed to forecast the time-series data obtained in the previous step, and the forecasting results are denormalized and superimposed to produce waiting time for passengers. Compared with LSTM, GRU, EMD-LSTM, EMD-GRU, CNN, and BP, the experimental results from a case study indicate that EMDN-GRU is significantly superior to others. In particular, from three data sets of weekday, weekend, and one week, the MAPE values of EMDN-GRU are reduced by 92.48%, 95.01%, and 90.47% at most. [ABSTRACT FROM AUTHOR]

Graph mining contributes a key role in data mining and as the size of the data increases, it becomes complicated. Identifying the interesting subgraphs in the graph is a commonly researched issue, where the subgraphs denote the commonly occurring pattern exhibiting a particular structure. Frequent Subgraph Mining (FSM) is an important task for exploratory data analysis on graph data. Though many techniques are proposed for FSM, the large dimension of the data makes FSM complex. This research proposes a novel technique for performing FSM in a Hadoop framework. Here, FSM is carried out using the proposed Holoentropy Gaston algorithm (HE-Gaston), which is developed by incorporating the Holoentropy support measure instead of the Recurrent support measure in the Recurrent-Gaston (R-Gaston) technique. Here, the weblog files are considered for FSM and are fed to the Spark framework, which encompasses a master and several slaves. The slave nodes generate the frequent subgraphs based on the Holoentropy support measure and the generated frequent subgraphs are applied to the master node which produces the final frequent subgraphs by utilizing an aggregate Holoentropy support measure. Further, the HE-Gaston shows that it recorded execution time of 43 ms, memory of 2.161 MB, and number of structures mined as 117. [ABSTRACT FROM AUTHOR]

As a popular spatial operation, the k-Nearest Neighbors (kNN) query is widely used in various spatial application systems. How to efficiently process a kNN query on spatial big data has always been an important research topic in the field of spatial data management. The centralized solutions are not suitable for spatial big data due to their poor scalability, while the existing distributed solutions are not efficient enough to meet the high real-time requirements of some spatial applications. Therefore, we introduce the Proportional Integral Derivative (PID) control technology into kNN query processing and propose a PID-based kNN query processing algorithm (PIDKNN) for spatial big data based on Spark. In this algorithm, the whole data space is divided into grid cells of the same size using the grid partition method, and the grid-based index is constructed. On this basis, the grid-based density peak clustering algorithm is used to cluster spatial data, and the corresponding PID parameters are set for each cluster. When performing kNN queries, the PID algorithm is used to estimate the radius growth step size of kNN queries, thereby realizing kNN query processing with a variable query radius growth step based on a feedback mechanism, which greatly improves the efficiency of kNN query processing. A series of experimental results show that the PIDKNN algorithm has good performance and scalability and is superior to the existing parallel kNN query processing methods. [ABSTRACT FROM AUTHOR]

With the widespread popularity of the mobile Internet, the domestic online game market has become increasingly saturated. The cost of acquiring new users for game companies continues to increase. How to prevent the loss of existing users has become the focus of marketing. This paper predicts user churn based on a real game log data. First, user features are extracted and computed from log data. Second, a set of important features is selected by weight. Finally, a binary classification model is constructed with features as input and churn as output. 6 common algorithms such as random forest, support vector machine, multi-layer perceptron, gradient boosting decision tree, and logistic regression are comprehensively compared. The experimental results show that the random forest algorithm performs the best, and its model prediction accuracy reaches 91%. [ABSTRACT FROM AUTHOR]

Since Ada targets safety-critical programs, many features of the language introduce safety nets in the form of language-mandated checks. Even if compile-time verification is preferred to runtime verification whenever possible, many of these checks are still done dynamically, an exception being raised in case of violation. The addition of contracts in Ada 2012 follows a similar trend, as a violation causes an exception to be raised when the code is compiled with assertions enabled. The SPARK tool aims at statically verifying that language-mandated checks and the user-written contracts can never fail at runtime. In this article, we give insights on how the tool works in practice and what are the most important challenges as of today. [ABSTRACT FROM AUTHOR]

The rapidly increasing complexity and scale of optimization problems pose challenges to search ability and performance of traditional evolutionary algorithms which could be only executed sequentially without scalability and are difficult to obtain an ideal solution in a reasonable time. In this paper, Spark-ITGO, a parallel and scalable invasive tumor growth optimization algorithm on Spark, is presented based on the serial invasive tumor growth optimization (ITGO). In Spark-ITGO, a parallel multiple-tumor evolution model is proposed to search the optimal solution of problems. A balanced multi-island optimal migration strategy is designed to increase diversity of population and prevent converging into a local optimum. Additionally, a universal parallel evolutionary algorithm framework is implemented based on resilient distributed dataset (RDD) and central broadcast mechanism. Spark-ITGO is evaluated on benchmark experiments of CEC2013 and CEC2010 LSGO and the results show that it achieves great scalability and performs better than other evolutionary algorithms. [ABSTRACT FROM AUTHOR]