Search

Purpose: To examine the contextual and personal factors that influence teachers' reported adoption of one or more instructional models (IMs). Methods: Participants (n = 25) were interviewed that reported adopting Sport Education; Teaching Games for Understanding; Teaching Personal and Social Responsibility; and/or Sports, Play, and Active Recreation for Kids in their K-12 curriculum. Results: Viewed through the lens of occupational socialization theory and utilizing constant comparison methodology, the interview data were analyzed and three major themes, each with related subthemes, were developed: (a) IM adoption depends on context and socialization, (b) socializing agents influence IM adherence, and (c) IM implementation is selective. Discussion/Conclusions: Teachers vary in their rationale for adopting one or more IMs, and adherence is frequently a result of principal and student encouragement. Future efforts to disseminate IMs should consider pathways to adoption identified in this study, including teaching conventions, department supervisors, and teaching colleagues. [ABSTRACT FROM AUTHOR]

Aiming at the problem that the Spark scalable distributed platform does not consider the computing capabilities of heterogeneous cluster nodes and load balance during job task scheduling, which affects the system performance, this paper constructs heterogeneous cluster nodes load balance scheduling policy under the Spark environment, Heterogeneous cluster node predicts the data distribution characteristics according to the sampling algorithm. divides the data into balancing partitions. According to the static load and dynamic load weight distribution, heterogeneous cluster node obtains the real-time load, and dynamically schedules job tasks. Finally. Wordcount, TeraSort, and K-means three benchmark tests were used to compare and analyze during heterogeneous cluster operation. Experimental results show that this algorithm can reduce the execution time significantly, and improve the performance of heterogeneous cluster. [ABSTRACT FROM AUTHOR]

In recent years, cloud computing applications have facilitated the distribution of heterogeneous, unstructured digitized data among users' social networks of varying opinions. Text processing at a large scale requires high-precision computational techniques, which increases the computational burden. The advent of big data analytics along with Natural Language Processing is a powerful factor in improving the efficiency of processing large-scale text data, using the kernel of the MapReduce big data analytics frameworks that allow parallelization of large computational operations. In this paper, we propose an intelligent sentiment prediction approach based on deep learning, batch, and streaming big data analytics. In fact, our main objective is to take advantages of the powerful tools provided by the distributed platforms, such as, Hadoop and Spark to preprocess streaming data. This involves various tasks such as cleaning the data, reducing its size, minimizing access time, and decreasing storage volume. This step prepares the big streaming data to be fully exploited by Deep Learning models. This work includes a research study on processing big data related to short volume scripts based on batch and streaming distributed frameworks as well as deep learning approaches in Natural Language Processing. We detail our idea for analyzing short texts to determine their semantic context and categorize them into pros and cons poles. There were different stages in building this model, the first involving data reduction and refinement using selected features and big data analysis tools. In the second stage, words are embedded by global vector to be computed in layers of convolutional and recurrent neural networks. The experimental study and the analysis of the results confirm the usefulness of our proposed model and its superiority over the main approaches studied in the literature. Our model achieved a performance of 96% in terms of accuracy. [ABSTRACT FROM AUTHOR]

In this study, the duplex stainless steel is machined using a wire-cut electrical discharge machining process to overcome the issues of poor finishing, lower dimensional accuracy, and rapid wear of cutting tools in conventional machining processes. The higher rate of material removal and superior finish on the machine surface is dependent on kerf loss, which is expressed as the mean kerf width assessed at certain locations. Nevertheless, this approach is typically not effective in explicating the intricacies of true kerf loss due to the random selection of distinctive points. This work proposes an area-based measuring approach to quantify the kerf loss by taking into account any imperfections on the kerf wall. At the pulse on time of 20 μs and current of 3 A, the machining appears not stable. The pulse on time, current, and voltage of 12 µs, 1 A, and 80 V proved suitable for obtaining excellent dimensional precision. [ABSTRACT FROM AUTHOR]

Early detection and diagnosis of breast cancer are critical for saving lives. This paper addresses two major challenges associated with this task: the vast amount of data processing involved and the need for early detection of breast cancer. To tackle these issues, we developed thirty hybrid architectures by combining five deep learning techniques (Xception, Inception-V3, ResNet50, VGG16, VGG19) as feature extractors and six classifiers (random forest, logistic regression, naive Bayes, gradient-boosted tree, decision tree, and support vector machine) implemented on the Spark framework. We evaluated the performance of these architectures using four classification criteria. The results, analyzed using Scott Knott's statistical test, demonstrated the effectiveness of merging deep learning feature extraction techniques with traditional classifiers for classifying breast cancer into malignant and benign tumors. Notably, the hybrid architecture using logistic regression as the classifier and ResNet50 for feature extraction (RESLR) emerged as the top performer. It achieved impressive accuracy scores of 98.20%, 96.59%, 96.64%, and 94.84% across the Break-His dataset at different magnifications (40X, 100X, 200X, and 400X) respectively. Additionally, RESLR achieved an accuracy of 97.05% on the ICIAR dataset and a remarkable accuracy of 95.31% on the FNAC dataset. [ABSTRACT FROM AUTHOR]

Copyright of Retos: Nuevas Perspectivas de Educación Física, Deporte y Recreación is the property of Federacion Espanola de Asociaciones de Docentes de Educacion Fisica and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

The commonly used method for network intrusion is based on pattern matching systems, but these systems do not have high detection accuracy when facing large-scale high-speed network traffic environments. In addition, when facing a wide variety of network attacks, relying solely on a certain network security technology is difficult to ensure network security. Therefore, a distributed network intrusion prevention system based on the Spark framework and dynamic information security theory model was innovatively proposed to improve the detection efficiency of network intrusion and solve these issues. The architecture and functional structure of the network intrusion prevention system were constructed by improving the random forest algorithm and Spark. In addition, the dynamic network intrusion prevention system was designed on the basis of the Policy Protection Detection Response (P2DR) model and the Protection Detection Reaction Recovery (PDRR) model to cope with the diverse network attacks and make up for the lack of dynamic defense based on improved forest algorithm and Spark. The test results showed that the network intrusion prevention system based on the improved random forest algorithm and Spark had a maximum detection time of 13,593 ms, a minimum detection time of 13,318 ms, and an average detection time of 13,468 ms when the data were 6000 pieces. The average success rate of the dynamic network intrusion prevention system based on the dynamic information security theory model was 87.72%, the average detection rate was 71.68%, and the average false alarm rate was 17.23%. The F1 values corresponding to the improved random forest algorithm under 7 different attack types of data were 0.985, 0.983, 0.876, 0.843, 0.797, 0.983, and 0.890, respectively, which were significantly better than the comparison algorithm. It can be seen that the dynamic network intrusion prevention system based on the improved random forest algorithm, Spark, and the dynamic information security theoretical model, has good performance and can effectively detect network intrusions, providing technical support for solving network intrusion problems in reality. The contribution of the research is reflected in the improvement of the detection performance of network intrusion detection systems and the reduction of detection time and false alarm rates. [ABSTRACT FROM AUTHOR]

The performance optimization of Apache Spark, a widely used distributed computing framework, is crucial for the efficient execution of data-intensive workloads. However, the automatic tuning of around 180 internal configuration features of Spark for getting optimum performance can be a complex, expensive, and time-consuming task. This research paper aims to address these challenges along with the task of reducing execution time and tuning cost for benchmark Spark MLlib workloads on a benchmark dataset by automating the tuning process. A grid search based novel approach Reduced Internal Configuration Grid (RICG) is proposed that reduces the Spark's configuration feature values grid levaraging exponential projective sampling. A significant reduction in Spark workload execution time is obtained at a low cost without compromising the performance of the CPU, memory and network resources of the system. Extensive experimentation and evaluation demonstrate the effectiveness of the proposed approach, showcasing notable reductions of around 26–30% in execution time for benchmark workloads. This reduction is achieved by cutting the features tuning cost by approximately 58 to 60% compared to the prior best grid search based approach. The proposed RICG model outperforms existing statistical trial-test approaches, striking a favorable balance between performance optimization and cost optimization. Lastly, the logs generated by the RICG model are used to predict the performance of Spark application workloads by using the boosting algorithms with around 90% accuracy. The findings of this research highlight the potential of automated tuning techniques to enhance the performance of Apache Spark-based applications with cost efficiency. [ABSTRACT FROM AUTHOR]

传统Bootstrap抽样和Bagging集成学习通常以串行方式实现，计算效率低，且存在样本不可重用、扩展性差等问题，不适合高效的大规模Bagging集成学习。从大数据分布式计算的思维入手，提出新的Bootstrap样本划分（BSP）大数据模型和分布式集成学习方法。BSP数据模型通过分布式生成算法将训练数据表达成分布式Bootstrap样本集的集合，存储成HDFS分布式数据文件，为后续的分布式集成学习提供数据支持。分布式集成学习方法从BSP数据模型中随机选取多个BSP数据块，读入集群各个节点的虚拟机，用串行算法对选取的数据块并行计算统计量或训练建模，再将所有的计算子结果回传至主节点中，生成最终的集成学习结果，此过程中可加入对子结果的质量选择以进一步提高预测效果。BSP数据模型的生成和分布式集成学习采用非Map-Reduce计算范式进行，每个数据块的计算独立完成，减少了计算节点间的数据通信开销。提出的算法在Spark开源系统中以新的算子方式实现，供Spark应用程序调用。实验表明，新方法可以高效地生成训练数据的BSP数据模型，提高数据样本的可重用性，在基于有监督机器学习算法构建的大规模Bagging集成学习实验中，计算效率能提高50%以上，同时预测精度进一步提高约2%。

Credit fraud is a common financial crime that causes significant economic losses to financial institutions. To address this issue, researchers have proposed various fraud detection methods. Recently, research on deep forests has opened up a new path for exploring deep models beyond neural networks. It combines the features of neural networks and ensemble learning, and has achieved good results in various fields. This paper mainly studies the application of deep forests to the field of fraud detection and proposes a distributed dense rotation deep forest algorithm (DRDF-spark) based on the improved RotBoost. The model has three main characteristics: firstly, it solves the problem of multi-granularity scanning due to the lack of spatial correlation in the data by introducing RotBoost. Secondly, Spark is used for parallel construction to improve the processing speed and efficiency of data. Thirdly, a pre-aggregation mechanism is added to the distributed algorithm to locally aggregate the statistical results of sub-forests in the same node in advance to improve communication efficiency. The experiments show that DRDF-spark performs better than deep forests and some mainstream ensemble learning algorithms on the fraud dataset in this paper, and the training speed is up to 3.53 times faster. Furthermore, if the number of nodes is further increased, the speedup ratio will continue to increase. [ABSTRACT FROM AUTHOR]

A threat to governments and the medical community globally are infectious illness outbreaks such as COVID-19, Spanish flu, and Ebola, which kill millions of people. Furthermore, because people are afraid to go to their employment, infectious diseases hinder economic progress. Therefore, it is imperative to implement interventions to lessen the spread of infectious diseases. From this angle, the practice of tracking contacts might be viewed as a mitigation strategy to stop the spread of contagious diseases. As this is going on, users' trace data sets are growing exponentially larger over time, making it difficult to conduct contact tracing queries over them. In this paper, a novel Spark contact-tracing query processing method is proposed. This method determines if users are suspected cases by analyzing their paths based on two variables: nearby social distance and exposure time. Additionally, the developed method makes full use of the Spark framework to address scalability issues and effectively respond to contact tracing inquiries over a wide range of trajectories. [ABSTRACT FROM AUTHOR]

A very promising alternative fuel that can take the place of present fossil fuels is ammonia. Given that it is a hydrogen carrier with no carbon-based emissions and can be manufactured using renewable sources, it is a green fuel for Internal Compression Engine (ICE) with promising future applications. The notion of using pure ammonia in fueling combustion engines is not new, but despite numerous studies that looked into the possibility, Combustion engine fueling techniques based on ammonia are not yet ready for commercialization. This is because using this toxic molecule still presents significant challenges due to its supply, safety concerns, and physical differences from conventional fossil fuels. The objective of this review is to provide readers with a broad overview of the green ammonia research as a potential replacement fuel for ICE, as well as a look at potential future uses and workable solutions to related issues. The physicochemical characteristics of ammonia are discussed to determine the key benefits and drawbacks of its application in combustion engines, with an emphasis on the difficulties, advantages, and potentials for the future as well as recent technological advancements to get around the obstacles related to its production, storage, and utilization. Additionally, this work delves into sustainable approaches for the production of green ammonia and its application in spark ignition (SI) and compression ignition (CI) engines. Within this context, this study elucidates the foundational principles, research findings, practical considerations, economic implications, and prospective outlook towards a paradigm shift wherein ammonia assumes a significant role as a primary energy source in combustion engines. More study of ammonia combustion phenomena is necessary to propose solutions to these problems. This overview clarifies recent developments in ammonia technology from the point of manufacture to the point of consumption. [Display omitted] • A comprehensive review on NH 3 as a potential fuel in internal combustion engine (ICE). • Assessed physiochemical properties and different sustainable NH 3 production methods. • Spark ignition and compression ignition engines can work with NH 3 only or NH 3 dual-fuel. • Studies are required to ensured complete NH 3 combustion and to address NO x formation issues. [ABSTRACT FROM AUTHOR]

Wavelet neural networks (WNN) have been applied in many fields to solve regression as well as classification problems. After the advent of big data, as data gets generated at a brisk pace, it is imperative to analyze it as soon as it is generated owing to the fact that the nature of the data may change dramatically in short time intervals. This is necessitated by the fact that big data is all pervasive and throws computational challenges for data scientists. Therefore, in this paper, we built an efficient Scalable, Parallelized Wavelet Neural Network (SPWNN) which employs the parallel stochastic gradient algorithm (SGD) algorithm. SPWNN is designed and developed under both static and streaming environments in the horizontal parallelization framework. SPWNN is implemented by using Morlet and Gaussian functions as activation functions. This study is conducted on big datasets like gas sensor data which has more than 4 million samples and medical research data which has more than 10,000 features, which are high dimensional in nature. The experimental analysis indicates that in the static environment, SPWNN with Morlet activation function outperformed SPWNN with Gaussian on the classification datasets. However, in the case of regression, there is no clear trend was observed. In contrast, in the streaming environment i.e., Gaussian outperformed Morlet on the classification and Morlet outperformed Gaussian on the regression datasets. Overall, the proposed SPWNN architecture achieved a speedup of 1.22 - 1.78. [ABSTRACT FROM AUTHOR]

Falls detection approaches struggle with both Big Data scalability and upholding individual privacy, this research work proposed a novel approach for posture recognition followed by fall detection, taking advantage of the synergy between Random Forests and Uniform Local Binary Patterns (uLBP) histograms for an accurate and fast posture identification while respecting privacy. Additionally, it referred to deep recurrent neural networks distributed on a Hadoop and Spark platform for time series analysis in fall detection. This combination of methods allowed us to achieve acceptable real-time monitoring precision. This study, therefore addressed two objectives simultaneously: efficiency and scalability in posture recognition using Random Forests and uLBP, and fall detection relying on the recurrent neural network (RNN) for time series processing. The suggested solution is designed for home telemonitoring, where scalability and effective data management are supported through Hadoop/Spark. The integration of these technologies promotes reliable detection without any privacy violation, paving the way for a wider adoption of home monitoring systems for an increasing population of dependent individuals. [ABSTRACT FROM AUTHOR]

The MeanShift algorithm is a nonparametric method based on gradient ascent and it can effectively handle complex variations in lychee orchard scenes as well as changes in lychee tree canopies due to its adaptability, multi-scale analysis capabilities, and robustness, making it widely used in the segmentation processing of drone-based remote sensing images of lychee orchards. However, due to the high computational complexity of the MeanShift algorithm, its performance in processing large-scale drone remote sensing images of lychee tree canopies is not highly efficient, leading to low segmentation efficiency, which hampers its broader application. To address these issues, this study proposes high-speed MeanShift parallel segmentation algorithms for drone remote sensing images of lychee tree canopies based on MapReduce and Spark distributed computing frameworks. In this study, a cluster consisting of four nodes with Hadoop and Spark was set up, and 4000 drone remote sensing images were used as test data to evaluate the algorithm. Experimental results show that, the MeanShift algorithm based on MapReduce reduced the task execution time by 86.1% compared to the traditional MeanShift algorithm, while the MeanShift algorithm based on Spark reduced the task execution time by 88.0%, without compromising segmentation accuracy. The MeanShift parallel segmentation algorithm based on Hadoop and Spark platform can overcome the bottleneck of task execution efficiency and significantly enhance computational speed on a single machine. [ABSTRACT FROM AUTHOR]

In today's era, digital data is being created and transmitted majorly in the form of images and videos. Storing such a huge number of images and transmitting them requires a lot of computer resources. Instead of storing the image data as is, if we compress and store it, it saves a lot of resources. Image compression is the act of removing the maximum possible redundant data from an image and maintaining only the non-redundant data. In this paper, to compress and decompress such big image data, a distributed environment with a map-reduce paradigm using Hadoop Distributed File System (HDFS) and Spark is used. In addition to these, Microsoft Azure cloud environment is also used. Various setups like a single system, 1 + 4, 1 + 15, and 1 + 18 node clusters are used to show the time comparisons among these setups with the self-created large image dataset. On these four self-made clusters, more than 200 million (219,340,800) images are compressed and decompressed; the execution times are compared with two of the traditional image compression methods: JPEG and CCITT Group-3. To evaluate the efficiency of these two compression methods: Compression Ratio, Root Mean Square Error (RMSE), and Peak Signal to Noise Ratios (PSNR) are used. [ABSTRACT FROM AUTHOR]

Developing and executing large-scale data analysis applications in parallel and distributed environments can be a complex and time-consuming task. Developers often find themselves diverted from their application logic to handle technical details about the underlying runtime and related issues. To simplify this process, ParSoDA, a Java library, has been proposed to facilitate the development of parallel data mining applications executed on HPC systems. It simplifies the process by providing built-in scalability mechanisms relying on the Hadoop and Spark frameworks. This paper presents ParSoDA-Py, the Python version of the ParSoDA library, which allows for further support of commonly used runtimes and libraries for big data analysis. After a complete library redesign, ParSoDA can be now easily integrated with other Python-based distributed runtimes for HPC systems, such as COMPSs and Apache Spark, and with the large ecosystem of Python-based data processing libraries. The paper discusses the adaptation process, which takes into consideration the new technical requirements, and evaluates both usability and scalability through some case study applications. [ABSTRACT FROM AUTHOR]

Typhoon trajectory related data involve many factors, such as atmospheric factors, oceanic factors, and physical factors. It has the characteristics of high dimension, strong spatio-temporal correlation, and nonlinear correlation, which increases the difficulty of typhoon trajectory prediction. Using feature selection approaches to select appropriate prediction factors becomes an important means to reduce the dimension of typhoon trajectory related data and improve the performance and accuracy of typhoon trajectory prediction methods. However, the existing feature selection methods based on linear correlation analysis cannot well depict the nonlinear correlation between data features, which results in low accuracy of feature selection. The feature selection methods based on nonlinear correlation analysis are computationally expensive, which affects the timeliness of feature selection. To solve the problem, we propose a parallel feature selection method NX-Spark-DC based on the Spark platform for typhoon trajectory related data. The method firstly filters out the redundant features of typhoon related data by normalized mutual information (NMI) method, subsequently eliminates the useless features by XGBoost machine learning model, and thus reducing the dimension of typhoon related data. On this basis, an improved Spark-based parallel distance correlation algorithm (Spark-DC) is proposed to select the feature combinations with strong correlation. A series of experimental results show that NX-Spark-DC method has high execution efficiency and accuracy, which is significantly better than the existing methods. [ABSTRACT FROM AUTHOR]

Floating car GPS data serves as the foundation for processing traffic information, with the rapid increase in the number of monitored vehicles, a massive amount of GPS data is generated, posing great challenges to map matching. To address the shortcomings of traditional matching methods in terms of matching efficiency and accuracy, this paper proposed a real time parallel map matching algorithm for massive GPS data that ensured both high matching accuracy and computational efficiency. This paper firstly reconstructed an efficient and accurate real-time map matching algorithm for streaming data by introducing a comprehensive weight factor that considered velocity and direction to enhance the offline map matching algorithm that relied on historical trajectories. Then, it introduced the Spark Streaming distributed computing framework to achieve real-time and parallel computation of the map matching algorithm, significantly improving the efficiency of real-time map matching. Experimental results demonstrate that the proposed algorithm achieves more than a 10% increase in matching accuracy compared to conventional topological matching algorithms on complex road sections, with an overall matching accuracy of over 95%. In terms of matching efficiency, it achieves approximately a fourfold improvement compared to an equivalent number of standalone servers. The experimental results show that the proposed algorithm achieves real-time map matching of 80 million GPS data points on a computing cluster composed of 11 machines, proving that the proposed algorithm can achieve real-time vehicle matching in urban areas. [ABSTRACT FROM AUTHOR]

In the realm of distributed computing frameworks, such as Apache Spark and MapReduce Hadoop, the efficacy of these frameworks varies across diverse applications and algorithms contingent upon distinctive evaluation metrics and critical parameters. This research paper diligently scrutinizes the extant body of research that compares these two frameworks concerning said evaluation metrics and parameters. Subsequently, it conducts empirical investigations to authenticate the performance of these frameworks in the context of an iterative Gradient Boosting Tree Regression (GBTR) algorithm. Remarkably, the comparative analyses in previous studies encompass a spectrum of iterative machine learning regression and classification techniques, batch processing, SQL, and Graph processing algorithms. Furthermore, numerous investigations have explored the application of machine learning algorithms encompassing logistic regression, Page Rank, K-Means, KNN, and the HiBench suite. This paper presents the comparison between the two distributed computing platforms on iterative GBTR for classification task on the HIGGS dataset from the physics domain and for the regression task on the Covid-19 dataset from the healthcare domain. The empirical findings corroborate that Apache Spark exhibits superior execution speed in iterative tasks when the available physical memory significantly exceeds the dataset size. Conversely, Hadoop outperforms Spark when dealing with substantial datasets or constrained physical memory resources. [ABSTRACT FROM AUTHOR]

Spark, a distributed computing platform, has rapidly developed in the field of big data. Its in-memory computing feature reduces disk read overhead and shortens data processing time, making it have broad application prospects in large-scale computing applications such as machine learning and image processing.However, the performance of the Spark platform still needs to be improved.When a large number of tasks are processed simultaneously, Spark's cache replacement mechanism cannot identify high-value data partitions, resulting inmemory resources not being fully utilized and affecting the performance of the Spark platform. To address the problem that Spark's default cache replacement algorithm cannot accurately evaluate high-value data partitions, firstly the weight influence factors of data partitions are modeled and evaluated. Then, based on this weighted model, a cache replacement algorithm based on dynamic weighted data value is proposed, which takes into account hit rate and data difference. Better integration and usage strategies are implemented based on LRU(LeastRecentlyUsed). Theweight update algorithm updates the weight value when the data partition information changes, accurately measuring the importance of the partition in the current job; the cache removal algorithm clears partitions without useful values in the cache to releasememory resources; the weight replacement algorithm combines partition weights and partition information to replace RDD partitions when memory remaining space is insufficient. Finally, by setting up a Spark cluster environment, the algorithm proposed in this paper is experimentally verified. Experiments have shown that this algorithmcan effectively improve cache hit rate, enhance the performance of the platform, and reduce job execution time by 7.61% compared to existing improved algorithms. [ABSTRACT FROM AUTHOR]

Copyright of Big Data Research (2096-0271) is the property of Beijing Xintong Media Co., Ltd. and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Copyright of Systems Engineering & Electronics is the property of Journal of Systems Engineering & Electronics Editorial Department and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

As the information age develops, the scale and form of data become more diverse and diverse. Therefore, people need to use effective means to process information. For large-scale data mining problems, a clustering-based kernel matrix inner product filtering method is introduced to decompose the original quadratic programming problem into multiple sub-problems to support parallel training. And a Spark-based multiple submodels parallel support vector machine is proposed. By introducing open-source tools such as OpenCV, image feature extraction can be performed on large-scale video data. Finally, combined with the designed parallel support vector machine algorithm, video facial and expression recognition is carried out. These experiments confirmed that the research method achieved a maximum acceleration ratio of 2090 times when processing Covtype datasets. The research model could achieve an accuracy of over 99 %. Under the maximum data scale experiment, the research model improved prediction accuracy by 21 percentage points with an acceptable additional time cost of about 4 min only. Task parallel processing could more fully utilize cluster performance, increasing by approximately 3.5 m/s2 from 30 to 150 cores. The research model had the highest recognition accuracy for facial expressions, further demonstrating the effectiveness and superiority of this method. The research method has improved the efficiency of big data analysis and mining, and is of great significance in parallel analysis of video data.

In recent years, various studies have been conducted on SVMs and their applications in different area. They have been developed significantly in many areas. SVM is one of the most robust classification and regression algorithms that plays a significant role in pattern recognition. However, SVM has not been developed significantly in some areas like large-scale datasets, unbalanced datasets, and multiclass classification. Efficient SVM training in large-scale datasets is of great importance in the big data era. However, as the number of samples increases, the time and memory required to train SVM increase, making SVM impractical even for a medium-sized problem. With the emergence of big data, this problem becomes more significant. This paper presents a novel distributed method for SVM training in which a very small subset of training samples is used for classification, which reduces the problem size and thus the required memory and computational resources. The solution of this problem almost converges to standard SVM. This method includes three steps: first, detecting a subset of distributed training samples, second, creating local models of SVM and obtaining partial vectors, and finally combining the partial vectors and obtaining the global vector and the final model. In addition, the datasets which suffer from unbalanced number of samples and tend to the majority class, the proposed method balances the samples of the two classes and it can be used in unbalanced datasets. The empirical results show that using this method is efficient for large-scale problems. [ABSTRACT FROM AUTHOR]

The cooperative coevolution framework was introduced to address the shortcomings of metaheuristic algorithms in solving continuous large-scale global optimization problems. By dividing the problem into subcomponents which can be optimized separately, the framework can improve on both the solution's quality as well as the computational speed by exposing a degree of parallelism. Distributed computing platforms, such as Apache Spark, have long been used to improve the speed of different algorithms in solving computational problems. This work proposes a distributed implementation of the cooperative coevolution framework for solving large-scale global optimization problems on the Apache Spark distributed computing platform. By using a formerly outlined distributed variant of the cooperative coevolution framework, features of the Spark platform are utilized to enhance the computational speed of the algorithm while maintaining comparable search quality to other works in the literature. To test for the proposed implementation's improvement in computational speed, the CEC 2010 large-scale global optimization benchmark functions are used due to the diversity they offer in terms of complexity, separability and modality. Results of the proposed distributed implementation suggest that a speedup of up to ×3.36 is possible on large-scale global optimization benchmarks using the Apache Spark platform. [ABSTRACT FROM AUTHOR]

The large amount of data generated every day makes necessary the re-implementation of new methods capable of handle with massive data efficiently. This is the case of Association Rules, an unsupervised data mining tool capable of extracting information in the form of IF-THEN patterns. Although several methods have been proposed for the extraction of frequent itemsets (previous phase before mining association rules) in very large databases, the high computational cost and lack of memory remains a major problem to be solved when processing large data. Therefore, the aim of this paper is three fold: (1) to review existent algorithms for frequent itemset and association rule mining, (2)to develop new efficient frequent itemset Big Data algorithms using distributive computation, as well as a new association rule mining algorithm in Spark, and (3) to compare the proposed algorithms with the existent proposals varying the number of transactions and the number of items. To this purpose, we have used the Spark platform which has been demonstrated to outperform existing distributive algorithmic implementations. [ABSTRACT FROM AUTHOR]

Background: Fire simulators are increasingly used to predict fire spread. Australian fire agencies have been concerned at not having an objective basis to choose simulators for this purpose. Aims: We evaluated wildland fire simulators currently used in Australia: Australis, Phoenix, Prometheus and Spark. The evaluation results are outlined here, together with the evaluation framework. Methods: Spatial metrics and visual aids were designed in consultation with simulator end-users to assess simulator performance. Simulations were compared against observations of fire progression data from 10 Australian historical fire case studies. For each case, baseline simulations were produced using as inputs fire ignition and fuel data together with gridded weather forecasts available at the time of the fire. Perturbed simulations supplemented baseline simulations to explore simulator sensitivity to input uncertainty. Key results: Each simulator showed strengths and weaknesses. Some simulators displayed greater sensitivity to different parameters under certain conditions. Conclusions: No simulator was clearly superior to others. The evaluation framework developed can facilitate future assessment of Australian fire simulators. Implications: Collection of fire behaviour observations for routine simulator evaluation using this framework would benefit future simulator development. Australian fire and land management agencies use fire simulators to predict characteristics of fires in the landscape, and require evaluation of simulator accuracy. Simulations of fire spread were assessed against observations. Inputs were perturbed to evaluate simulator sensitivity. Against the evaluation measures, no simulators proved universally superior to the others. [ABSTRACT FROM AUTHOR]

In recent years, the surge in continuously accelerating data generation has given rise to the prominence of big data technology. The MapReduce architecture, situated at the core of this technology, provides a robust parallel environment. Spark, a leading framework in the big data landscape, extends the capabilities of the traditional MapReduce model. Coping with big data, especially in the realm of clustering, requires more efficient techniques. Meta-heuristic-based clustering, known for offering global solutions within reasonable time frames, emerges as a promising approach. This paper introduces a parallel-distributed clustering algorithm for big data within the Spark Framework, named Spark, chaotic improved PSO (S-CIPSO). Centered on particle swarm optimization (PSO), the proposed algorithm is enhanced with a chaotic map and an efficient procedure. Test results, conducted on both real and artificial datasets, establish the superior performance and quality of clustering results achieved by the proposed approach. Additionally, the scalability and robustness of S-CIPSO are validated, demonstrating its effectiveness in handling large-scale datasets. [ABSTRACT FROM AUTHOR]

Big data analytics is an emerging topic in academic and industrial engineering fields, where the large-scale data issue is the most attractive challenge. It is crucial to design an effective large-scale data processing model to handle big data. In this paper, we aim to improve the accuracy of the classification task and reduce the execution time for large-scale data within a small cluster. In order to overcome these challenges, this paper presents a novel ensemble-based paradigm that consists of the procedure of splitting large-scale data files and developing ensemble models. Two different splitting methods are first developed to partition large-scale data into small data blocks without overlapping. Then we propose two ensemble-based methods with high accuracy and less execution time: bagging-based and boosting-based methods. Finally, the proposed paradigm can be implemented by four predictive models, which are combinations of two splitting methods and two ensemble-based methods. A series of persuasive experiments was conducted to evaluate the effectiveness of the proposed paradigm with four different combinations. Overall, the proposed paradigm with boosting-based is the best in terms of the accuracy metric compared with existing methods. In addition, boosting-based methods achieve 91.6% accuracy compared with 52% accuracy of base line model for a big data file with 10 million samples. However, the paradigm with bagging-based takes the least execution time to yield results. This paper also reveals the effectiveness of the computing Spark cluster for large-scale data and points out the weakness of RDD (Resilient Distributed dataset). [ABSTRACT FROM AUTHOR]

Cloud computing is an on-demand model of computing that utilizes virtualization expertise to offer cloud resources such as CPU, memory, storage, and network in virtual machines. As a result, most big data analytics in many modern enterprise applications are run from the cloud. Since resources in these private clouds are limited, getting the most out of resource applications and providing guaranteed user service by efficiently scheduling tasks and resources is the ultimate goal. However, existing big data processing system schedulers need to consider application performance and resource utilization when performing allocations. Therefore, it is difficult to design workflows for low turnaround time and high resource consumption in extensive data systems. In this paper, we propose a resource management system for efficient job scheduling, called RMS, which dynamically schedules big data jobs in Kubernetes cluster nodes for Spark applications and autonomously adjusts scheduling policies in heterogeneous node clusters to enhance application execution and resource consumption. The RMS mechanism will ensure adequate guidance and resources available in its planning objectives and satisfactory resource utilization. The experimental analysis of different RMS and performance preferences using different methods depends on the predicted completion time and the benchmark statistical result of different significant data performance indicators traces. The results show that RMS decreases the cost and scheduling overhead and improves job execution performance. [ABSTRACT FROM AUTHOR]

Spark is one of the most popular big data analytical platforms. To save time, achieve high resource utilization, and remain cost-effective for Spark jobs, it is challenging but imperative for data scientists to configure suitable resource portions.In this paper, we investigate the proper parameter values that meet workloads' performance requirements with minimized resource cost and resource utilization time. We propose SimCost, a simulation-based cost model, to predict the performance of jobs accurately. We achieve low-cost training by taking advantage of simulation framework, i.e., Monte Carlo simulation, which uses a small amount of data and resources to make a reliable prediction for larger datasets and clusters. Our method's salient feature is that it allows us to invest low training costs while obtaining an accurate prediction. Through empirical experiments with 12 benchmark workloads, we show that the cost model yields less than 5% error on average prediction accuracy, and the recommendation achieves up to 6x resource cost saving. [ABSTRACT FROM AUTHOR]

The present work concerns microchannel fabrication using magnetic-assisted electrochemical discharge machining on silica glass. The magnetic field regulates the process by controlling bubble formation and separation adjacent to the tool edge. Due to the applied magnetic and electric flux on the electrolyte and electrode interface, the Lorentz force impacts the bubble establishment and film quality. The fractional bubble coverage decreases with the magnetic field, which increases the current density and affects the discharge frequency. The analytical model is explored here to investigate the magnetic field influences on departure radius. A novel approach has been adopted to couple the Laplace and Navier stokes equations for studying the magnetohydrodynamic effect. The experimental and analytical results reveal that magnetic flux decreases the bubble departure radius. The regression model explains the impact of controllable factors on surface roughness, material removal rate, and width overcut. The multi-response optimization with the hybridization of response surface methodology and grey relation analysis technique provides the optimum value of material removal rate, width overcut and surface roughness as 1.2 mg/min, 0.046 mm, and 1.58 µm, respectively. [ABSTRACT FROM AUTHOR]

Autism is heterogeneous at many levels, including clinical symptoms and etiology. A key strategy in studying heterogeneous conditions is having large enough sample sizes to stratify into smaller groups that are more homogeneous. SPARK and Simons Searchlight are large and growing research cohorts of individuals with autism in the United States and individuals with genetically defined neurodevelopmental conditions around the world, respectively. They both provide freely available phenotypic and genotypic data with the ability to re-contact participants through the research match program. Deep dives into each gene in Searchlight provide comprehensive natural history data to understand the differing clinical courses to inform proper clinical care, and work toward treatment for each condition. Moreover, pilots of genetically based newborn screening programs for neurogenetic disorders can provide opportunities for equitable and early diagnosis to try to improve outcomes with earlier interventions. [ABSTRACT FROM AUTHOR]

With the use of cloud computing, which provides the infrastructure necessary for the efficient delivery of smart city services to every citizen over the internet, intelligent systems may be readily integrated into smart cities and communicate with one another. Any smart system at home, in a car, or in the workplace can be remotely controlled and directed by the individual at any time. Continuous cloud service availability is becoming a critical subscriber requirement within smart cities. However, these cost-cutting measures and service improvements will make smart city cloud networks more vulnerable and at risk. The primary function of Intrusion Detection Systems (IDS) has gotten increasingly challenging due to the enormous proliferation of data created in cloud networks of smart cities. To alleviate these concerns, we provide a framework for automatic, reliable, and uninterrupted cloud availability of services for the network data security of intelligent connected devices. This framework enables IDS to defend against security threats and to provide services that meet the users' Quality of Service (QoS) expectations. This study's intrusion detection solution for cloud network data from smart cities employed Spark and Waikato Environment for Knowledge Analysis (WEKA). WEKA and Spark are linked and made scalable and distributed. The Hadoop Distributed File System (HDFS) storage advantages are combined with WEKA's Knowledge flow for processing cloud network data for smart cities. Utilizing HDFS components, WEKA's machine learning algorithms receive cloud network data from smart cities. This research utilizes the wrapper-based Feature Selection (FS) approach for IDS, employing both the Pigeon Inspired Optimizer (PIO) and the Particle Swarm Optimization (PSO). For classifying the cloud network traffic of smart cities, the tree-based Stacking Ensemble Method (SEM) of J48, Random Forest (RF), and eXtreme Gradient Boosting (XGBoost) are applied. Performance evaluations of our system were conducted using the UNSW-NB15 and NSL-KDD datasets. Our technique is superior to previous works in terms of sensitivity, specificity, precision, false positive rate (FPR), accuracy, F1 Score, and Matthews correlation coefficient (MCC). [ABSTRACT FROM AUTHOR]

The Spark computing framework provides an efficient solution to address the major requirements of big data processing, but data partitioning and job scheduling in the Spark framework are the two major bottlenecks that limit Spark's performance. In the Spark Shuffle phase, the data skewing problem caused by unbalanced data partitioning leads to the problem of increased job completion time. In response to the above problems, a balanced partitioning strategy for intermediate data is proposed in this article, which considers the characteristics of intermediate data, establishes a data skewing model and proposes a dynamic partitioning algorithm. In Spark heterogeneous clusters, because of the differences in node performance and task requirements, the default task scheduling algorithm cannot complete scheduling efficiently, which leads to low system task processing efficiency. In order to deal with the above problems, an efficient job scheduling strategy is proposed in this article, which integrates node performance and task requirements, and proposes a task scheduling algorithm using greedy strategy. The experimental results prove that the dynamic partitioning algorithm for intermediate data proposed in this article effectively alleviates the problem that data skew leads to the decrease of system task processing efficiency and shortens the overall task completion time. The efficient job scheduling strategy proposed in this article can efficiently complete the job scheduling tasks under heterogeneous clusters, allocate jobs to nodes in a balanced manner, decrease the overall job completion time and increase the system resource utilization. [ABSTRACT FROM AUTHOR]

Processing extensive and diverse data in real-time is a significant challenge in the context of smart cities. Timely access to information and efficient analytics is essential for smart city services to make data-driven decisions and enhance urban living. Scheduling algorithms play a crucial role in ensuring the prompt delivery of services and efficient task completion. This paper explores various scheduling techniques, including static, dynamic, and hybrid schedulers, and compares their objectives and performance. Additionally, the study examines two prominent data processing frameworks, Hadoop and Spark, and compares their capabilities in handling big data in smart cities. With its ability to process large amounts of data quickly and efficiently, Spark has shown superiority over Hadoop in realtime data processing and performance optimization. The paper concludes by highlighting the strengths and limitations of each framework. It discusses the need for further research in optimizing scheduling techniques and exploring hybrid artificial intelligence scheduling for Spark. Overall, the findings contribute to a better understanding of data processing in real-time and provide insights for researchers and practitioners in smart cities. [ABSTRACT FROM AUTHOR]

In the dynamic realm of big data processing, conquering the challenges imposed by highdimensional datasets is imperative. This paper introduces a groundbreaking advancement in dimensionality reduction, employing Variational Auto-Encoder (VAE) within the Spark distributed framework. The deliberate selection of the "TLC" dataset, representative of New York City taxi trips with inherent high dimensionality, highlights the practicality of our approach. Our research showcases the virtuoso performance of VAE, achieving an impressive 95.12% reduction ratio and 89.26% accuracy. This highlights VAE's ability to elegantly distill essential information while discarding superfluous dimensions, achieving a harmonious balance between reduction and accuracy. Furthermore, building on the demonstrated superiority of Spark over Hadoop in prior successes, our adoption of VAE aligns with the overarching goal of enhancing big data processing. Spark's consistent advantage as a distributed framework reaffirms its reliability in handling diverse machine learning algorithms. This paper not only contributes to the advancement of machine learning in big data processing but also underscores the adaptability, versatility, and consistent performance of our approach across various methodologies and frameworks. The success of VAE in reducing dimensionality, coupled with Spark's inherent advantages, positions this research as a valuable contribution to the exploration of advanced techniques in distributed big data processing. [ABSTRACT FROM AUTHOR]

Research in movie recommendation systems addresses several specific challenges to enhance the accuracy, relevance, and user satisfaction of the recommendations. Some of the key challenges include sparsity of data and cold start for new movies. Integrating collaborative filtering and content-based filtering in a seamless and effective manner also poses another key challenge. This article explores strategies for optimizing hybrid recommendation systems to leverage the strengths of each approach. By addressing these three key challenges, we aim to advance the state of movie recommendation systems, providing more effective and user-centric solutions that cater to the diverse preferences and needs of movie enthusiasts. Similarity-based models (memory-based) combined with matrix factorization-based models (model-based) were used to generate an optimal hybrid movie recommendation model. We evaluated the models using a large dataset of movies and we showed that the proposed model is both efficient and scalable. [ABSTRACT FROM AUTHOR]

This study uses the Random Forest algorithm to evaluate the efficiency of Hadoop, and Spark distributed computing systems for intrusion detection, highlighting the growing importance of efficient distributed systems in handling big data. This research aims to assess and compare the performance of Hadoop and Spark in the context of an intelligent intrusion detection system. We use the Random Forest machine learning algorithm to train and test the system. The methods developed an intrusion detection system using Hadoop and Spark frameworks, followed by a thorough performance assessment using a real-world dataset. The problem this study tackles is the ever-increasing demand for processing data swiftly and accurately in a distributed fashion. We aim to identify the strengths and weaknesses of Hadoop and Spark in the context of machine learning-based intrusion detection. The “intelligent network detection system for intrusions” in this study uses a sophisticated security system using machine learning algorithms to detect potential intrusions, assessing Hadoop and Spark's performance in realworld scenarios and handling large-scale data processing. The findings provide insightful information about the efficacy and efficiency of distributed systems in machine learning activities, which can help select big data application frameworks. [ABSTRACT FROM AUTHOR]

When there is a need to make an ultimate decision about the unique features of big data platforms, one should note that they have configurable parameters. Apache Spark is an open-source big data processing platform that can process real-time data, and it requires an advanced central processing unit and high memory capacity. Therefore, it gives us a great number of configurable parameters such as the number of cores and driver memory that are tuned during the execution. Different from the preceding works, in this study, a Kriging-based multi-objective optimization method is developed. Kriging-based means executing a surrogate model to create a response surface by providing a set of optimal solutions. The most important advantage of the proposed method over the alternatives is that it consists of three fitness functions. The method is evaluated on the MLlib library and the benchmarks of Hibench. MLlib provides various machine learning algorithms that are suitable to execute on resilient distributed data sets. The experimental results show that the proposed method outperformed the alternatives in hypervolume improvement and reducing uncertainty. Further, the results support the hypothesis that focusing on the parameters associated with data compression and memory usage improves the effectiveness of multi-objective optimization methods developed for Spark. Multi-objective optimization leads to an inevitable complexity in Spark due to the dimensionality of objective functions. Despite the fact that simplifying the setup and steps of optimization has proven to be the most effective way to reduce that complexity, it is not very effective to avoid ambiguity of the Pareto front. While the proposed method achieved 1.93x speedup in benchmark experiments, there is a remarkable difference (0.63 of speedup) between the speedup of our method and that of the closest competitor. Increasing the number of cores in multi-objective optimization does not contribute to speedup; rather, it leads to waste of CPU sources. Instead, the optimal number of cores should be determined by checking the changes of speedup with varying Spark configurations. [ABSTRACT FROM AUTHOR]