Search

Copyright of Retos: Nuevas Perspectivas de Educación Física, Deporte y Recreación is the property of Federacion Espanola de Asociaciones de Docentes de Educacion Fisica and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

传统Bootstrap抽样和Bagging集成学习通常以串行方式实现，计算效率低，且存在样本不可重用、扩展性差等问题，不适合高效的大规模Bagging集成学习。从大数据分布式计算的思维入手，提出新的Bootstrap样本划分（BSP）大数据模型和分布式集成学习方法。BSP数据模型通过分布式生成算法将训练数据表达成分布式Bootstrap样本集的集合，存储成HDFS分布式数据文件，为后续的分布式集成学习提供数据支持。分布式集成学习方法从BSP数据模型中随机选取多个BSP数据块，读入集群各个节点的虚拟机，用串行算法对选取的数据块并行计算统计量或训练建模，再将所有的计算子结果回传至主节点中，生成最终的集成学习结果，此过程中可加入对子结果的质量选择以进一步提高预测效果。BSP数据模型的生成和分布式集成学习采用非Map-Reduce计算范式进行，每个数据块的计算独立完成，减少了计算节点间的数据通信开销。提出的算法在Spark开源系统中以新的算子方式实现，供Spark应用程序调用。实验表明，新方法可以高效地生成训练数据的BSP数据模型，提高数据样本的可重用性，在基于有监督机器学习算法构建的大规模Bagging集成学习实验中，计算效率能提高50%以上，同时预测精度进一步提高约2%。

Falls detection approaches struggle with both Big Data scalability and upholding individual privacy, this research work proposed a novel approach for posture recognition followed by fall detection, taking advantage of the synergy between Random Forests and Uniform Local Binary Patterns (uLBP) histograms for an accurate and fast posture identification while respecting privacy. Additionally, it referred to deep recurrent neural networks distributed on a Hadoop and Spark platform for time series analysis in fall detection. This combination of methods allowed us to achieve acceptable real-time monitoring precision. This study, therefore addressed two objectives simultaneously: efficiency and scalability in posture recognition using Random Forests and uLBP, and fall detection relying on the recurrent neural network (RNN) for time series processing. The suggested solution is designed for home telemonitoring, where scalability and effective data management are supported through Hadoop/Spark. The integration of these technologies promotes reliable detection without any privacy violation, paving the way for a wider adoption of home monitoring systems for an increasing population of dependent individuals. [ABSTRACT FROM AUTHOR]

The MeanShift algorithm is a nonparametric method based on gradient ascent and it can effectively handle complex variations in lychee orchard scenes as well as changes in lychee tree canopies due to its adaptability, multi-scale analysis capabilities, and robustness, making it widely used in the segmentation processing of drone-based remote sensing images of lychee orchards. However, due to the high computational complexity of the MeanShift algorithm, its performance in processing large-scale drone remote sensing images of lychee tree canopies is not highly efficient, leading to low segmentation efficiency, which hampers its broader application. To address these issues, this study proposes high-speed MeanShift parallel segmentation algorithms for drone remote sensing images of lychee tree canopies based on MapReduce and Spark distributed computing frameworks. In this study, a cluster consisting of four nodes with Hadoop and Spark was set up, and 4000 drone remote sensing images were used as test data to evaluate the algorithm. Experimental results show that, the MeanShift algorithm based on MapReduce reduced the task execution time by 86.1% compared to the traditional MeanShift algorithm, while the MeanShift algorithm based on Spark reduced the task execution time by 88.0%, without compromising segmentation accuracy. The MeanShift parallel segmentation algorithm based on Hadoop and Spark platform can overcome the bottleneck of task execution efficiency and significantly enhance computational speed on a single machine. [ABSTRACT FROM AUTHOR]

Developing and executing large-scale data analysis applications in parallel and distributed environments can be a complex and time-consuming task. Developers often find themselves diverted from their application logic to handle technical details about the underlying runtime and related issues. To simplify this process, ParSoDA, a Java library, has been proposed to facilitate the development of parallel data mining applications executed on HPC systems. It simplifies the process by providing built-in scalability mechanisms relying on the Hadoop and Spark frameworks. This paper presents ParSoDA-Py, the Python version of the ParSoDA library, which allows for further support of commonly used runtimes and libraries for big data analysis. After a complete library redesign, ParSoDA can be now easily integrated with other Python-based distributed runtimes for HPC systems, such as COMPSs and Apache Spark, and with the large ecosystem of Python-based data processing libraries. The paper discusses the adaptation process, which takes into consideration the new technical requirements, and evaluates both usability and scalability through some case study applications. [ABSTRACT FROM AUTHOR]

Typhoon trajectory related data involve many factors, such as atmospheric factors, oceanic factors, and physical factors. It has the characteristics of high dimension, strong spatio-temporal correlation, and nonlinear correlation, which increases the difficulty of typhoon trajectory prediction. Using feature selection approaches to select appropriate prediction factors becomes an important means to reduce the dimension of typhoon trajectory related data and improve the performance and accuracy of typhoon trajectory prediction methods. However, the existing feature selection methods based on linear correlation analysis cannot well depict the nonlinear correlation between data features, which results in low accuracy of feature selection. The feature selection methods based on nonlinear correlation analysis are computationally expensive, which affects the timeliness of feature selection. To solve the problem, we propose a parallel feature selection method NX-Spark-DC based on the Spark platform for typhoon trajectory related data. The method firstly filters out the redundant features of typhoon related data by normalized mutual information (NMI) method, subsequently eliminates the useless features by XGBoost machine learning model, and thus reducing the dimension of typhoon related data. On this basis, an improved Spark-based parallel distance correlation algorithm (Spark-DC) is proposed to select the feature combinations with strong correlation. A series of experimental results show that NX-Spark-DC method has high execution efficiency and accuracy, which is significantly better than the existing methods. [ABSTRACT FROM AUTHOR]

In the realm of distributed computing frameworks, such as Apache Spark and MapReduce Hadoop, the efficacy of these frameworks varies across diverse applications and algorithms contingent upon distinctive evaluation metrics and critical parameters. This research paper diligently scrutinizes the extant body of research that compares these two frameworks concerning said evaluation metrics and parameters. Subsequently, it conducts empirical investigations to authenticate the performance of these frameworks in the context of an iterative Gradient Boosting Tree Regression (GBTR) algorithm. Remarkably, the comparative analyses in previous studies encompass a spectrum of iterative machine learning regression and classification techniques, batch processing, SQL, and Graph processing algorithms. Furthermore, numerous investigations have explored the application of machine learning algorithms encompassing logistic regression, Page Rank, K-Means, KNN, and the HiBench suite. This paper presents the comparison between the two distributed computing platforms on iterative GBTR for classification task on the HIGGS dataset from the physics domain and for the regression task on the Covid-19 dataset from the healthcare domain. The empirical findings corroborate that Apache Spark exhibits superior execution speed in iterative tasks when the available physical memory significantly exceeds the dataset size. Conversely, Hadoop outperforms Spark when dealing with substantial datasets or constrained physical memory resources. [ABSTRACT FROM AUTHOR]

Copyright of Big Data Research (2096-0271) is the property of Beijing Xintong Media Co., Ltd. and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

As the information age develops, the scale and form of data become more diverse and diverse. Therefore, people need to use effective means to process information. For large-scale data mining problems, a clustering-based kernel matrix inner product filtering method is introduced to decompose the original quadratic programming problem into multiple sub-problems to support parallel training. And a Spark-based multiple submodels parallel support vector machine is proposed. By introducing open-source tools such as OpenCV, image feature extraction can be performed on large-scale video data. Finally, combined with the designed parallel support vector machine algorithm, video facial and expression recognition is carried out. These experiments confirmed that the research method achieved a maximum acceleration ratio of 2090 times when processing Covtype datasets. The research model could achieve an accuracy of over 99 %. Under the maximum data scale experiment, the research model improved prediction accuracy by 21 percentage points with an acceptable additional time cost of about 4 min only. Task parallel processing could more fully utilize cluster performance, increasing by approximately 3.5 m/s2 from 30 to 150 cores. The research model had the highest recognition accuracy for facial expressions, further demonstrating the effectiveness and superiority of this method. The research method has improved the efficiency of big data analysis and mining, and is of great significance in parallel analysis of video data.

In recent years, various studies have been conducted on SVMs and their applications in different area. They have been developed significantly in many areas. SVM is one of the most robust classification and regression algorithms that plays a significant role in pattern recognition. However, SVM has not been developed significantly in some areas like large-scale datasets, unbalanced datasets, and multiclass classification. Efficient SVM training in large-scale datasets is of great importance in the big data era. However, as the number of samples increases, the time and memory required to train SVM increase, making SVM impractical even for a medium-sized problem. With the emergence of big data, this problem becomes more significant. This paper presents a novel distributed method for SVM training in which a very small subset of training samples is used for classification, which reduces the problem size and thus the required memory and computational resources. The solution of this problem almost converges to standard SVM. This method includes three steps: first, detecting a subset of distributed training samples, second, creating local models of SVM and obtaining partial vectors, and finally combining the partial vectors and obtaining the global vector and the final model. In addition, the datasets which suffer from unbalanced number of samples and tend to the majority class, the proposed method balances the samples of the two classes and it can be used in unbalanced datasets. The empirical results show that using this method is efficient for large-scale problems. [ABSTRACT FROM AUTHOR]

Background: Fire simulators are increasingly used to predict fire spread. Australian fire agencies have been concerned at not having an objective basis to choose simulators for this purpose. Aims: We evaluated wildland fire simulators currently used in Australia: Australis, Phoenix, Prometheus and Spark. The evaluation results are outlined here, together with the evaluation framework. Methods: Spatial metrics and visual aids were designed in consultation with simulator end-users to assess simulator performance. Simulations were compared against observations of fire progression data from 10 Australian historical fire case studies. For each case, baseline simulations were produced using as inputs fire ignition and fuel data together with gridded weather forecasts available at the time of the fire. Perturbed simulations supplemented baseline simulations to explore simulator sensitivity to input uncertainty. Key results: Each simulator showed strengths and weaknesses. Some simulators displayed greater sensitivity to different parameters under certain conditions. Conclusions: No simulator was clearly superior to others. The evaluation framework developed can facilitate future assessment of Australian fire simulators. Implications: Collection of fire behaviour observations for routine simulator evaluation using this framework would benefit future simulator development. Australian fire and land management agencies use fire simulators to predict characteristics of fires in the landscape, and require evaluation of simulator accuracy. Simulations of fire spread were assessed against observations. Inputs were perturbed to evaluate simulator sensitivity. Against the evaluation measures, no simulators proved universally superior to the others. [ABSTRACT FROM AUTHOR]

In recent years, the surge in continuously accelerating data generation has given rise to the prominence of big data technology. The MapReduce architecture, situated at the core of this technology, provides a robust parallel environment. Spark, a leading framework in the big data landscape, extends the capabilities of the traditional MapReduce model. Coping with big data, especially in the realm of clustering, requires more efficient techniques. Meta-heuristic-based clustering, known for offering global solutions within reasonable time frames, emerges as a promising approach. This paper introduces a parallel-distributed clustering algorithm for big data within the Spark Framework, named Spark, chaotic improved PSO (S-CIPSO). Centered on particle swarm optimization (PSO), the proposed algorithm is enhanced with a chaotic map and an efficient procedure. Test results, conducted on both real and artificial datasets, establish the superior performance and quality of clustering results achieved by the proposed approach. Additionally, the scalability and robustness of S-CIPSO are validated, demonstrating its effectiveness in handling large-scale datasets. [ABSTRACT FROM AUTHOR]

Spark is one of the most popular big data analytical platforms. To save time, achieve high resource utilization, and remain cost-effective for Spark jobs, it is challenging but imperative for data scientists to configure suitable resource portions.In this paper, we investigate the proper parameter values that meet workloads' performance requirements with minimized resource cost and resource utilization time. We propose SimCost, a simulation-based cost model, to predict the performance of jobs accurately. We achieve low-cost training by taking advantage of simulation framework, i.e., Monte Carlo simulation, which uses a small amount of data and resources to make a reliable prediction for larger datasets and clusters. Our method's salient feature is that it allows us to invest low training costs while obtaining an accurate prediction. Through empirical experiments with 12 benchmark workloads, we show that the cost model yields less than 5% error on average prediction accuracy, and the recommendation achieves up to 6x resource cost saving. [ABSTRACT FROM AUTHOR]

With the use of cloud computing, which provides the infrastructure necessary for the efficient delivery of smart city services to every citizen over the internet, intelligent systems may be readily integrated into smart cities and communicate with one another. Any smart system at home, in a car, or in the workplace can be remotely controlled and directed by the individual at any time. Continuous cloud service availability is becoming a critical subscriber requirement within smart cities. However, these cost-cutting measures and service improvements will make smart city cloud networks more vulnerable and at risk. The primary function of Intrusion Detection Systems (IDS) has gotten increasingly challenging due to the enormous proliferation of data created in cloud networks of smart cities. To alleviate these concerns, we provide a framework for automatic, reliable, and uninterrupted cloud availability of services for the network data security of intelligent connected devices. This framework enables IDS to defend against security threats and to provide services that meet the users' Quality of Service (QoS) expectations. This study's intrusion detection solution for cloud network data from smart cities employed Spark and Waikato Environment for Knowledge Analysis (WEKA). WEKA and Spark are linked and made scalable and distributed. The Hadoop Distributed File System (HDFS) storage advantages are combined with WEKA's Knowledge flow for processing cloud network data for smart cities. Utilizing HDFS components, WEKA's machine learning algorithms receive cloud network data from smart cities. This research utilizes the wrapper-based Feature Selection (FS) approach for IDS, employing both the Pigeon Inspired Optimizer (PIO) and the Particle Swarm Optimization (PSO). For classifying the cloud network traffic of smart cities, the tree-based Stacking Ensemble Method (SEM) of J48, Random Forest (RF), and eXtreme Gradient Boosting (XGBoost) are applied. Performance evaluations of our system were conducted using the UNSW-NB15 and NSL-KDD datasets. Our technique is superior to previous works in terms of sensitivity, specificity, precision, false positive rate (FPR), accuracy, F1 Score, and Matthews correlation coefficient (MCC). [ABSTRACT FROM AUTHOR]

In modern society, people's pace of life is fast, and the pressure is enormous, leading to increasingly prominent issues such as obesity and sub-health. Traditional fitness methods cannot meet people's needs to a certain extent. Therefore, this work aims to use technology to change people's lifestyles and compensate for traditional fitness methods' shortcomings. Firstly, this work overviews neurorobotics, providing neural perception and control functions for aerobics intelligent fitness system. Secondly, the connection between big data and machine learning (ML), big data technology products, and the ML process are discussed. The Spark big data platform builds node data for calculation, and the decision tree algorithm is used for data preprocessing. These are important for future intelligent fitness analysis. This work proposes an aerobics intelligent fitness system based on neurorobotics technology and big data analysis and develops a recommendation system for the best fitness exercise. This system utilizes neural perception and control functions, combined with big data and ML technology, to solve the obesity and sub-health problems faced by people in fast-paced and high-pressure lifestyles. By harnessing the computational capabilities of the Spark big data platform and applying the decision tree algorithm for data preprocessing, the system can furnish users with personalized fitness plans and optimization recommendations. This work conducts a model performance study on 35 % aerobic fitness data on intelligent fitness Android v1.0.8 to evaluate the system's data processing ability and training effectiveness. Moreover, the aerobics intelligent fitness system models based on neurorobotics, big data, and ML are evaluated. The results indicate that normalizing the data using the Min-Max method leads to a decrease in the F1 value and a reduction in data set errors. Consequently, the dataset studied by the system model is beneficial to improving the work efficiency of the aerobics intelligent fitness system. After the comprehensive human quality of the system model is evaluated, the actual average score of the comprehensive human quality of the 13 users tested before the aerobics intelligent fitness system test is 91.44, and the average prediction score is 90.88. The results of the two tests are similar. Thus, using the intelligent fitness system can enable the user to obtain system feedback according to the actual training effect, thereby playing a guiding role in the intelligent fitness of aerobics for the user. This work designs and implements the aerobics intelligent fitness system close to the human body's training effect, further enhancing the specialization and individualization of sports and fitness.

Abstract Distributed deep learning is a promising approach for training and deploying large and complex deep learning models. This paper presents a comprehensive workflow for deploying and optimizing the YOLACT instance segmentation model as on big data clusters. OpenVINO, a toolkit known for its high-speed data processing and ability to optimize deep learning models for deployment on a variety of devices, was used to optimize the YOLACT model. The model is then run on a big data cluster using BigDL, a distributed deep learning library for Apache Spark. BigDL provides a high-level programming interface for defining and training deep neural networks, making it suitable for large-scale deep learning applications. In distributed deep learning, input data is divided and distributed across multiple machines for parallel processing. This approach offers several advantages, including the ability to handle very large data that can be stored in a distributed manner, scalability to decrease processing time by increasing the number of workers, and fault tolerance. The proposed workflow was evaluated on virtual machines and Azure Databricks, a cloud-based platform for big data analytics. The results indicated that the workflow can scale to large datasets and deliver high performance on Azure Databricks. This study explores the benefits and challenges of using distributed deep learning on big data clusters for instance segmentation. Popular distributed deep learning frameworks are discussed, and BigDL is chosen. Overall, this study highlights the practicality of distributed deep learning for deploying and scaling sophisticated deep learning models on big data clusters.

With the use of cloud computing, which provides the infrastructure necessary for the efficient delivery of smart city services to every citizen over the internet, intelligent systems may be readily integrated into smart cities and communicate with one another. Any smart system at home, in a car, or in the workplace can be remotely controlled and directed by the individual at any time. Continuous cloud service availability is becoming a critical subscriber requirement within smart cities. However, these cost-cutting measures and service improvements will make smart city cloud networks more vulnerable and at risk. The primary function of Intrusion Detection Systems (IDS) has gotten increasingly challenging due to the enormous proliferation of data created in cloud networks of smart cities. To alleviate these concerns, we provide a framework for automatic, reliable, and uninterrupted cloud availability of services for the network data security of intelligent connected devices. This framework enables IDS to defend against security threats and to provide services that meet the users' Quality of Service (QoS) expectations. This study's intrusion detection solution for cloud network data from smart cities employed Spark and Waikato Environment for Knowledge Analysis (WEKA). WEKA and Spark are linked and made scalable and distributed. The Hadoop Distributed File System (HDFS) storage advantages are combined with WEKA's Knowledge flow for processing cloud network data for smart cities. Utilizing HDFS components, WEKA's machine learning algorithms receive cloud network data from smart cities. This research utilizes the wrapper-based Feature Selection (FS) approach for IDS, employing both the Pigeon Inspired Optimizer (PIO) and the Particle Swarm Optimization (PSO). For classifying the cloud network traffic of smart cities, the tree-based Stacking Ensemble Method (SEM) of J48, Random Forest (RF), and eXtreme Gradient Boosting (XGBoost) are applied. Performance evaluations of our system were conducted using the UNSW-NB15 and NSL-KDD datasets. Our technique is superior to previous works in terms of sensitivity, specificity, precision, false positive rate (FPR), accuracy, F1 Score, and Matthews correlation coefficient (MCC).

Distributed deep learning is a promising approach for training and deploying large and complex deep learning models. This paper presents a comprehensive workflow for deploying and optimizing the YOLACT instance segmentation model as on big data clusters. OpenVINO, a toolkit known for its high-speed data processing and ability to optimize deep learning models for deployment on a variety of devices, was used to optimize the YOLACT model. The model is then run on a big data cluster using BigDL, a distributed deep learning library for Apache Spark. BigDL provides a high-level programming interface for defining and training deep neural networks, making it suitable for large-scale deep learning applications. In distributed deep learning, input data is divided and distributed across multiple machines for parallel processing. This approach offers several advantages, including the ability to handle very large data that can be stored in a distributed manner, scalability to decrease processing time by increasing the number of workers, and fault tolerance. The proposed workflow was evaluated on virtual machines and Azure Databricks, a cloud-based platform for big data analytics. The results indicated that the workflow can scale to large datasets and deliver high performance on Azure Databricks. This study explores the benefits and challenges of using distributed deep learning on big data clusters for instance segmentation. Popular distributed deep learning frameworks are discussed, and BigDL is chosen. Overall, this study highlights the practicality of distributed deep learning for deploying and scaling sophisticated deep learning models on big data clusters. [ABSTRACT FROM AUTHOR]

This study introduces UWF-ZeekDataFall22, a newly created dataset labeled using the MITRE ATT&CK framework. Although the focus of this research is on classifying the never-before classified resource development tactic, the reconnaissance and discovery tactics were also classified. The results were also compared to a similarly created dataset, UWF-ZeekData22, created in 2022. Both of these datasets, UWF-ZeekDataFall22 and UWF-ZeekData22, created using Zeek Conn logs, were stored in a Big Data Framework, Hadoop. For machine learning classification, Apache Spark was used in the Big Data Framework. To summarize, the uniqueness of this work is its focus on classifying attack tactics. For UWF-ZeekdataFall22, the binary as well as the multinomial classifier results were compared, and overall, the results of the binary classifier were better than the multinomial classifier. In the binary classification, the tree-based classifiers performed better than the other classifiers, although the decision tree and random forest algorithms performed almost equally well in the multinomial classification too. Taking training time into consideration, decision trees can be considered the most efficient classifier. [ABSTRACT FROM AUTHOR]

Apache Spark is a high-speed computing engine for processing massive data. With its widespread adoption, there is a growing need to analyze its correctness and temporal properties. However, there is scarce research focused on the verification of temporal properties in Spark programs. To address this gap, we employ the code-level runtime verification tool UMC4M based on the Modeling, Simulation, and Verification Language (MSVL). To this end, a Spark program S has to be translated into an MSVL program M, and the negation of the property P specified by a Propositional Projection Temporal Logic (PPTL) formula that needs to be verified is also translated to an MSVL program M1; then, a new MSVL program "M and M1" can be compiled and executed. Whether program S violates the property P is determined by the existence of an acceptable execution of "M and M1". Thus, the key issue lies in how to formalize model Spark programs using MSVL programs. We previously proposed a solution to this problem—using the MSVL functions to perform Resilient Distributed Datasets (RDD) operations and converting the Spark program into an MSVL program based on the Directed Acyclic Graph (DAG) of the Spark program. However, we only proposed this idea. Building upon this foundation, we implement the conversion from RDD operations to MSVL functions and propose, as well as implement, the rules for translating Spark programs to MSVL programs based on DAG. We confirm the feasibility of this approach and provide a viable method for verifying the temporal properties of Spark programs. Additionally, an automatic translation tool, S2M, is developed. Finally, a case study is presented to demonstrate this conversion process. [ABSTRACT FROM AUTHOR]

Miller and Urey applied electric sparks to a reducive mixture of CH4, NH3, and water to obtain a complex organic mixture including biomolecules. In this study, we examined the impact of temperature, initial pressure, ammonia concentration, and the spark generator on the chemical profile of a Miller–Urey-type prebiotic broth. We analyzed the broth composition using Gas Chromatography combined with Mass Spectroscopy (GC/MS). The results point towards strong compositional changes with the nature of the spark. Ammonia exhibited catalytic properties even with non-nitrogen-containing compounds. A more elevated temperature led to a higher variety of substances. We conclude that to reproduce such a broth as well as possible, all the studied parameters need to be tightly controlled, the most difficult and important being spark generation. [ABSTRACT FROM AUTHOR]

The stochastic nature of environmental factors that govern the behavior of fire, such as wind and fuel, exposes wildfire modeling to a degree of uncertainty. In order to produce more realistic wildfire predictions, it is, therefore, necessary to incorporate these uncertainties within wildfire models in a way that reflects the influence of environmental stochasticity on wildfire propagation. Otherwise, the risks of the potential danger of a given wildfire may be under-represented. Specifically, environmental stochasticity in the form of wind variability results in considerable uncertainty in the output of fire spread models. Here, we consider two stochastic wind models and their implementation in the spark fire simulator framework to capture the environmental uncertainty related to wind variability. The results are compared with the output from purely deterministic wildfire spread models and are discussed in the context of the potential ramifications for wildfire risk management. [ABSTRACT FROM AUTHOR]

У використанні сучасних інструментів оброблення великих даних виникає проблема підвищення продуктивності сучасних фреймворків у контексті ефективного налаштування різних конфігураційних параметрів. Об’єктом дослідження є обчислювальні процеси оброблення великих даних із застосуванням технологій надпродуктивних фреймворків. Предметом є методи та підходи до ефективного налаштування конфігураційних параметрів фреймворків в умовах обмежень середовищ віртуалізації та локального ресурсу. Мета дослідження полягає в підвищенні продуктивності режимів розгортання Apache Spark та Apache Hadoop на основі комбінованого підходу, що містить передпроцесне сегментування вхідних даних та налаштування основних та додаткових конфігураційних параметрів з огляду на обмеження віртуального середовища та локального ресурсу. Досягнення поставленої мети передбачає виконання низки завдань: 1) створити синтезований набір тестових даних WordCount для використання методів сегментування вхідної інформації; 2) визначити склад загальних та специфічних конфігураційних параметрів Apache Spark та Apache Hadoop, що найбільше впливають на продуктивність роботи фреймворків у режимах розгортання Spark Standalone та Hadoop Yarn (FIFO); 3) обґрунтувати зміни значень конфігураційних параметрів (прийняті за замовчуванням) за допомогою налаштувань рівня паралелізму, кількості розбиттів вхідного файлу відповідно до кількості ядер процесора, кількості завдань, що призначаються на кожне ядро та виконавця в системі; 4) перевірити теоретичні результати та довести їх використання на практиці. У дослідженні впроваджено такі методи: статистичний аналіз; метод генерації тестових даних за визначеними характеристиками сегментування з довільними обсягами інформації; системний підхід для комплексного оцінювання та аналізу продуктивності фреймворків на основі обраних конфігураційних параметрів. Результати. На основі запропонованого методу вибору складу параметрів для оцінювання продуктивності досліджуваних фреймворків проведено експерименти, що передбачали: застосування методу сегментування вхідної інформації на основі розділення вхідного файлу на абзаци (рядки) для різних значень діапазонів кількості слів та кількості літер у кожному слові; налаштування основних параметрів та специфічних, зокрема партиціонування та паралелізму з огляду на характеристики віртуального середовища та локального ресурсу. За досягнутими результатами детально проаналізовано запропоновані методи, впроваджені для покращення продуктивності досліджуваних фреймворків із рекомендаціями вибору оптимальних значень параметрів сегментування даних та конфігураційних параметрів. Висновки. Упровадження запропонованих методів налаштування конфігураційних параметрів Spark та Hadoop дає змогу підвищити продуктивність оброблення даних: для невеликих файлів (0,5–1 ГБ) у середньому до 25–30%; для великих (1,5–2,5 ГБ) – у середньому до 10–20 %. Водночас середнє значення часу виконання одного завдання зменшилося на 10–15 % для файлів різних розмірів та з різною кількістю слів у рядку.

Abstract The decision tree is a widely used decision support model, which can quickly mine effective decision rules based on the dataset. The decision tree induction algorithm for continuous-valued attributes, based on unbalanced cut points, is efficient for mining decision rules; however, extending it to big data remains an unresolved. In this paper, two solutions are proposed to solve this problem: the first one is based on partitioning instance subsets, whereas the second one uses partitioning attribute subsets. The crucial of these two solutions is how to find the global optimal cut point from the set of local optimal cut points. For the first solution, the calculation of the Gini index of the cut points between computing nodes and the selection of the global optimal cut point by communication between these computing nodes is proposed. However, in the second solution, the division of the big data into subsets using attribute subsets in a way that all cut points of an attribute are on the same map node is proposed, the local optimal cut points can be found in this map node, then the global optimal cut point can be obtained by summarizing all local optimal cut points in the reduce node. Finally, the proposed solutions are implemented with two big data platforms, Hadoop and Spark, and compared with three related algorithms on four datasets. Experimental results show that the proposed algorithms can not only effectively solve the scalability problem, but also have lowest running time, the fastest speed and the highest efficiency under the premise of preserving the classification performance.

This paper proposes a performance optimization strategy for Spark-based machine learning algorithms in Shuffle and memory data management modules. The Shuffle module is optimized by introducing Observer monitoring module in Spark cluster to achieve task status monitoring and dynamic ShuffleWrite task generation. Meanwhile, an adaptive caching mechanism for RDD data addresses the lack of in-memory data caching. The performance-optimized algorithm performs well in the experiments, with a clustering accuracy of 89% and a response time that is 5% faster than the Random Forest algorithm. In road network traffic state discrimination, the optimized algorithm’s classification decision F-measure value is as high as 99.53%, which is 5.32% higher than that before unoptimization, and the running time is 767 seconds less than that of the unoptimized algorithm when dealing with about 6,880,000 pieces of data, which significantly improves the efficiency and accuracy.

In today’s era of digital development of big data information, cross-cultural and cross-regional communication are becoming more and more frequent, and human beings have the right and desire to enjoy certain common cultural information, and it has been 38 years since China and Thailand established diplomatic relations in 1975. Cultural differences in Sino-Thai translation based on big data are studied in four aspects: inter-translation of personal names, inter-translation of geographical names, inter-translation of quantity sets, and inter-translation of date and time. This paper focuses on the research of cultural differences in Sino-Thai mutual translation based on big data analysis, constructing a bilingual word translation model with monolingual word vectors under the big data platform Spark, and conducting data analysis on the accuracy and efficiency of translation by collecting Chinese dataset and Thai dataset in parallelization algorithm in two structures SchemeA and SchemeB . The results show that the accuracy of the bilingual word vectors generated by either SchemeA or SchemeB improves as the size of the dataset increases. In addition, SchemeA is better than SchemeB in terms of accuracy and SchemeB is better than SchemeA in terms of efficiency performance, so Sche A me is suitable for translation tasks with high accuracy requirements, while Sche B me is suitable for translation tasks on large data sets. This study can improve the efficiency and accuracy of mutual translation between China and Thailand and train more translators. Today’s high technology brings much convenience to people, and social knowledge is constantly updated and developed. It is necessary to study in order to keep up with the changes of the times and to grasp the updates of the words of the times in order to translate well and more accurately, and this study not only contributes to the translation work in China but also has an important historical significance to the cultural exchange and communication between China and Thailand.

With the popularization of big data technology, agricultural data processing systems have become more intelligent. In this study, a data processing method for farmland environmental monitoring based on improved Spark components is designed. It introduces the FAST-Join (Join critical filtering sampling partition optimization) algorithm in the Spark component for equivalence association query optimization to improve the operating efficiency of the Spark component and cluster. The experimental results show that the amount of data written and read in Shuffle by Spark optimized by the FAST-join algorithm only accounts for 0.958 and 1.384% of the original data volume on average, and the calculation speed is 202.11% faster than the original. The average data processing time and occupied memory size of the Spark cluster are reduced by 128.22 and 76.75% compared with the originals. It also compared the cluster performance of the FAST-join and Equi-join algorithms. The Spark cluster optimized by the FAST-join algorithm reduced the processing time and occupied memory size by an average of 68.74 and 37.80% compared with the Equi-join algorithm, which shows that the FAST-join algorithm can effectively improve the efficiency of inter-data table querying and cluster computing.

Mosaicking is a crucial step in the application of remote sensing images. The amount of remote sensing image data has grown rapidly, along with the expansion of observed areas and increased image resolution. As a result, traditional serial mosaicking techniques are facing significant challenges. In recent times, various studies have utilized high-performance computing to hasten image mosaicking and attain favorable outcomes. Nevertheless, the current research only accelerates mosaicking through external technology, without optimizing from the perspective of algorithm flow, which introduces unnecessary data I/O and slows down the mosaicking. This paper introduces a rapid parallel remote sensing image mosaicking algorithm utilizing read filtering. To begin with, the target images are divided into blocks and stored in a distributed file system. Subsequently, the image blocks are read and filtered based on a designated input format. Finally, the overlapping and non-overlapping areas are read and processed asynchronously, reducing the data I/O and computing overhead, thereby improving the efficiency of parallel computing. The experiments indicate that the mosaicking algorithm introduced in this paper enhances throughput and speedup by an average of 1.38 MB/S and 0.87 relative to the current techniques, respectively, concerning various datasets and cores. This study provides a theoretical foundation and novel ideas for processing remote sensing images on cluster platforms. [ABSTRACT FROM AUTHOR]

This study introduces a novel methodology designed to assess the accuracy of data processing in the Lambda Architecture (LA), an advanced big-data framework qualified for processing streaming (data in motion) and batch (data at rest) data. Distinct from prior studies that have focused on hardware performance and scalability evaluations, our research uniquely targets the intricate aspects of data-processing accuracy within the various layers of LA. The salient contribution of this study lies in its empirical approach. For the first time, we provide empirical evidence that validates previously theoretical assertions about LA, which have remained largely unexamined due to LA's intricate design. Our methodology encompasses the evaluation of prospective technologies across all levels of LA, the examination of layer-specific design limitations, and the implementation of a uniform software development framework across multiple layers. Specifically, our methodology employs a unique set of metrics, including data latency and processing accuracy under various conditions, which serve as critical indicators of LA's accurate data-processing performance. Our findings compellingly illustrate LA's "eventual consistency". Despite potential transient inconsistencies during real-time processing in the Speed Layer (SL), the system ultimately converges to deliver precise and reliable results, as informed by the comprehensive computations of the Batch Layer (BL). This empirical validation not only confirms but also quantifies the claims posited by previous theoretical discourse, with our results indicating a 100% accuracy rate under various severe data-ingestion scenarios. We applied this methodology in a practical case study involving air/ground surveillance, a domain where data accuracy is paramount. This application demonstrates the effectiveness of the methodology using real-world data-intake scenarios, therefore distinguishing this study from hardware-centric evaluations. This study not only contributes to the existing body of knowledge on LA but also addresses a significant literature gap. By offering a novel, empirically supported methodology for testing LA, a methodology with potential applicability to other big-data architectures, this study sets a precedent for future research in this area, advancing beyond previous work that lacked empirical validation. [ABSTRACT FROM AUTHOR]

Every day, hundreds of thousands of vehicles, including buses, taxis, and ride-hailing cars, continuously generate GPS positioning records. Simultaneously, the traffic big data platform of urban transportation systems has already collected a large amount of GPS trajectory datasets. These incremental and historical GPS datasets require more and more storage space, placing unprecedented cost pressure on the big data platform. Therefore, it is imperative to efficiently compress these large-scale GPS trajectory datasets, saving storage cost and subsequent computing cost. However, a set of classical trajectory compression algorithms can only be executed in a single-threaded manner and are limited to running in a single-node environment. Therefore, these trajectory compression algorithms are insufficient to compress this incremental data, which often amounts to hundreds of gigabytes, within an acceptable time frame. This paper utilizes Spark, a popular big data processing engine, to parallelize a set of classical trajectory compression algorithms. These algorithms consist of the DP (Douglas–Peucker), the TD-TR (Top-Down Time-Ratio), the SW (Sliding Window), SQUISH (Spatial Quality Simplification Heuristic), and the V-DP (Velocity-Aware Douglas–Peucker). We systematically evaluate these parallelized algorithms on a very large GPS trajectory dataset, which contains 117.5 GB of data produced by 20,000 taxis. The experimental results show that: (1) It takes only 438 s to compress this dataset in a Spark cluster with 14 nodes; (2) These parallelized algorithms can save an average of 26% on storage cost, and up to 40%. In addition, we design and implement a pipeline-based solution that automatically performs preprocessing and compression for continuous GPS trajectories on the Spark platform. [ABSTRACT FROM AUTHOR]

The proliferation of semantic data in the form of Resource Description Framework (RDF) triples demands an efficient, scalable, and distributed storage along with a highly available and fault-tolerant parallel processing strategy. There are three open issues with distributed RDF data management systems that are not well addressed altogether in existing work. First is the querying efficiency, second is that solutions are optimized for certain types of query patterns and don't necessarily work well for all types, and third is concerned with reducing pre-processing cost. More precisely, the rapid growth of RDF data raises the need for an efficient partitioning strategy over distributed data management systems to improve SPARQL (SPARQL Protocol and RDF Query Language) query performance regardless of its pattern shape with minimized pre-processing time. In distributed RDF systems, both the data and the query processing are highly distributed. On the other hand, SPARQL workloads are dynamic and structurally diverse that can have different degrees of complexity. A complex SPARQL query over a large RDF graph in distributed systems requires combining a lot of distributed pieces of data through join operations. Therefore, designing an efficient data-partitioning schema and join strategy to minimize data transfer is the fundamental challenge in distributed RDF data management systems. In this context, we propose a new relational partitioning schema called Property Table Partitioning (PTP) for RDF data, that further partitions existing Property Table into multiple tables based on distinct properties (comprising of all subjects with non-null values for those distinct properties) in order to minimize input size and number of join operations of a query. This paper proposed a distributed RDF data management system called S3QLRDF, which is built on top of Spark and utilizes SQL to execute SPARQL queries over PTP schema. The experimental analysis with respect to preprocessing costs and query performance, using synthetic and real datasets shows that S3QLRDF outperforms state-of-the-art distributed RDF management systems. [ABSTRACT FROM AUTHOR]

For big data applications, it is important to allocate resources reasonably and schedule tasks effectively. As one of the popular big data processing frameworks, the default scheduling strategy of Spark still suffers from low resource utilization and high resource cost. In this paper, a low-cost task scheduling algorithm for Spark based on heterogeneous cloud environment is proposed to minimize cost while improving resource utilization. First of all, we construct a cost model for Spark based on the hierarchical relationship among applications, jobs, stages, and tasks. Then, based on the model, a low-cost task scheduling algorithm is proposed, which improves the utilization of computational resources by adjusting the task parallelism and achieves task scheduling with priority based on the distribution of data to be computed. We also propose a Reduce task load balancing partitioning algorithm (RTLBPA) based on prior information of data by dispersing and aggregating different keys with the same hash value in the data into different partitions. This algorithm can achieve load balancing and improve the execution time of the job by reducing the whole Reduce phase. Finally, we performed extensive experiments on the proposed algorithm using Hibench's workloads in the cloud environment. The result shows that the cost can be reduced by at least 22.71% compared with the existing algorithm under different workloads. As a result, the proposed algorithm can improve the cost efficiency of Spark cluster effectively while improving resource utilization. [ABSTRACT FROM AUTHOR]

Copyright of Frontiers of Information Technology & Electronic Engineering is the property of Springer Nature and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

The decision tree is a widely used decision support model, which can quickly mine effective decision rules based on the dataset. The decision tree induction algorithm for continuous-valued attributes, based on unbalanced cut points, is efficient for mining decision rules; however, extending it to big data remains an unresolved. In this paper, two solutions are proposed to solve this problem: the first one is based on partitioning instance subsets, whereas the second one uses partitioning attribute subsets. The crucial of these two solutions is how to find the global optimal cut point from the set of local optimal cut points. For the first solution, the calculation of the Gini index of the cut points between computing nodes and the selection of the global optimal cut point by communication between these computing nodes is proposed. However, in the second solution, the division of the big data into subsets using attribute subsets in a way that all cut points of an attribute are on the same map node is proposed, the local optimal cut points can be found in this map node, then the global optimal cut point can be obtained by summarizing all local optimal cut points in the reduce node. Finally, the proposed solutions are implemented with two big data platforms, Hadoop and Spark, and compared with three related algorithms on four datasets. Experimental results show that the proposed algorithms can not only effectively solve the scalability problem, but also have lowest running time, the fastest speed and the highest efficiency under the premise of preserving the classification performance. [ABSTRACT FROM AUTHOR]

There are still many defects in the process of data processing, storage and analysis, which cannot adapt to the needs of big data processing and analysis. In order to more fully reflect the value of big data processing, it is necessary to realize efficient data processing. Therefore, people invented the Spark operating system and integrated the idea of SIMBA, built a big data analysis system, and searched the information through the database query method. In Spark operating system can import search mechanism, in order to improve the search effect of information, through the establishment of segment tree storage method can improve the retrieval effect, in big data analysis can use partition strategy method for big data partition, and adopted the global filtering and local search method, make in information retrieval can achieve high throughput and low delay performance. [ABSTRACT FROM AUTHOR]

Copyright of Telecommunications Science is the property of Beijing Xintong Media Co., Ltd. and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

The frequently difficult process of examining large and diverse amounts of information is known as "big data analysis." The goal is to find insights, such as hidden patterns, unexpected correlations, or market trends like consumer preferences. Thus, these insights can help companies make quick and informed business decisions. First, we suggest a manual database schema creation technique to reduce the response time of the data analysis process in the context of big data under the Spark platform. Then, in order to distribute the activities to be performed on the Spark slots more evenly, we suggest a data allocation technique. The data is then saved in RAM, using the cache for faster reading. Finally, to solve the problems of loading data into the cache, we suggest using a different file format called "Parquet." We evaluated our strategy using the "fire calls for service" (CFS) data set, which led to a 97.77% faster response time. This demonstrates how the suggested optimization options considerably reduce the time needed to import the cache and read the database. [ABSTRACT FROM AUTHOR]

The term "big data" refers to the vast amount of structured and unstructured data generated by businesses, organizations, and individuals on a daily basis. The rapid growth of big data has led to the development of new technologies and techniques for storing, processing, and analyzing these data in order to extract valuable information. This study examines some of these technologies, compares their pros and cons, and provides solutions for handling specific types of reporting using big data tools. In addition, this paper discusses some of the challenges associated with big data and suggests approaches that could be used to manage and analyze these data. The findings demonstrate the benefits of efficiently managing the datasets and choosing the appropriate tools, as well as the efficiency of the proposed solution with hands-on examples. [ABSTRACT FROM AUTHOR]

Simultaneously, the application of resilient distributed datasets (RDD) in cloud computing provides a good environment for data analysis of big data. In addition, the combination of Machine Learning (ML) algorithms of the edge computing paradigm and the SFUP-SP algorithm may be able to also be used to improve local computing capabilities and speed up data analysis and user decision-making. [ABSTRACT FROM AUTHOR]

The Join task between Spark large tables takes a long time to run and produces a lot of disk I/O, network I/O and disk occupation in the Shuffle process. This paper proposes a lightweight distributed data filtering model that combines broadcast variables and accumulators using RoaringBitmap. When the data in the two tables are not exactly matched, the dimension table Key is collected through the accumulator, compressed by RoaringBitmap and distributed to each node using broadcast variables. The distributed fact table data can be pre-filtered on the local server, which effectively reduces the data transmission and disk reading and writing in the Shuffle phase. Experimental results show that this optimization method can reduce disk usage, shorten the running time and reduce network I/O and disk I/O for Spark Join tasks in the case of massive data, and the effect is more obvious when the two tables have a higher incomplete matching degree or a fixed matching degree but a larger amount of data. This optimization scheme has the advantages of being easy to use, being easy to maintain and having an obvious effect, and it can be applied to many development scenarios. [ABSTRACT FROM AUTHOR]

The evolution of technologies has unleashed a wealth of challenges by generating massive amount of data. Recently, biological data has increased exponentially, which has introduced several computational challenges. DNA short read alignment is an important problem in bioinformatics. The exponential growth in the number of short reads has increased the need for an ideal platform to accelerate the alignment process. Apache Spark is a cluster-computing framework that involves data parallelism and fault tolerance. In this article, we proposed a Spark-based algorithm to accelerate DNA short reads alignment problem, and it is called Spark-DNAligning. Spark-DNAligning exploits Apache Spark ’s performance optimizations such as broadcast variable, join after partitioning, caching, and in-memory computations. Spark-DNAligning is evaluated in term of performance by comparing it with SparkBWA tool and a MapReduce based algorithm called CloudBurst. All the experiments are conducted on Amazon Web Services (AWS). Results demonstrate that Spark-DNAligning outperforms both tools by providing a speedup in the range of 101–702 in aligning gigabytes of short reads to the human genome. Empirical evaluation reveals that Apache Spark offers promising solutions to DNA short reads alignment problem.

To solve the problem of low efficiency in real-time processing and matching of CNAME records in massive DNS log data, a parallel AC automaton enhancement method based on Spark was proposed. The method is based on the Spark distributed cluster computing engine of Hadoop, which ensures the stability of massive DNS log data storage with high fault tolerance and 24-hour real-time processing. At the same time, the Spark distributed cluster uses the multi-thread parallel computing method combined with the improved AC automaton algorithm, which not only reduces the memory occupied by trie construction, but also improves the efficiency of rapid matching of CNAME records of massive DNS logs. Simulation results show that the proposed method can quickly match CNAME records of massive DNS log data. Compared with the original AC algorithm, the efficiency is significantly improved, and the time complexity and storage space are reduced.

In lunar exploration missions, path planning for lunar rovers using digital elevation models (DEMs) is currently a hot topic in academic research. However, research on path planning using large-scale DEMs has rarely been discussed, owing to the low time efficiency of existing algorithms. Therefore, in this article, we propose a fast path-planning method using a distributed tile pyramid strategy and an improved A* algorithm. The proposed method consists of three main steps. First, the tile pyramid is generated for the large lunar DEM and stored in Hadoop distributed file system. Second, a distributed path-planning strategy based on tile pyramid (DPPS-TP) is used to accelerate path-planning tasks on large-scale lunar DEMs using Spark and Hadoop. Finally, an improved A* algorithm was proposed to improve the speed of the path-planning task in each tile. The method was tested using lunar DEM images. Experimental results demonstrate that: in a single-machine serial strategy using source DEM generated by the Chang'e-2 CCD stereo camera, the proposed A* algorithm for open list and closed list with random access feature (OC-RA-A* algorithm) is 3.59 times faster than the traditional A* algorithm in long-distance path planning tasks and compared to the distributed parallel computation strategy using source DEM generated by the Chang'e-2 CCD stereo camera, the proposed DPPS-TP based on tile pyramid DEM is 113.66 times faster in the long-range path planning task.

Diabetic Retinopathy is a significant complication of diabetes, caused by a high blood sugar level, which damages the retina. In its earliest stages, diabetic retinopathy is asymptomatic and can lead to blindness if not discovered and treated promptly. As a result, there is a need for a reliable screening method. According to studies, this problem affects a large section of the population, and it is thus linked to Big Data. There are several obstacles and issues with Big Data, but Deep Learning is providing solutions to these issues. As a result, academics are extremely interested in Big Data with Deep Learning. It has been our goal in this study to employ effective preprocessing and Deep Learning approaches to accomplish binary classification of Diabetic Retinopathy. The experiment is done out using a dataset from Kaggle that was collected from India. The peculiarity of the paper is that the work is implemented on the Spark platform, and the performance of three models, InceptionV3, Xception, and VGG19 with the Logistic Regression classifier is compared. The accuracy of the models is used as a comparison criterion. Based on the results of the trial, the accuracy of InceptionV3 is 95 percent, the accuracy of Xception is 92.50 percent, and the accuracy of VGG19 is 89.94 percent. Consequently, InceptionV3 outperforms the other two models.

In this study, the effects of different engine speed values on performance and combustion characteristics were investigated by converting a diesel engine to a spark-ignition engine using natural gas. In numerical analysis, G-equation combustion model, reduced methane chemical kinetic mechanism that represent natural gas consisting of 29 types and 171 equations, and RANS k-e turbulence model were used. Analyzes were performed at full load, 17.5:1 compression ratio, constant ignition timing, and 6 different engine speeds. In order to examine only the effect of speed, the initial value, boundary conditions and spark plug ignition time were considered constant. While engine power and fuel consumption increased with increasing engine speed, engine efficiency decreased. In addition, increasing engine speed also increased the ignition delay time and combustion duration, and the flame front reached the squish zone later.

In recent days, social media, online services, smartphones, and the Internet of Things (IoT) produces large quantities of data every second. The generated data is structured, unstructured, or semi-structured and available in various formats. Therefore, traditional approaches are not sufficient to handle such kind of data effectively. High utility pattern mining is a famous study area of data analytics that incorporates utility measures to consider user-based constraints such as number of units and benefit, in addition to frequency statistics of datasets. It is also essential to make effective decisions, and their demand is increasing in the last decade. Several techniques have been suggested for utility-based frequent pattern extraction. However, these approaches are limited to data size and operate on standalone systems. We have proposed a parallel approach named distributed memory-optimized utility mining (DMOUM) for high utility-based frequent pattern mining to address this issue. A new pruning technique is implemented that greatly reduces processing time and memory use. The proposed approach can handle huge amounts of data, i.e., big data. DMOUM approach is implemented in cluster-node architecture using Spark. The results are validated on various real-time datasets and found proposed approach has better performance than cutting edge approaches in terms of run-time, memory consumption, and scalability measures. The proposed work will provide solutions to real-time issues such as health care, education, e-commerce, and so on.

The world has been observing unprecedented circumstances with the outbreak of the COVID-19 pandemic. The upsurge of infection cases has left hospitals overwhelmed, education quality compromised, caused unemployment issues, and affected the international economy, tourism sector, and financial markets. The number of confirmed cases and deaths has been increasing on daily basis. However, a vaccination drive has been conducted globally to reduce the transmission. The purpose of this study was to present the basic system configuration of the Big Data technology called Spark and to performe an analysis of Thailand’s COVID-19 pandemic data. Data were collected from an open Thai government data center between January 2020 and 28. May 2021. Bangkok province has the most COVID-19 cases followed by Samut Sakhon province. It has been observed that the ages between 20 and 40 were the most infected by COVID-19 in Thailand.

In this paper, we introduce MARVEL, a system designed to simplify the teaching of MapReduce, a popular distributed programming paradigm, through software visualization. At its core, it allows a teacher to describe and recreate a MapReduce application by interactively requesting, through a graphical interface, the execution of a sequence of MapReduce transformations that target an input dataset. Then, the execution of each operation is illustrated on the screen by playing an appropriate graphical animation stage, highlighting aspects related to its distributed nature. The sequence of all animation stages, played back one after the other in a sequential order, results in a visualization of the whole algorithm. The content of the resulting visualization is not simulated or fictitious, but reflects the real behavior of the requested operations, thanks to the adoption of an architecture based on a real instance of a distributed system running on Apache Spark. On the teacher's side, it is expected that by using MARVEL he/she will spend less time preparing materials and will be able to design a more interactive lesson than with electronic slides or a whiteboard. To test the effectiveness of the proposed approach on the learner side, we also conducted a small scientific experiment with a class of volunteer students who formed a control group. The results are encouraging, showing that the use of software visualization guarantees students a learning experience at least equivalent to that of conventional approaches. [ABSTRACT FROM AUTHOR]