Search

Recently, software defined networking (SDN) has been deployed extensively in diverse practical domains, providing a new direction in network management by separating the control plane from the data plane. Nevertheless, SDN is vulnerable to distributed denial of service (DDoS) attacks resulting from its centralized controller. Several studies have been suggested to address the DDoS attacks in SDN utilizing machine learning approaches. However, these approaches are resource-intensive and cause performance degradation since they cannot perform effectively in large-scale SDN networks that generate vast traffic statistics. To handle all these challenges, we build a DDoS attack detection model in SDN using Spark as a big data tool to overcome the limitations of conventional data processing methods. Four machine learning algorithms are employed. The decision tree (DT) is elected to be used for real-time deployment based on the performance results, which indicates that it has the best accuracy of 0.936. The model performance is compared with state-of-the-art and shows an overall better performance.

[EN] The Large Hadron Collider (LHC) at CERN has generated a vast amount of information from physics events, reaching peaks of TB of data per day which are then sent to large storage facilities. Traditionally, data processing workflows in the High Energy Physics (HEP) field have leveraged grid computing resources. In this context, users have been responsible for manually parallelising the analysis, sending tasks to computing nodes and aggregating the partial results. Analysis environments in this field have had a common building block in the ROOT software framework. This is the de facto standard tool for storing, processing and visualising HEP data. ROOT offers a modern analysis tool called RDataFrame, which can parallelise computations from a single machine to a distributed cluster while hiding most of the scheduling and result aggregation complexity from users. This is currently done by leveraging Apache Spark as the distributed execution engine, but other alternatives are being explored by HEP research groups. Notably, Dask has rapidly gained popularity thanks to its ability to interface with batch queuing systems, widespread in HEP grid computing facilities. Furthermore, future upgrades of the LHC are expected to bring a dramatic increase in data volumes. This paper presents a novel implementation of the Dask backend for the distributed RDataFrame tool in order to address the aforementioned future trends. The scalability of the tool with both the new backend and the already available Spark backend is demonstrated for the first time on more than two thousand cores, testing a real HEP analysis., Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature. This work benefited from the support of grant PID2020-113656RBC22 funded by Ministerio de Ciencia e Innovacion (Spain) MCIN/AEI/10.13039/501100011033.

Frequent itemset mining (FIM) is a crucial tool for identifying hidden patterns in information. FP-Growth is an FIM algorithm used to find associations. When the data size increases, the execution of FIM algorithms on a single machine suffers from computational problems, such as memory and time consumption. For these reasons, parallel and distributed processing on platforms such as Spark is essential. The parallel frequent pattern (PFP) is the implementation of FP-Growth in Spark. The main problem with PFP is that it does not consider the load balancing between cluster units. This research proposes an enhanced balanced parallel frequent pattern “EBPFP” algorithm to enhance and balance the PFP. The proposed algorithm (EBPFP) proposes two ideas. First, a strategy for load balancing between groups is proposed to ensure that the items are evenly divided between the nodes, and the cluster resources are used more effectively. Second, the improved conditional pattern base (ICPB) method aims to remove infrequent items from the conditional pattern base before constructing local FP-Trees. The experimental results show that the proposed EBPFP algorithm outperforms PFP, and the difference in running time between EBPFP and PFP was 21.56% and 39.72%, respectively.

This paper aims to estimate the effectiveness of the Priority Development Areas (PDA) program in Russian monotowns at the firm level. Using data from the SPARK-Interfax database on Russian companies in 2014–2018, the authors estimate the effect of the PDA residency on the firms’ revenue growth in monotowns in Russia. The authors test two hypotheses. First, the status of the PDA resident is obtained by firms that have been successful in previous periods. Second, the PDA resident status does not have a positive effect on firm growth. To measure the treatment effect, the authors use the nearest neighbor propensity score matching method, which allows drawing conclusions about causality, as opposed to the ordinary least squares (OLS) method. The regressions consider the firm size, industry, and geographic location of the city. According to the results of the study, both hypotheses are confirmed. Resident status is determined by the gains in the company’s revenue for the two previous periods (the coefficients are significant in all specifications at the 5% level). The influence of residency on the firm’s revenue growth is neglected (becomes insignificant) when comparing enterprises with the same pre-2015 trends. Thus, the authors conclude that the success of the PDA program in Russian monotowns in terms of business support is questionable. The findings of the study are valid for enterprises established before 2015.

With the huge increase of large-scale multimedia over Internet, especially images, building Content-Based Image Retrieval (CBIR) systems for large-scale images has become a big challenge. One of the drawbacks associated with CBIR is the very long execution time. In this article, we propose a fast Content-Based Image Retrieval system using Spark (CBIR-S) targeting large-scale images. Our system is composed of two steps. (i) image indexation step, in which we use MapReduce distributed model on Spark in order to speed up the indexation process. We also use a memory-centric distributed storage system, called Tachyon, to enhance the write operation (ii) image retrieving step which we speed up by using a parallel k-Nearest Neighbors (k-NN) search method based on MapReduce model implemented under Apache Spark, in addition to exploiting the cache method of spark framework. We have showed, through a wide set of experiments, the effectiveness of our approach in terms of processing time.

In recent years, with the widespread use of location-aware handheld devices and the development of wireless networks, trajectory data have shown a trend of rapid growth in data volume and coverage, which has led to the prosperous development of location-based services (LBS). Spatio-temporal range query, as the basis of many services, remains a challenge in supporting efficient analysis and calculation of data, especially when large volumes of trajectory data have been accumulated. We propose a Hilbert-GeoSOT spatio-temporal meshing and coding method called HGST to improve the efficiency of spatio-temporal range queries on massive trajectory data. First, the method uses Hilbert to encode the grids obtained based on the GeoSOT space division model, and then constructs a unified time division standard to generate the space–time location identification of trajectory data. Second, this paper builds a novel spatio-temporal index to organize trajectory data, and designs an adaptive spatio-temporal scaling and coding method based on HGST to improve the query performance on indexed records. Finally, we implement a prototype system based on HBase and Spark, and develop a Spark-based algorithm to accelerate the spatio-temporal range query for huge trajectory data. Extensive experiments on a real taxi trajectory dataset demonstrate that HGST improves query efficiency levels by approximately 14.77% and 34.93% compared with GeoSOT-ST and GeoMesa at various spatial scales, respectively, and has better scalability under different data volumes.

The advent of the era of big remote sensing data has transformed traditional data management and analysis models, among which visualization analysis has gradually become an effective method, and map tiles for remote sensing data have always played an important role. However, in high-latitude regions, especially in polar regions, the deformation caused by map projection still exists, which lowers the accuracy of global or large-scale visual analysis, as well as the execution efficiency of big data. To solve the above problems, this paper proposes an algorithm called HexTile, which uses a hexagonal discrete global grid system (DGGS) model to effectively avoid problems caused by map projection and ensure global consistency. At the same time, the algorithm was implemented based on the Spark platform, which also has advantages in efficiency. Based on the DGGS model, hierarchical hexagon map tile construction and a visualization algorithm were designed, including hexagonal slicing, merging, and stitching. The above algorithms were parallelized in Spark to improve the big data execution efficiency. Experiments were carried out with Landsat-8, and the results show that the HexTile algorithm can not only guarantee the quality of global data, but also give full play to the advantages of the cluster in terms of efficiency. Additionally, the visualization was conducted with Cesium and OpenLayers to validate the integration and completeness of hexagon tiles. The scheme proposed in this paper could provide a reference for spatiotemporal big data visualization technology.

Big Data is a phenomenon that affects today’s world, with new data being generated every second. Today’s enterprises face major challenges from the increasingly diverse data, as well as from indexing, searching, and analyzing such enormous amounts of data. In this context, several frameworks and libraries for processing and analyzing Big Data exist. Among those frameworks Hadoop MapReduce, Mahout, Spark, and MLlib appear to be the most popular, although it is unclear which of them best suits and performs in various data processing and analysis scenarios. This paper proposes EverAnalyzer, a self-adjustable Big Data management platform built to fill this gap by exploiting all of these frameworks. The platform is able to collect data both in a streaming and in a batch manner, utilizing the metadata obtained from its users’ processing and analytical processes applied to the collected data. Based on this metadata, the platform recommends the optimum framework for the data processing/analytical activities that the users aim to execute. To verify the platform’s efficiency, numerous experiments were carried out using 30 diverse datasets related to various diseases. The results revealed that EverAnalyzer correctly suggested the optimum framework in 80% of the cases, indicating that the platform made the best selections in the majority of the experiments.

In response to the rapid growth of many sorts of information, highway data has continued to evolve in the direction of big data in terms of scale, type, and structure, exhibiting characteristics of multi-source heterogeneous data. The k-nearest neighbor (KNN) join has received a lot of interest in recent years due to its wide range of applications. Processing KNN joins is time-consuming and inefficient due to the quadratic structure of the join method. As the number of applications dealing with vast amounts of data develops, KNN joins get more sophisticated. The authors seek to save money on computer resources by leveraging a large number of threads and multiprocessors. Six popular datasets are used to apply the method and evaluate the sequential and parallel performance of the KNN technique. These datasets are used to compare the sequential and parallel performance of the KNN method. When compared to a matching multi-core solution, the final implementation saves computing resources. It has been optimized to utilize as little RAM as possible, allowing it to manage high-resolution photo data without sacrificing efficiency. The authors will use the technique they presented using Spark Radoop. Our performance research validates the supplied method’s efficacy and scalability.

K-Means clustering algorithm is a typical unsupervised learning method, and it has been widely used in the field of fault diagnosis. However, the traditional serial K-Means clustering algorithm is difficult to efficiently and accurately perform clustering analysis on the massive running-state monitoring data of rolling bearing. Therefore, a novel fault diagnosis method of rolling bearing using Spark-based parallel ant colony optimization (ACO)-K-Means clustering algorithm is proposed. Firstly, a Spark-based three-layer wavelet packet decomposition approach is developed to efficiently preprocess the running-state monitoring data to obtain eigenvectors, which are stored in Hadoop Distributed File System (HDFS) and served as the input of ACO-K-Means clustering algorithm. Secondly, ACO-K-Means clustering algorithm suitable for rolling bearing fault diagnosis is proposed to improve the diagnosis accuracy. ACO algorithm is adopted to obtain the global optimal initial clustering centers of K-Means from all eigenvectors, and the K-Means clustering algorithm based on weighted Euclidean distance is used to perform clustering analysis on all eigenvectors to obtain a rolling bearing fault diagnosis model. Thirdly, the efficient parallelization of ACO-K-Means clustering algorithm is implemented on a Spark platform, which can make full use of the computing resources of a cluster to efficiently process large-scale rolling bearing datasets in parallel. Extensive experiments are conducted to verify the effectiveness of the proposed fault diagnosis method. Experimental results show that the proposed method can not only achieve good fault diagnosis accuracy but also provide high model training efficiency and fault diagnosis efficiency in a big data environment.

Viewshed analysis is an indispensable part of digital terrain analysis and widely used in many application domains. High-resolution raster digital elevation model (DEM) data bring significant computational challenges to the existing viewshed analysis algorithms, which are computationally intensive and require a large memory space and massive computing power. The visibility calculation can be accelerated using Apache Spark. In this article, we present a Spark-based parallel computing approach for the XDraw algorithm, which is composed of a tile-based raster data storing strategy, an equivolume computing strategy, and a stream-merging write-back strategy. The parallel implementation of the XDraw algorithm mainly consists of three parts: partitioning a raster DEM file into square tile sets and reorganizing these tile sets to prevent tile overlap across data divisions of Hadoop Distributed File System, subdividing the DEM into multiple equivolume data sectors according to the viewpoint position, and performing the XDraw algorithm on the corresponding tile sets of each sector independently and writing back the viewshed results efficiently. Experiments on real-world datasets show that the proposed computing approach can achieve higher speedup and efficiency for XDraw viewshed analysis as the raster DEM data volume is dramatically increased. The results also show that the approach has also satisfactory scalability as the number of data nodes in clusters is increased.

Data skew has an essential impact on the performance of big data processing. Spark task scheduling with data skew and deadline constraints is considered to minimize the total rental cost in this paper. A modified scheduling architecture is developed in terms of the unique characteristics of the considered problem. A mathematical model is constructed, and a Spark task scheduling algorithm is proposed considering both the data skew and deadline constraints. The algorithm consists of three components: stage sequencing, task scheduling, and scheduling adjustment. Strategies for each of the components are presented. The parameters and components of the proposed algorithm are calibrated over many random instances. The calibrated algorithm is compared to two existing algorithms for similar problems over classical scientific workflow applications. Experimental results show that the proposed algorithm outperforms the compared algorithms statistically.

For more than a decade, “big data” has been an industry and academia buzz phrase. Over this time, many companies adopted Apache Hadoop and Spark frameworks for their massive data storage and analysis efforts, using powerful, energy-hungry, general-purpose server as their big data processing platforms. But not all industry or academic fields want, or even need, such large systems. Moreover, capital costs aside, power consumption has also become a primary data center concern. Consequently, lower-cost, lower-power microservers have emerged as viable alternatives in many settings. Now, the latest generation Raspberry Pi (RPi), model 4B, exhibits significant computational performance improvements over its predecessors, and is presently considered a sufficiently powerful single board computer (SBC) to run many mainstream operating systems and accommodate heavy workloads. This paper reexamines SBC cluster big data processing possibilities by integrating the most powerful (presently) RPi model–the RPi 4B with 4 Gigabytes (GB) main memory. We examine external storage’s performance impact on such an SBC cluster’s big data processing performance by employing three different external storage solutions with measurably distinct performance characteristics. Moreover, we discuss challenges we encountered and identify further SBC cluster performance optimizations. We perform several representative big data application benchmarks and measure various key performance metrics such as execution time, power consumption, throughput, performance-per-dollars, etc. Our extensive experiments and comprehensive studies conclude this current, fourth-generation RPi has evolved to become the first generation to effectively run massive (i.e., more than 100GB) workloads in big data processing applications.

This Conference paper was published as book chapter Applications of AI and Machine Learning ICAML2020

Using software applications or services, which provide word or even word pattern recommendation service has become part of our lives. Those services appear in many form in our daily basis, just think of our smartphones keyboard, or Google search suggestions and this list can be continued. With the help of these tools, we can not only find the suitable word that fits into our sentence, but we can also express ourselves in a much more nuanced, diverse way. To achieve this kind of recommendation service, we use an algorithm which is capable to recommend word by word pattern queries. Word pattern queries, can be expressed as a combination of words, part-of-speech (POS) tags and wild card words. Since there are a lot of possible patterns and sentences, we use Big Data frameworks to handle this large amount of data. In this paper, we compared two popular framework Hadoop and Spark with the proposed algorithm and recommend some enhancement to gain faster word pattern generation.

Modern organizations typically store their data in a raw format in data lakes. These data are then processed and usually stored under hybrid layouts, because they allow projection and selection operations. Thus, they allow (when required) to read less data from the disk. However, this is not very well exploited by distributed processing frameworks (e.g., Hadoop, Spark) when analytical queries are posed. These frameworks divide the data into multiple partitions and then process each partition in a separate task, consequently creating tasks based on the total file size and not the actual size of the data to be read. This typically leads to launching more tasks than needed, which, in turn, increases the query execution time and induces significant waste of computing resources. To allow a more efficient use of resources and reduce the query execution time, we propose a method that decides the number of tasks based on the data being read. To this end, we first propose a cost-based model for estimating the size of data read in hybrid layouts. Next, we use the estimated reading size in a multi-objective optimization method to decide the number of tasks and computational resources to be used. We prototyped our solution for Apache Parquet and Spark and found that our estimations are highly correlated (0.96) with the real executions. Further, using TPC-H we show that our recommended configurations are only 5.6% away from the Pareto front and provide 2.1 × speedup compared with default solutions.

The Prime of Miss Jean Brodie（1961）では表には出ないBrodieとBrodie組の少女たちの心理的攻防が見られる。そこから特に主要人物であるSandyとMiss Jean Brodieに注目し、それぞれの性格と二人の関係や役割について研究する。作品内での関係の崩壊がどのようなきっかけ、タイミングで起こったかを考察しSandyがMiss Jean Brodieに求めていたことはなんだったのかを追求する。

Privacy protection is an important problem in workflow scheduling, for which scheduling tasks with privacy constraints to reliable resources is essential. In this article, we consider the scheduling problem of Spark applications in a hybrid cloud with deadline and privacy constraints. A scheduling algorithm framework is proposed, which consists of four algorithm components. Candidate strategies are developed for each algorithm component. The components and parameter values are statistically calibrated over a comprehensive set of random instances. The proposed algorithm is compared to modified classical algorithms for similar problems. The experimental results indicate that the proposed algorithm outperforms the compared algorithms under different application scales, deadlines, and private VMs.

A framework for traffic congestion prediction and visualization based on machine learning and Fuzzy Comprehensive Evaluation named MF-TCPV is proposed in this paper. The framework uses DataX and DataV to implement the integration of multi-source heterogeneous traffic data and the visualization of congestion prediction results. A deep prediction model named LSTM-SPRVM based on deep learning algorithms, machine learning algorithms, and Spark parallelization technology for the prediction of traffic congestion features in the future is proposed. In MF-TCPV, traffic congestion is divided into six levels based on Fuzzy Comprehensive Evaluation and traffic congestion features such as average speed, road occupancy rate, and traffic flow density. MF-TCPV is validated based on the real data of Whitemud Drive in Canada. The experimental results demonstrate that MF-TCPV is capable of predicting the traffic congestion accurately and displaying prediction results visually. LSTM-SPRVM is better than other existing deep learning models in terms of prediction accuracy, and MF-TCPV can intuitively visualize the prediction results of traffic congestion.

Electric vehicles (EVs) are increasing popular as clean transportation. However electric vehicle (EV) users often encounter a shortage of electricity in the driving process. Especially when vehicle charging path planning system request reaches peak hour, it will increase the system calculation pressure. To promote penetration and popularity of electric vehicles, it is of critical importance to determine location and size of charging stations more scientifically and improve the computational power of path planning. In this paper, a method is developed for charge warning and path planning of insufficient energy EVs. The method utilizes the appropriate energy consumption factor to construct the electricity early warning model, analyzes the consumption of electricity in the vehicle in real time, and promptly warns the users when the vehicle has insufficient electric energy. Meanwhile, it combines the actual traffic information and map information to construct a network topology based on time weight, queuing mechanism, and charging calculation model to obtain path network model data, charging station queuing times and charging times as parameters. EV path planning problem is solved to minimize the total times of travel using the Dijkstra algorithm with the input path network model data, charging station queuing times and charging times as parameters. At the same time, the Spark computing framework is joined innovatively to improve the computing speed and solve the problem of path planning peak period. The Spark framework is utilized to parallelize the optimized Dijkstra algorithm. The experiment shows that the electric vehicle charging warning and path planning method provides an effective way for drivers of electric vehicles to charge.

Songklanakarin Journal of Science an Technology (SJST), 44, 4, 1119-1124

Los objetivos de este proyecto, tal como se menciona en la introducción, consisten en el análisis y migración de una solución previamente realizada en Python y Cassandra, que actúa como base de datos para las cadenas de ADN, para obtener mayor integración, rendimiento y prestaciones. El objetivo principal será, como se ha comentado en el párrafo superior, la mejora de rendimiento y para ello se realizará una migración a Scala (lenguaje de programación funcional, pionero en el área de Big Data) y se comprobará este hecho llevando a cabo un análisis de la “performance” de ambas soluciones. El rendimiento se verá afectado debido a que la migración supondría poder aprovechar mejor la conexión con Cassandra, ya que nos permite un uso más preciso del conector [8] (estándar de acceso a las bases de datos). En otras palabras, accesos más rápidos debido a la reducción en el tiempo de conexión con la base de datos y sobretodo, en las consultas realizadas. Una vez realizados el análisis y la migración, se proseguirá con una explicación de los resultados en base a los tiempos obtenidos en ambos casos. En conclusión, la migración a Scala, debería aportar una mejora sustancial de rendimiento, prestaciones y productividad debido a que el conector tiene más afinidad con el nuevo lenguaje en el que estará implementado.

With the rapid rate at which networking technologies are changing, there is a need to regularly update network activity datasets to accurately reflect the current state of network infrastructure/traffic. The uniqueness of this work was that this was the first network dataset collected using Zeek and labelled using the MITRE ATT&CK framework. In addition to identifying attack traffic, the MITRE ATT&CK framework allows for the detection of adversary behavior leading to an attack. It can also be used to develop user profiles of groups intending to perform attacks. This paper also outlined how both the cyber range and hadoop’s big data platform were used for creating this network traffic data repository. The data was collected using Security Onion in two formats: Zeek and PCAPs. Mission logs, which contained the MITRE ATT&CK data, were used to label the network attack data. The data was transferred daily from the Security Onion virtual machine running on a cyber range to the big-data platform, Hadoop’s distributed file system. This dataset, UWF-ZeekData22, is publicly available at datasets.uwf.edu.

The continuous evolution of IT (information technology) technologies is radically transforming many technical areas and social aspects, also reshaping the way we behave and looking for entertainment and leisure services. In that context, tourism experiences request to enhance the level of user involvement and integration and to create an ever more personalized and connected experience, by leveraging on the differentiated tourist services and information locally present in the territory, by pushing active participation of customers, and by taking advantage of the ever-increasing presence of sensors and IoT (Internet of Things) devices deployed in many realities. However, the deep fragmentation of services and technologies adopted in tourism context characterizes the whole information provided also by customer sensing and IoTs (Internet of Things) heterogeneity and deep clashes with an effective organization of smart tourism. This article presents APERTO5.0 (an Architecture for Personalization and Elaboration of services and data to Reshape Tourism Offers 5.0), an innovative architecture aiming at a whole integration and deep facilitation of tourism service and information organization and blending, to enable the re-provisioning of novel services as advanced aggregates or re-elaborated ones. The proposed solution will demonstrate its effectiveness in the context of Smart Tourism by choosing the real use case of the “Francigena way” (a pilgrim historical path), the Italian part.

Introduction: Children with autism spectrum disorder (ASD) experience problems in motor skills which can affect other abilities such as academic and daily life skills. In this regard, the cost of interventions and the long distance from the rehabilitation centers put pressure on the family of children with ASD. Previous studies have shown the potential for using telerehabilitation as a more affordable way to provide interventions for children with ASD. Therefore, the present study aimed to investigate the non-attendant family-centered Sports, Play and Active Recreation for Kids (SPARK) motor program intervention to develop gross motor skills in children with a high-function ASD. Materials and Methods: The research is a quasi-experimental study with a pretest-posttest design. The participants were 9 families of children with high-functioning ASD referring to the rehabilitation centers in Tehran, Isfahan, Quds, and Kermanshah provinces, Iran. Telerehabilitation of SPARK motor program involves exercising and playing performed in 24 sessions of 45 minutes each (3 sessions per week) for 8 weeks applied for parents. The test of gross motor development-2 (TGMD-2) was used for data collection. The collected data were analyzed by paired t-test. Results: The results showed that family-based telerehabilitation of the SPARK program significantly improved gross motor skills in children with high-functioning ASD (P

To support the sustainable development of photovoltaic (PV) projects in the context of the carbon neutrality aim, a scientific and reliable evaluation technique is crucial. In this research, an AdaBoost-LS-WSVM intelligent evaluation model built on the Spark platform is suggested to increase evaluation accuracy and timeliness. Firstly, the sustainability evaluation index system of PV projects is constructed from five levels: geographic resource sustainability, technical sustainability, economic sustainability, social sustainability, and environmental sustainability in the context of the carbon neutrality target. Then, the AdaBoost-LS-WSVM intelligent evaluation model with Spark as the platform is constructed, and the wavelet kernel function is applied to the LSSVM model to form the LS-WSVM regression model with stronger nonlinear fitting ability. The learning and training of training samples are completed by the AdaBoost model, and multiple weak LS-WSVM regressors are weighted to get a strong LS-WSVM regressor. The regression model is used for assessing the sustainability of PV projects on Spark Big Data runtime platform. Lastly, the scientific accuracy and reliability of the proposed model is confirmed by a case study, which facilitates a timely and effective assessment of the sustainability of PV projects in the context of carbon neutrality target and can provide scientific and reasonable decision support for the construction of a sustainable development model of PV projects.

Hidden Markov models (HMMs) are one of machine learning algorithms which have been widely used and demonstrated their efficiency in many conventional applications. This paper proposes a modified posterior decoding algorithm to solve hidden Markov models decoding problem based on MapReduce paradigm and spark’s resilient distributed dataset (RDDs) concept, for large-scale data processing. The objective of this work is to improve the performances of HMM to deal with big data challenges. The proposed algorithm shows a great improvement in reducing time complexity and provides good results in terms of running time, speedup, and parallelization efficiency for a large amount of data, i.e., large states number and large sequences number.

This article proposes a new parallel performance model for different workloads of Spark Big Data applications running on Hadoop clusters. The proposed model can predict the runtime for generic workloads as a function of the number of executors, without necessarily knowing how the algorithms were implemented. For a certain problem size, it is shown that a model based on serial boundaries for a 2D arrangement of executors can fit the empirical data for various workloads. The empirical data was obtained from a real Hadoop cluster, using Spark and HiBench. The workloads used in this work were included WordCount, SVM, Kmeans, PageRank and Graph (Nweight). A particular runtime pattern emerged when adding more executors to run a job. For some workloads, the runtime was longer with more executors added. This phenomenon is predicted with the new model of parallelisation. The resulting equation from the model explains certain performance patterns that do not fit Amdahl’s law predictions, nor Gustafson’s equation. The results show that the proposed model achieved the best fit with all workloads and most of the data sizes, using the R-squared metric for the accuracy of the fitting of empirical data. The proposed model has advantages over machine learning models due to its simplicity, requiring a smaller number of experiments to fit the data. This is very useful to practitioners in the area of Big Data because they can predict runtime of specific applications by analysing the logs. In this work, the model is limited to changes in the number of executors for a fixed problem size.

As a popular spatial operation, the k-Nearest Neighbors (kNN) query is widely used in various spatial application systems. How to efficiently process a kNN query on spatial big data has always been an important research topic in the field of spatial data management. The centralized solutions are not suitable for spatial big data due to their poor scalability, while the existing distributed solutions are not efficient enough to meet the high real-time requirements of some spatial applications. Therefore, we introduce the Proportional Integral Derivative (PID) control technology into kNN query processing and propose a PID-based kNN query processing algorithm (PIDKNN) for spatial big data based on Spark. In this algorithm, the whole data space is divided into grid cells of the same size using the grid partition method, and the grid-based index is constructed. On this basis, the grid-based density peak clustering algorithm is used to cluster spatial data, and the corresponding PID parameters are set for each cluster. When performing kNN queries, the PID algorithm is used to estimate the radius growth step size of kNN queries, thereby realizing kNN query processing with a variable query radius growth step based on a feedback mechanism, which greatly improves the efficiency of kNN query processing. A series of experimental results show that the PIDKNN algorithm has good performance and scalability and is superior to the existing parallel kNN query processing methods.

Spatial transcriptomic studies are becoming increasingly common and large, posing important statistical and computational challenges for many analytic tasks. Here, we present SPARK-X, a non-parametric method for rapid and effective detection of spatially expressed genes in large spatial transcriptomic studies. SPARK-X not only produces effective type I error control and high power but also brings orders of magnitude computational savings. We apply SPARK-X to analyze three large datasets, one of which is only analyzable by SPARK-X. In these data, SPARK-X identifies many spatially expressed genes including those that are spatially expressed within the same cell type, revealing new biological insights. Supplementary Information The online version contains supplementary material available at 10.1186/s13059-021-02404-0.

In recent years, with rapid economic development, air pollution has become extremely serious, causing many negative effects on health, environment and medical costs. PM2.5 is one of the main components of air pollution. Therefore, it is necessary to know the PM2.5 air quality in advance for health. Many studies on air quality are based on the government’s official air quality monitoring stations, which cannot be widely deployed due to high cost constraints. Furthermore, the update frequency of government monitoring stations is once an hour, and it is hard to capture short-term PM2.5 concentration peaks with little warning. Nevertheless, dealing with short-term data with many stations, the volume of data is huge and is calculated, analyzed and predicted in a complex way. This alleviates the high computational requirements of the original predictor, thus making Spark suitable for the considered problem. This study proposes a PM2.5 instant prediction architecture based on the Spark big data framework to handle the huge data from the LASS community. The Spark big data framework proposed in this study is divided into three modules. It collects real time PM2.5 data and performs ensemble learning through three machine learning algorithms (Linear Regression, Random Forest, Gradient Boosting Decision Tree) to predict the PM2.5 concentration value in the next 30 to 180 min with accompanying visualization graph. The experimental results show that our proposed Spark big data ensemble prediction model in next 30-min prediction has the best performance (R2 up to 0.96), and the ensemble model has better performance than any single machine learning model. Taiwan has been suffering from a situation of relatively poor air pollution quality for a long time. Air pollutant monitoring data from LASS community can provide a wide broader monitoring, however the data is large and difficult to integrate or analyze. The proposed Spark big data framework system can provide short-term PM2.5 forecasts and help the decision-maker to take proper action immediately.

Nowadays, video cameras are increasingly used for surveillance, monitoring, and activity recording. These cameras generate high resolution image and video data at large scale. Processing such large scale video streams to extract useful information with time constraints is challenging. Traditional methods do not offer scalability to process large scale data. In this paper, we propose and evaluate cloud services for high resolution video streams in order to perform line detection using Canny edge detection followed by Hough transform. These algorithms are often used as preprocessing steps for various high level tasks including object, anomaly, and activity recognition. We implement and evaluate both Canny edge detector and Hough transform algorithms in Hadoop and Spark. Our experimental evaluation using Spark shows an excellent scalability and performance compared to Hadoop and standalone implementations for both Canny edge detection and Hough transform. We obtained a speedup of 10.8x and 9.3x for Canny edge detection and Hough transform respectively using Spark. These results demonstrate the effectiveness of parallel implementation of computer vision algorithms to achieve good scalability for real-world applications. 2019, Springer Science+Business Media, LLC, part of Springer Nature. This work was made possible by NPRP Grant # 7-481-1-088 from the Qatar National Research Fund (a member of Qatar Foundation). The statements made herein are solely the responsibility of the authors. Scopus

Data has evolved into a large-scale data as big data in the recent era. The analysis of big data involves determined attempts on previous data. As new era of data has spatiotemporal facts that involve the time and space factors, which make them distinct from traditional data. The big data with spatiotemporal aspects helps achieve more efficient results and, therefore, many different types of frameworks have been introduced in cooperate world. In the present research, a qualitative approach is used to present the framework classification in two categories: architecture and features. Frameworks have been compared on the basis of architectural characteristics and feature attributes as well. These two categories project a significant effect on the execution of spatiotemporal data in big data. Frameworks are able to solve the real-time problems in less time of cycle. This study presents spatiotemporal aspects in big data with reference to several dissimilar environments and frameworks.

The proliferation of information technology produces a huge amount of data called big data that cannot be processed by traditional database systems. These Various types of data come from different sources. However, stragglers are a major bottleneck in big data processing, and hence the early detection and accurate identification of stragglers can have important impacts on the performance of big data processing. This work aims to assess five stragglers identification methods: Hadoop native scheduler, LATE Scheduler, Mantri, MonTool, and Dolly. The performance of these techniques was evaluated based on three benchmarked methods: Sort, Grep and WordCount. The results show that the LATE Scheduler performs the best and it would be efficient to obtain better results for stragglers identification.

В данной статье анализируются причины возникновения больших данных в мобильных приложениях, а также их особенности и характеристики. Также для анализа больших данных мобильных приложений, рассматривается распространённый фреймворк для работы с Big Data под названием Spark, входящий в проектную экосистему Hadoop и предлагающий крайне эффективный и экономичный способ хранения слабоструктурированных и неструктурированных данных в значительных объёмах., This article analyzes the reasons for the emergence of big data in mobile applications, as well as their features and characteristics. Also, for the analysis of big data for mobile applications, we consider a common framework for working with Big Data called Spark, which is part of the Hadoop project ecosystem and offers an extremely efficient and cost-effective way to store weakly structured and unstructured data in significant volumes.

Se han utilizado diferentes algoritmos de Machine Learning aplicados a la predicción de fallos en discos duros utilizando los atributos SMART. El problema de clasificación consiste en detectar anomalías en una serie temporal de datos ruidosos. A lo largo de todo el trabajo se hizo uso de Spark para la paralelización de procesos con el fin de reducir los tiempos de ejecución. Máster en Ciencia de Datos

One of the main goals of Big Data research, is to find new data mining methods that are able to process large amounts of data in acceptable times. In Big Data classification, as in traditional classification, class imbalance is a common problem that must be addressed, in the case of Big Data also looking for a solution that can be applied in an acceptable execution time. In this paper we present Approx-SMOTE, a parallel implementation of the SMOTE algorithm for the Apache Spark framework. The key difference with the original SMOTE, besides parallelism, is that it uses an approximated version of k-Nearest Neighbor which makes it highly scalable. Although an implementation of SMOTE for Big Data already exists (SMOTE-BD), it uses an exact Nearest Neighbor search, which does not make it entirely scalable. Approx-SMOTE on the other hand is able to achieve up to 30 times faster run times without sacrificing the improved classification performance offered by the original SMOTE., “La Caixa” Foundation, under agreement LCF/PR/PR18/51130007. This work was supported by the Junta de Castilla y León under project BU055P20 and by the Ministry of Science and Innovation of Spain under project PID2020-119894 GB-I00, co-financed through European Union FEDER funds. It also was supported through Consejería de Educación of the Junta de Castilla y León and the European Social Fund through a pre-doctoral grant (EDU/1100/2017). This material is based upon work supported by Google Cloud.

Spark NLP is a Natural Language Processing (NLP) library built on top of Apache Spark ML. It provides simple, performant and accurate NLP annotations for machine learning pipelines that can scale easily in a distributed environment. Spark NLP comes with 1100 pre trained pipelines and models in more than 192 languages. It supports nearly all the NLP tasks and modules that can be used seamlessly in a cluster. Downloaded more than 2.7 million times and experiencing nine times growth since January 2020, Spark NLP is used by 54% of healthcare organizations as the worlds most widely used NLP library in the enterprise., Comment: =Accepted as a publication in Elsevier, Software Impacts Journal. arXiv admin note: substantial text overlap with arXiv:2012.04005

The Rotation Forest classifier is a successful ensemble method for a wide variety of data mining applications. However, the way in which Rotation Forest transforms the feature space through PCA, although powerful, penalizes training and prediction times, making it unfeasible for Big Data. In this paper, a MapReduce Rotation Forest and its implementation under the Spark framework are presented. The proposed MapReduce Rotation Forest behaves in the same way as the standard Rotation Forest, training the base classifiers on a rotated space, but using a functional implementation of the rotation that enables its execution in Big Data frameworks. Experimental results are obtained using different cloud-based cluster configurations. Bayesian tests are used to validate the method against two ensembles for Big Data: Random Forest and PCARDE classifiers. Our proposal incorporates the parallelization of both the PCA calculation and the tree training, providing a scalable solution that retains the performance of the original Rotation Forest and achieves a competitive execution time (in average, at training, more than 3 times faster than other PCA-based alternatives). In addition, extensive experimentation shows that by setting some parameters of the classifier (i.e., bootstrap sample size, number of trees, and number of rotations), the execution time is reduced with no significant loss of performance using a small ensemble., This work was supported through project TIN2015-67534-P (MINECO/FEDER, UE) of the Ministerio de Economía y Competitividad of the Spanish Government, projects BU085P17 and BU055P20 (JCyL/FEDER, UE) of the Junta de Castilla y León, Spain (both projects co-financed through European Union FEDER funds), and by the Consejería de Educación of the Junta de Castilla y León, Spain and the European Social Fund through a pre-doctoral grant (EDU/1100/2017). The project leading to these results has received also funding from “la Caixa” Foundation, Spain , under agreement LCF/PR/PR18/51130007. This material is based upon work supported by Google Cloud.

В данной статье рассматривается такое явление как статическое электричество, его пожарная опасность, условия возникновения, а также влияние данного явления непосредственно на организм человека. Отражены условия безопасности, правила безопасного поведения и способы защиты от статического электричества как человека, так и самого производственного объекта. В статье приведены примеры пожара на производстве, отраженные в статистике научного сообщества, рассмотрены причины данного пожара и его последствия, This article considers such a phenomenon as static electricity, its fire hazard, the conditions of occurrence, as well as the effect of this phenomenon directly on the human body. Safety conditions, safe behavior rules and ways to protect against static electricity of both the person and the production facility are reflected. The article provides examples of a fire at the factory, reflected in the statistics of the scientific community, considered the causes of this fire and its consequences.

Spatial neighboring analysis is an indispensable part of geo-raster spatial analysis. In the big data era, high-resolution raster data offer us abundant and valuable information, and also bring enormous computational challenges to the existing focal statistics algorithms. Simply employing the in-memory computing framework Spark to serve such applications might incur performance issues due to its lack of native support for spatial data. In this article, we present a Spark-based parallel computing approach for the focal algorithms of neighboring analysis. This approach implements efficient manipulation of large amounts of terrain data through three steps: (1) partitioning a raster digital elevation model (DEM) file into multiple square tile files by adopting a tile-based multifile storing strategy suitable for the Hadoop Distributed File System (HDFS), (2) performing the quintessential slope algorithm on these tile files using a dynamic calculation window (DCW) computing strategy, and (3) writing back and merging the calculation results into a whole raster file. Experiments with the digital elevation data of Australia show that the proposed computing approach can effectively improve the parallel performance of focal statistics algorithms. The results also show that the approach has almost the same calculation accuracy as that of ArcGIS. The proposed approach also exhibits good scalability when the number of Spark executors in clusters is increased.

In recent times Big Data analytics has got tremendous attention and it involves storing, processing, and analysing large scale datasets. The advent of distributed computing frameworks such as Hadoop and Spark offers an efficient solution to analyse vast amounts of data. Due to the availability of an application program ming interface (API) and its performance, Spark become very popular, even more popular than the MapReduce framework. Both these frameworks have more than 150 parameters and the combination of these parameters have a huge impact on cluster performance. The system default parameters help the system administrator to deploy their system applications without much effort, and they can measure their specific cluster performance with factory-set parameters. However, an open question remains: can new parameter selection improve cluster performance? In this regard, our study investigates the most impacting parameters such as input splits and shuffling, in order to compare the performance between Hadoop and Spark, using a specific cluster implemented in our department. We used a trial-and-error approach for tuning these parameters based on a large number of experiments. In order to evaluate the frameworks comparison and analysis, we select two work- loads: WordCount and TeraSort. The performance metrics are carried out based on three criteria, namely execution time, throughput, and speedup. Our experimental results revealed that both system performances heavily depends on input data size and correct parameter selection. The analysis results found, unsurprisingly, that Spark has better performance as compared to Hadoop, achieving up to 2 times speedup in WordCount workload and up to 14 times in TeraSort workloads when default parameters are replaced. Finally, we conclude that the system performance depends on different parameters configuration alternatives, and they depend on the data size.

With the flourishing of Internet of Things (IoT) technology, ubiquitous power data can be linked to the Internet and be analyzed for real-time monitoring requirements. Numerous power data would be accumulated to even Tera-byte level as the time goes. To approach a real-time power monitoring platform on them, an efficient and novel implementation techniques has been developed and formed to be the kernel material of this thesis. Based on the integration of multiple software subsystems in a layered manner, the proposed power-monitoring platform has been established and is composed of Ubuntu (as operating system), Hadoop (as storage subsystem), Hive (as data warehouse), and the Spark MLlib (as data analytics) from bottom to top. The generic power-data source is provided by the so-called smart meters equipped inside factories located in an enterprise practically. The data collection and storage are handled by the Hadoop subsystem and the data ingestion to Hive data warehouse is conducted by the Spark unit. On the aspect of system verification, under single-record query, these software modules: HiveQL and Impala SQL had been tested in terms of query-response efficiency. And for the performance exploration on the full-table query function. The relevant experiments have been conducted on the same software modules as well. The kernel contributions of this research work can be highlighted by two parts: the details of building an efficient real-time power-monitoring platform, and the relevant query-response efficiency for reference.

This manuscript explores the application of big data analytics in online structural health monitoring. As smart sensor technology is making progress and low cost online monitoring is increasingly possible, large quantities of highly heterogeneous data can be acquired during the monitoring, thus exceeding the capacity of traditional data analytics techniques. This paper investigates big data techniques to handle the highvolume data obtained in structural health monitoring. In particular, we investigate the analysis of infrared thermal images for structural damage diagnosis. We explore the MapReduce technique to parallelize the data analytics and efficiently handle the high volume, high velocity and high variety of information. In our study, MapReduce is implemented with the Spark platform, and image processing functions such as uniform filter and Sobel filter are wrapped in the mappers. The methodology is illustrated with concrete slabs, using actual experimental data with induced damage

International audience; This paper proposes a model for specifying data flow-based parallel data processing programs agnostic of target Big Data processing frameworks. The paper focuses on the formal abstract specification of non-iterative and iterative programs, generalizing the strategies adopted by data flow Big Data processing frameworks. The proposed model relies on Monoid Algebra and Petri Nets to abstract Big Data processing programs in two levels: a higher level representing the program data flow and a lower level representing data transformation operations (e.g., filtering, aggregation, join). We extend the model for data processing programs, for modeling iterative data processing programs. The general specification of these programs implemented by data flow-based parallel programming models is essential given the democratization of iterative and greedy Big Data analytics algorithms. Indeed, these algorithms call for revisiting parallel programming models to express iterations. The paper gives a comparative analysis of the iteration strategies proposed by Apache Spark, DryadLINQ, Apache Beam, and Apache Flink. It discusses how the model achieves to generalize these strategies.

Big data is the biggest challenges as we need huge processing power system and good algorithms to make an decision. We need Hadoop environment with pig hive, machine learning and hadoopecosystem components. The data comes from industries. Many devices around us and sensor, and from social media sites. According to McKinsey There will be a shortage of 15000000 big data professionals by the end of 2020. There are lots of technologies to solve the problem of big data Storage and processing. Such technologies are Apache Hadoop, Apache Spark, Apache Kafka, and many more. Here we analyse the processing speed for the 4GB data on cloudx lab with Hadoop mapreduce with varing mappers and reducers and with pig script and Hive querries and spark environment along with machine learning technology and from the results we can say that machine learning with Hadoop will enhance the processing performance along with with spark, and also we can say that spark is better than Hadoop mapreduce pig and hive, spark with hive and machine learning will be the best performance enhanced compared with pig and hive, Hadoop mapreduce jar.

Outpatients who fail to attend their appointments have a negative impact on the healthcare outcome. Thus, healthcare organizations facing new opportunities, one of them is to improve the quality of healthcare. The main challenges is predictive analysis using techniques capable of handle the huge data generated. We propose a big data framework for identifying subject outpatients’ no-show via feature engineering and machine learning (MLlib) in the Spark platform. This study evaluates the performance of five machine learning techniques, using the (2,011,813, 19) outpatients’ visits data. Conducting several experiments and using different validation methods, the Gradient Boosting (GB) performed best, resulting in an increase of accuracy and ROC to 79% and 81%, respectively. In addition, we showed that exploring and evaluating the performance of the machine learning models using various evaluation methods is critical as the accuracy of prediction can significantly differ. The aim of this paper is exploring factors that affect no-show rate and can be used to formulate predictions using big data machine learning techniques

Desarrollo de una aplicación que hace uso de técnicas de inteligencia artificial con el objetivo de realizar predicciones acerca de los tiempos de aterrizaje de los vuelos. Los datos con los que se ha trabajado han sido recogidos tanto por los planes de vuelo, como por el sistema Automático Dependiente de Vigilancia - Difusión (ADS-B). Para agilizar el tratamiento de la información y el procesado de gran cantidad de datos, se utiliza Apache Spark y la biblioteca MLlib para extraer los modelos de aprendizaje., Grado en Ingeniería Informática de Servicios y Aplicaciones

15 pages, 9 figures, 5 tables, Optimization problems arise nowadays in all disciplines, not only in the scientific area but also in the field of engineering or economics, and in many others. Currently, challenging optimization problems require solution methods that consume a significant amount of computational resources. The application of High-Performance Computing techniques is a common approach to obtain efficient implementations in traditional parallel computing systems. However, more recent approaches are exploring distributed programming frameworks developed in recent years to achieve efficient computations on clusters and cloud systems. In this paper we present a parallel implementation of the enhanced Scatter Search metaheuristic using Spark. The parallel program was obtained as a particularization of a general software framework we developed to support different realisations of the Scatter Search metaheuristic. The aim of this paper is to provide helpful guidance to readers interested in applying, or developing their own, parallel metaheuristics to solve challenging problems in the Cloud. With the twofold objective of demonstrating the potential of the parallelization with Spark and also of studying the factors that influence the performance of the solution, the proposal has been thoroughly evaluated on two different platforms, a cluster and a cloud platform, using a representative set of parameter estimation problems in the field of Computational Systems Biology, This work was supported by the Ministry of Science and Innovation of Spain (DPI2017-82896-C2-2-R, TIN2016-75845-P and PID2019-104184RB-I00, AEI/FEDER/UE, 10.13039/501100011033); and by Xunta de Galicia and FEDER funds (Centro de Investigación de Galicia accreditation 2019-2022, ref. ED431G 2019/01, as well as Consolidation Program of Competitive Reference Groups, ref. ED431C 2017/04)