Search

Apache Storm is one of the most widely used platforms for processing of data streams due to its properties of being distributed, highly scalable, and fault-tolerant. It provides real-time processing, is fast and stateless, and uses master–slave architecture with ZooKeeper. In the Hadoop ecosystem, Apache Storm is the one that fills the present real-time functionality and provides strong coupling with many tools and technologies. In Storm framework, the Storm default scheduler is commonly used to schedule the task or data to be processed, whose basis for scheduling the task is the time quanta or time slots, which leads to increase in context switches and longer response time. In Storm default scheduler, the workload of a topology equally distributed among worker processes or Java virtual machine (JVM) all over the cluster using a simple round-robin algorithm without considering any priority or criteria. The proposed algorithm addresses the above-specified issues. An improvised the custom Storm scheduler was developed where the scheduling is based on the workload, which is calculated based on the total memory utilized per task and the total processing unit utilized per task, thereby resulting in lesser context switches and faster response time of distributed streaming applications.

There are many researchers working on handwritten text recognition (HTR) and also contributing to HTR domain. Even though many research methods are existing for HTR, there is a need for some more improvements in the accuracy of the HTR systems. This paper is a contribution of the application of the Deep Learning algorithm for the HTR system. In this paper first we will collect the data for training the handwritten texts, later features have been extracted from those text datasets and perform training of the model using Deep Learning approach. In this work we are going to use the strategy to recognize in terms of words rather those characters so that accuracy will be improved. The built model using LSTM deep model achieves a very good accuracy. Lastly, this developed approach of the HTR system is integrated into the OCR system and comparison of results are reported in this paper. Two approaches have been compared in this paper on IAM handwritten data set, and found that 2DLSTM based approach outperforms the other approach.

Edge computing is an emerging paradigm to assist intelligent decisions for cloud centric analytics. The major limitation of edge computing is the non-availability of a open-source platforms-as-a- service for various applications across cloud and edge. ECHO(an adaptive orchestration platform for streaming hybrid data flows across cloud and edge) attempts to fill this gap . It enables streaming data flows across distributed resources where user tasks are represented as vertices in a directed acyclic graph (DAG) and edges represent the routing channels between data and tasks. These DAGs are executed upon data arrival. ECHO’s current scheduler schedules jobs using round robin algorithm. This paper proposes improvement in ECHO’s scheduler. We consider current device health before scheduling dataflow in ECHO. The proposed scheduling algorithm makes use of CPU utilization and memory utilization as parameters. The experimental results show that the proposed algorithm improves the scheduler performance.

Data intensive systems aim to efficiently process “big” data. Several data processing engines have evolved over past decade. These data processing engines are modeled around the MapReduce paradigm. This article explores Hadoop's MapReduce engine and propose techniques to obtain a higher level of optimization by borrowing concepts from the world of High Performance Computing. Consequently, power consumed and heat generated is lowered. This article designs a system with a pipelined dataflow in contrast to the existing unregulated “bursty” flow of network traffic, the ability to carry out both Map and Reduce tasks in parallel, and a system which incorporates modern high-performance computing concepts using Remote Direct Memory Access (RDMA). To establish the claim of an increased performance measure of the proposed system, the authors provide an algorithm for RoCE enabled MapReduce and a mathematical derivation contrasting the runtime of vanilla Hadoop. This article proves mathematically, that the proposed system functions 1.67 times faster than the vanilla version of Hadoop.

Apache spark is one of the most demanded frameworks for High performance computing of Big Data. Data is growing day by day to such a large extent that the power of existing analytical tool is not sufficient. The degree of parallelism achieved directly impacts the performance of the framework. Parallelism can be achieved only when there is a proper division among the dataset. Partitioning helps in this division of work. Appropriate number of partitions and related data in partition will ensure the proper storage and faster access of data. Spark provides the inbuilt libraries and also flexibility to mould the available methods of partitioning according to convenience. In this paper we discus about the architecture of storage structure in spark, implementation of various partitioning techniques and performance comparison of those techniques.

Inspired by the victory of Apache's Hadoop this paper suggests a new reduce task scheduler. Hadoop is an open source implementation of Google's MapReduce framework. Programs which are written in this functional style are automatically executed and parallelized on a large cluster of commodity machines. The details how to partition the input data, setting up the program's for execution across a set of machines, handling failures of machine and managing the necessary inter-device communication is taken care by runtime system. In the current versions of Hadoop, the map tasks are scheduled with respect to the locality of their inputs in order to shrink network traffic and improve performance. On the other hand, the reduce tasks are scheduled without taking into consideration data locality leading to ruin the performance at requesting nodes. In this paper, we use data locality that is natural with reduce tasks. To accomplish the same, we schedule them on nodes that will result in least amount data- local traffic. Experimental results signify an 11-80 percent decrease in the number of bytes shuffled in a Hadoop cluster.

It is quite evident that everyone has certain objectives to be completed and quite a few expectations to be met. Accomplishments of these targets require proper planning in addition to an individual's abilities. Smartphones have seen substantial growth over the past decade and there is perceptible evidence of people relying on them. Although there are solutions in the form of to-do list applications to mitigate the scheduling problem, they provide little insight regarding an individual's time organization. Most of the existing applications just provide the feature of recording the tasks and users themselves need to set the time at which these tasks have to be executed. In our application focus is laid on learning the leisure time of the user by indirectly monitoring day to day activities through the app. Free time as predicted by the model and the list of tasks to be performed together form the input to the scheduling algorithm. The scheduling algorithm then allocates time for each task ensuring load balancing by evenly distributing the tasks across the week and the user is notified of their personalized final schedule.

A popular program ming model for running data intensive applications on the cloud is map reduce. In the Hadoop usually, jobs are scheduled in FIFO order by default. There are many map reduce applications which require strict deadline. In Hadoop framework, scheduler w i t h deadline c o n s t r a i n t s has not been implemented. Existing schedulers d o not guarantee that the job will be completed by a specific deadline. Some schedulers address the issue of deadlines but focus more on improving s y s t e m utilization. We have proposed an algorithm which facilitates the user to specify a jobs deadline and evaluates whether the job can be finished before the deadline. Scheduler with deadlines for Hadoop, which ensures that only jobs, whose deadlines c an be met are scheduled f o r execution. If the job submitted does not satisfy the specified deadline, physical or virtual nodes can be added dynamically to complete the job within deadline(8).

The improvement in the technology and the desperate need to store huge data has been increasing steadily. Hadoop uses MapReduce framework for processing big datasets, which consists of namenode and datanode. The hadoop framework technology has been extensively used to store, process, and use the enormous data that are being placed into server. Handling of huge small files is becoming laborious since the namenode has to manage the filenames and its corresponding metadata. For the purpose of fault tolerance, the data needs to be replicated on the data nodes. In order to handle the mammoth number of small files and to reduce the out of memory burden on the namenode, several techniques are being approached which involves HDFS, EHDFS, SFS, HAR, NHAR. In this paper we discuss about the different techniques that are developed to handle the storage of small files and propose a new method to solve the storage issue in hadoop. The new method takes the priority of the file as a distinguishing factor which further helps in reducing the memory usage of namenode.

Wireless Sensor Network is a group of wireless nodes exclusively designed for the continuous sensing of information at human inaccessible locates. Reprogramming is a definite need at such situations when the monitoring conditions vary according to the environmental changes or other user requirements. Insecure transmission of reprogramming code to such areas can ruin the entire operation of the network. To avoid this, Secure and Distributive Reprogramming Protocol (SDRP) was proposed for user privilege maintenance. In this paper, Diffie-Hellman key(DH) exchange mechanism is implemented as an enhancement to the existing method to further improve security between the forwarding nodes. A network simulator simulation analysis and discussion is provided for the proposed SDRP-DH.

Text Summarization is a method of reducing the original text document into a short description. This short version retains the meaning and information content of the original text document. It is a difficult task for human beings to generate the summary for very large documents manually. The linguistic and statistical features of sentence can be used to find the importance of sentences. The Latent Semantic Analysis (LSA) captures automatically the semantic relationships between the sentences as a human being thinks. In this paper Singular Value Decomposition (SVD) is used to generate the summary. SVD finds the dimensions of the sentence vectors which are principal and mutually orthogonal. These properties guaranty the relevance to original text document and non-redundancy respectively in machine generated summary.

In this paper we propose a game theoretic approach called The New Protocol and we integrate this into the reactive Ad hoc On-demand Distance Vector (AODV) routing protocol to provide defense against blackhole attacks. This idea is based on the concept of non-cooperative game theory. The AODV-NEW outperforms AODV in terms of the number of dropped packets when blackhole nodes exist within a MANET (Mobile AdHoc Network).