Search

Big data applications play an important role in real time data processing. Apache Spark is a data processing framework with in-memory data engine that quickly processes large data sets. It can also distribute data processing tasks across multiple computers, either on its own or in tandem with other distributed computing tools. Spark’s in-memory processing cannot share data between the applications and hence, the RAM memory will be insufficient for storing petabytes of data. Alluxio is a virtual distributed storage system that leverages memory for data storage and provides faster access to data in different storage systems. Alluxio helps to speed up data intensive Spark applications, with various storage systems. In this work, the performance of applications on Spark as well as Spark running over Alluxio have been studied with respect to several storage formats such as Parquet, ORC, CSV, and JSON; and four types of queries from Star Schema Benchmark (SSB). A benchmark is evolved to suggest the suitability of Spark Alluxio combination for big data applications. It is found that Alluxio is suitable for applications that use databases of size more than 2.6 GB storing data in JSON and CSV formats. Spark is found suitable for applications that use storage formats such as parquet and ORC with database sizes less than 2.6GB.

A key challenge in B2B and B2C applications involving dynamic composition of workflow activities is to ensure their reliable execution, in view of frequent failures in the internet environment where a composed service is executed. Existing approaches for transactional coordination of composite web services, consider forward and backward recovery alone, in order to maintain consistency in the case of failures. However, while achieving reliable execution, service cancellability is important due to the predominance of long running processes in business applications and frequently changing user requirements and business policies. If the execution of a long running process is not cancelled in the case of a changed requirement, the completion of the process results in wastage of resources and the outcome may no longer be useful. Hence, a transactional coordination framework named as Tx-FAITH is proposed to handle the cancellation recovery under the external interruption and cancellation of transactions. Tx-FAITH considers hierarchically composed web services whose components are recursively nested to form a composition and adapts a coordination approach based on context information. The proposed coordination and recovery approach when tested with 200 simultaneous users and with a database size of 0.35 million records, incurs only 10 seconds to process a request.

In Service-Oriented Computing (SOC), a business transaction comprises of several web services provided by multiple enterprises. The transactional behaviour of individual web services must be considered for service selection so that the composition of web services results in a reliable execution. It is difficult for a business analyst to envisage the desired business policies of a process in terms of transactional properties of the corresponding service. Hence, an abstract mechanism that enables the business analyst to specify the transactional properties in a simple manner must be introduced. Towards this objective, it is proposed to express the transactional properties in terms of the recoverability of services. The transactional web services are grouped into different levels of recoverability based on their recovery cost. The estimated recovery costs are empirically verified and validated.

Web service composition, that recursively constructs a composite web service out of the existing services based on a business workflow has been acknowledged as a promising approach to meet the user demands, whenever a single service alone cannot fulfil the needs. In view of frequent failures in the internet environment where the composed service is executed, reliability of the composed service must be ensured. The reliability is determined by the behavioral or transactional properties of component services. The component services for each activity of the workflow must be selected based on their behavior so that their execution results in a consistent termination. Service selection must happen at run-time in order to consider the services available in a service registry at the time of execution. Towards this need, a dynamic transaction aware web service selection approach is proposed in this paper. Further, whenever user requirements change, a long running transaction must be interrupted and cancelled which is not addressed by any of the existing works. Hence, service cancellability property is proposed in this paper and incorporated in the dynamic selection approach. The overhead of the proposed run-time selection approach is assessed and the impact of increased services on its performance is also measured.