Adaptive Scheduling for Efficient Execution of Dynamic Stream Workflows

Stream workflow application such as online anomaly detection or online traffic monitoring, integrates multiple streaming big data applications into data analysis pipeline. This application can be highly dynamic in nature, where the data velocity may change at runtime and therefore the resources should be managed overtime. To manage these changes, the orchestration of this application requires a dynamic execution environment and dynamic scheduling technique. For the former requirement, Multicloud environment is a visible solution to cope with the dynamic aspects of this workflow application. While for the latter requirement, dynamic scheduling technique not only need to adhere to end user's requirements in terms of data processing and deadline for decision making, and data stream sources location constraints, but also adjust provisioning and scheduling plan at runtime to cope with dynamic variations of stream data rates. Therefore, we propose a two-phase adaptive scheduling technique to efficiently schedule dynamic workflow application in Multicloud environment that can respond to changes in the velocity of data at runtime. The experimental results showed that the proposed technique is close to the lower bound and effective for different experiment scenarios.
Comment: 17 pages, 11 figures