Multi-objective optimisation of compute and data intensive e-science workflows

Abstract Raw e-Science data, which may be, for example, MRI brain scans, data from a high energy physics detector or metric data from the earth observation projects needs to undergo a series of computations before meaningful knowledge can be derived. One way to. de- scribe these series of computations on raw e-Science data are workflows. Workflows have emerged as the principle mechanism for describing and enacting complex e-Science analy- sis on distributed infrastructures such as Grids. Workflows provide domain scientists with a systematic, repeatable and reproducible means of conducting scientific analyses. Due to the demands of state-of-the-art e-Science applications, scientific workflows are increas- ing in complexity. This complexity is multi-dimensional; scientific workflows are scaling in terms of the number of computations and tasks they carry out. They are also scaling in terms of the data they manage and generate. Scientific workflows are also scaling in terms of the resources they consume. Due to all of these factors the optimisation of these workflows is a prime concern. State-of-the-art approaches to workflow optimisation pri- marily focus on compute optimisation. However, as e-Science is becoming increasingly data-centric, data optimisation is gaining increasing importance. This thesis explores the development of a multi-objective approach to the optimi- sation of scientific workflows. Differing and conflicting considerations are required to optimise a workflow for compute or data efficiency. The approach proposed formulates the optimisation of a scientific workflow as a multi-objective optimisation problem and demonstrates the optimisation of the same through the use of a multi-objective evolution- ary meta-heuristic. The results demonstrate that significant optimisation can be achieved through this approach.