Search

In the new digital age, the pace and volume of growing transportation related data is exceeding our ability to manage and analyze it. In this position paper, we present a data engine, Godzilla, to ingest real-time traffic data and support analytic and data mining over traffic data. Godzilla is a multi-cluster approach to handle large volumes of growing data, changing workloads and varying number of users. The data originates at multiple sources, and consists of multiple types. Meanwhile, the workloads belong to three camps, namely batch processing, interactive queries and graph analysis. Godzilla support multiple language abstractions from scripting to SQL-like language.

Both short and long term information of the transportation network is needed by commuters and planners. In order to obtain this information, there is a pressing need to consolidate, mine and analyze data collected from multiple sources. To enable these activities under a single umbrella, we propose a data platform in this position paper that transforms data into information. Finally, we discuss the research challenges facing our platform.

The rate of data growth in many domains is straining our ability to manage and analyze it. Cloud computing appears as a promising platform for data-intensive computing because it offers “infinite” resources on demand, and on a pay-as-you-go basis. Surprisingly, we observe that public clouds are not being used for “serious” data-processing on a continuous basis except by the cloud vendors. We primarily attribute this observation to the long data transfer times over wide-area networks between the clients and the centralized public clouds. We introduce the idea of a near-cloud that brings a public cloud in the close proximity of a user to overcome the data transfer bottlenecks. The near-clouds also provide a unique opportunity to localize the privacy and security policies to boost confidence of a user in using shared resources for data processing. To the cloud providers, near-cloud ensures a minimum constant stream of revenue from dedicated clients.

The promise of "infinite" resources given by the cloud computing paradigm has led to recent interest in exploiting clouds for large-scale data-intensive computing. Given this supposedly infinite resource set, we need a management function that regulates application workload on these resources. This doctoral research focuses on two aspects of workload management, namely scheduling and provisioning. We propose a novel framework for workload execution and resource provisioning, and associated models, algorithms, and protocols.