Search

The Tapis Streams API is a production grade quality service that provides REST APIs for storing, processing and analyzing real-time streaming data. This paper focuses on improvements made to Tapis 1.0 Streams API for making it up-to-date and easily accessible. The newer version, Tapis 1.2 Streams API adopts the latest version of InfluxDB, InfluxDB 2.X, which has built-in security features and supports next generation data analytics and processing with a data processing language Flux. This paper also discusses the measures implemented in the Tapis 1.2 Streams API to mitigate potential security risks involved in unauthorized data stream access by users who do not own it. Additionally, new data Channel Actions supporting 3rd Party notification and web-hooks has been released. Lastly a tool, Tapis UI, which is a self contained server less application to access Tapis Services via rest calls is discussed in the paper. Tapis UI is a lightweight browser only client application which allows interactive access to Streams resources and real-time streaming data.

The Tapis framework, an NSF-funded project, is an open-source, scalable API platform that enables researchers to perform distributed computational experiments securely and achieve faster scientific results with increased reproducibility. Tapis Streams API focuses on supporting scientific use cases that require working with real-time sensor data. The Streams Service, built on the top of the CHORDS time-series data service, allows storing, processing, annotating, querying, and archiving time-series data. This paper focuses on the new Tapis Streams API functionality that enables researchers to design and execute real-time data-driven event workflow for their research. We describe the architecture and design choices towards achieving this new capability with Streams API. Specifically, we demonstrate the integration of Streams API with Kapacitor, a native data processing engine for time-series database InfluxDB, and Abaco, an NSF Funded project, web service, and distributed computing platform providing function-as-a-Service (FaaS). The Streams API, which includes a wrapper interface for the Kapacitor alerting system, can define and enable alerts. Finally, simulation results from the water-quality use case depict that Streams API’s new capabilities can support real-time streaming data event-driven workflows.

An algorithm is proposed to detect and classify a change in the distribution of a stochastic process that has periodic statistical behavior. The problem is posed in the framework of independent and periodically identically distributed (i.p.i.d.) processes, a recently introduced class of processes to model statistically periodic data. It is shown that the proposed algorithm is asymptotically optimal as the rate of false alarms and the probability of misclassification goes to zero. This problem has applications in anomaly detection in traffic data, social network data, ECG data, and neural data, where periodic statistical behavior has been observed. The effectiveness of the algorithm is demonstrated by application to real and simulated data.

Safety-critical networks often have stringent real-time requirements; they must also be resilient to failures. In this paper, we propose the RealFlow framework that uses commodity software-defined networks (SDNs) to realize networks with end-to-end timing guarantees, while also: (a) increasing resiliency against link/switch failures and (b) increasing network utilization. The use of SDNs in this space also improves the management capabilities of the system due to the global visibility into the network. RealFlow is implemented as a northbound SDN controller application compatible with standard OpenFlow protocols with little to no runtime overheads. We demonstrate feasibility on a real hardware testbed (Pica8 SDN switches+Raspberry Pi endhosts) and a practical avionics case study. Our evaluations show that RealFlow can accommodate 63% more network flows with safety-critical guarantees when compared to current designs and up to 18% when link resiliency (via backup paths) is also considered.

Modern computational research increasingly spans multiple, geographically distributed data centers and leverages instruments, experimental facilities and a network of national and regional cyberinfrastructure (CI). Tapis is an open-source API platform developed at the Texas Advanced Computing Center at the University of Texas at Austin to increase reproducibility and minimize time-to-solution for distributed computational experiments. Core features of Tapis include data management and code execution, a fine-grained permissions system enabling objects to be saved privately, shared with individuals or “published” to a community, and provenance endpoints exposing the detailed history Tapis collects on analyses, enabling workflows to be repeated and results reproduced. In this paper, we describe the evolution of the Tapis platform, from its origins in 2008, and discuss the growth and success of the project as well as challenges and limitations that have led to a new design effort, funded by the National Science Foundation in September of 2019. We present a detailed overview of the new system, including reference architecture and new features such as support for streaming/sensor data, and we discuss some of the early science use cases driving its design. We conclude with the roadmap for future work.

International audience

There has been a recent major upsurge in the concerns about reproducibility in many areas of science. Within the neuroimaging domain, one approach is to promote reproducibility is to target the re-executability of the publication. The information supporting such re-executability can enable the detailed examination of how an initial finding generalizes across changes in the processing approach, and sampled population, in a controlled scientific fashion. ReproNim: A Center for Reproducible Neuroimaging Computation is a recently funded initiative that seeks to facilitate the "last mile" implementations of core re-executability tools in order to reduce the accessibility barrier and increase adoption of standards and best practices at the neuroimaging research laboratory level. In this report, we summarize the overall approach and tools we have developed in this domain.

There has been a recent major upsurge in the concerns about reproducibility in many areas of science. Within the neuroimaging domain, one approach is to promote reproducibility is to target the re-executability of the publication. The information supporting such re-executability can enable the detailed examination of how an initial finding generalizes across changes in the processing approach, and sampled population, in a controlled scientific fashion. ReproNim: A Center for Reproducible Neuroimaging Computation is a recently funded initiative that seeks to facilitate the ‘last mile’ implementations of core re-executability tools in order to reduce the accessibility barrier and increase adoption of standards and best practices at the neuroimaging research laboratory level. In this report, we summarize the overall approach and tools we have developed in this domain.

Real-time systems (RTS) require end-to-end delay guarantees for the delivery of network packets. In this paper, we propose a framework to reduce the management and integration overheads for such real-time (RT) network flows by leveraging the capabilities of software-defined networking (SDN) – capabilities that include global visibility and management of the network. Given the specifications of flows that must meet hard real-time requirements, our framework synthesizes paths through the network. To guarantee that these flows meet both, their bandwidth and end-to-end timing requirements, our framework solves a multi-constraint optimization problem using a heuristic algorithm. We use exhaustive emulations and experiments on hardware switches to demonstrate our techniques and feasibility of our approach. As a result of this work, SDNs become “delay-aware” and thus can be adapted for use in safety-critical and other delay-sensitive applications.

With growing diverse volumes of digital data becoming part of modern scientific workflows, many research projects today begin with a process of data wrangling, i.e. finding, manipulating, indexing, cleaning, and bringing together needed datasets. Brown Dog, a Science Driven Data Transformation service, aims to alleviate much of the overhead and heterogeneity involved in this step, which in turn hinders scientific reproducibility, by providing data transformations such as format conversions and content based extractions as a service. Through a REST API Brown Dog supports diverse usage by various clients such as gateways, programming languages, and tools. As a gateway it provides a venue to access and preserve data transformation tools, track provenance, track information loss, manage data movement, and process jobs in a scalable manner across a diverse set of computational resources. Overall, Brown Dog provides a low level data infrastructure to interface with digital data contents and through its capabilities enable a new era of science and applications at large over otherwise difficult to access datasets. Further, Brown Dog aims to serve not just the scientific community but the general public as a “DNS” for data, moving civilization towards an era where applications can be largely agnostic to the format/structure of the data and can instead focus on novel processes/applications on the contents.

Brown Dog is an extensible data cyberinfrastructure, that provides a set of extensible and distributed data conversion and metadata extraction services to enable access and search within unstructured, un-curated and inaccessible research data across different domains of sciences and social science, which ultimately aids in supporting reproducibility of results. We envision that Brown Dog, as a data cyberinfrastructure, is an essential service in a comprehensive cyberinfrastructure which includes data services, high performance computing services and more that would enable scholarly research in a variety of disciplines that today is not yet possible. Brown Dog focuses on four initial use cases, specifically, addressing the conversion and extraction needs in the research areas of ecology, civil and environmental engineering, library and information science, and use by the general public. In this paper, we describe an architecture that supports contribution of data transformation tools from users, and automatic deployment of the tools as Brown Dog services in diverse infrastructures such as cloud or high performance computing (HPC) based on user demands and load on the system. We also present results validating the performance of the initial implementation of Brown Dog.

We present Brown Dog, two highly extensible services that aim to leverage any existing pieces of code, libraries, services, or standalone software (past or present) towards providing users with a simple to use and programmable means of automated aid in the curation and indexing of distributed collections of uncurated and/or unstructured data. Data collections such as these encompassing large varieties of data, in addition to large amounts of data, pose a significant challenge within modern day "Big Data" efforts. The two services, the Data Access Proxy (DAP) and the Data Tilling Service (DTS), focusing on format conversions and content based analysis/extraction respectively, wrap relevant conversion and extraction operations within arbitrary software, manages their deployment in an elastic manner, and manages job execution from behind a deliberately compact REST API. We describe both the motivation and need/scientific drivers for such services, the constituent components that allow for arbitrary software/code to be used and managed, and lastly an evaluation of the systems capabilities and scalability.

DIBBs Brown Dog is a recent cyberinfrastructure effort which aims to create two new services to aid users in the searching, accessing, and usage of digital data and provide these services in a manner that is as broadly and easily accessible as possible. At its lowest level, the Data Access Proxy (DAP) providing file format conversion capabilities and the Data Tilling Service (DTS) providing content based extractions, will be accessible via a deliberately compact REST interface. On top of this a number of client libraries and applications can and are being constructed to even further reduce the overhead of accessing the provided functionality (e.g. libraries in Javascript, Python, R, Matlab and interfaces such as bookmarklets, browser extensions, and other standalone applications). At the heart of the two services, however, is their extensibility, allowing one to potentially incorporate any library or tool as a conversion or extraction component within the services towards both the leveraging of the functionality they provide as well as the preservation of the tools themselves.

Current monitoring systems for cloud infrastructures are based on local, centralized or hierarchical model approaches such as HP Openview and ArcSight. Additionally, they do not look deep into resilience and delivering trustworthy data of its own services under crash or Byzantine failures caused by attackers or any other kind of sources. This work proposes a fault and intrusion tolerant monitoring system for cloud computing infrastructures. We assume a Byzantine failure model and use state machine replication for providing the trustworthy and resilient monitoring service.

Compute platforms for wireless sensor networks and Vehicle-to-Vehicle (V2V) communications employ random channel access for message transmission and typically suffer from limited processing capability and on-board memory on a per-application basis because of the multiple processes going on in parallel. Appending digital signatures to transmitted messages in such systems increases information reliability, but requires an intemperate use of scarce resources, more so with an increased security requirement.It thus appears imperative to tradeoff security for network performance to conserve scarce resources for a given resource constrained platform. This requires a good understanding of the communication performance of these systems. We observe that resource constraints in these systems induce complex interaction between the security and MAC layers at a node, obviating the possibility of layer specific optimizations to improve system performance for broadcast applications. For example, reducing the channel access probability at the MAC layer reduces collision probabilities, in turn increasing the verification load on the security layer. There is hence a need to take a holistic approach to dimension such systems to improve performance. In achieving this objective, we provide:• an analytical framework to model these systems with or without an impersonation attacker. This analysis also takes into consideration the possibility of multiple digital signatures being attached to a message, so that the receiver has a choice of verifying one of these.• a characterization of the stability region of the system.• an information-theoretic approach towards reliable communication of application data over these systems. We view the combination of security-MAC-security layers as an erasure channel and provide attainable rates, jointly optimizing on sampling rate and channel transmission rate.

We consider a sensor network where the nodes are capable of energy harvesting, i.e., can use energy from surrounding environment to recharge their battery. We consider the problem of controlling the activity of a sensor node. This problem is cast as controlling the service rate of a station in a closed three-queue system. We obtain the Norton's equivalent of the system and show the structure of the optimal rate control policy for various combinations of rate structures and utility function.

Many of recent studies have proved the tail equivalence result for Egalitarian Processor Sharing system: [EQUATION], where B (resp. V) is service requirement (resp. sojourn time) of a customer; for PS, g = 1 - ρ. In this paper, we consider time-shared systems in which the server capacity is shared by existing customers in proportion to (dynamic) weights assigned to customers. We consider two systems, 1) in which the weight of a customer depends on it Age (attained service), and 2) in which the weight depends on the residual processing time (RPT). We allow for a parameterized family of weight functions such that the weight associated with a customer that has received a service (or, has a RPT) of x units is ω(x) = xα for some -∞