Search

This tutorial will focus on providing attendees exposure to state-of-the-art techniques for portable, reproducible research computing, enabling them to easily transport analyses from cloud to HPC resources, share computations with collaborators and disseminate final results to communities of interest. We will introduce various open source technologies, including Jupyter, Docker and Singularity, and show how to utilize these tools within the NSF-funded Tapis v3 platform, an Application Program Interface (API) for distributed computation. After a brief introduction to the open source technologies above, this tutorial will be focused on hands-on exercises in which the attendees will build a portable analysis that can be seamlessly moved to different execution environments, including a small virtual machine and a national-scale supercomputer. Using techniques covered in the tutorial, attendees will also be able to easily share their results with one or more additional users. The tutorial will make use of a specific machine learning image classifier analysis to illustrate the concepts, but the techniques introduced can be applied to a broad class of analyses in virtually any domain of science or engineering. Description and Format: TACC training accounts will be set up for all registered attendees, which will have access to allocations on XSEDE cloud systems and one or more HPC resources such as TACC’s Stampede2 or Frontera. The tutorials will be hands-on exercises, where the attendees will interact with the Tapis v3 services within a Jupyter notebook. Registered attendees will be notified with their account details closer to the tutorial date. All the course materials will be published on github pages so the attendees will have access to them during and after the tutorial. We will have enough proctors throughout the session, who will help attendees through slack or breakout sessions. Proposed tutorial schedule is as shown in Table 1. Learning Outcomes: In this tutorial, attendees will gain an understanding of the concepts of using container technology (Docker, Singularity) for portable analysis, programmatically executing analyses in both Cloud and HPC environments using an API, interacting with and visualizing the results in Jupyter notebooks and sharing results with collaborators. By the end of this workshop attendees will be able to: • Have a basic understanding of Docker and Singularity containers in relation to computational research. • Use Tapis to access HPC storage and compute resources in a programmatic and reproducible way. • Utilize Jupyter notebooks for interactive computing. • Use Tapis to share results with others. Content Level and Length: Beginner 70%, Intermediate 30% 3 hours. Audience Prerequisites: Basic familiarity with Jupyter notebooks and Python will be helpful. Attendees must use their own laptop for the hands-on part of the tutorial.

Over the last 15 years, a great investment has been made in providing web access to advanced computing resources for computational research. Historically, such web applications, or “science gateways”, have enabled users to run analyses asynchronously on remote systems. More recently, as a growing number of disciplines bring big data techniques to bear on fundamental problems, interactive computing modes such as Jupyter Notebooks have gained tremendous popularity for the ease within which one can perform a range of computational tasks in real time, including data cleansing, analysis, visualization, and post-processing. Nevertheless, to date there is not a national-scale offering that provides production-grade, scalable interactive computing that integrates deeply into the academic cyberinfrastructure (CI) ecosystem. The Texas Advanced Computing Center recently launched the Scientific and Interactive Computing (SCINCO) project to provide a hardened, production-grade Jupyter-notebooks-as-a-service platform capable of utilizing advanced storage and computing CI. Launched in 2020, SCINCO builds upon a custom JupyterHub offering developed at TACC since 2015. TACC’s custom JupyterHub supports more than 1600 users running across five different clusters at TACC. It has become a crucial component of independently funded gateway projects such as: DesignSafe Cyber-Infrastructure, Synergistic Data Discovery Environment, 3DEM and HETDEX. Researchers from Astronomy, Biology, Climate Science, Neuroscience, etc. are leveraging SCINCO to analyze big data, implement computational models, disseminate results and train researchers. SCINCO tackles scalability challenges faced by TACC’s original JupyterHub by combining state-of-the-art open source container technologies such as Kubernetes with a customizable JupyterHub to deliver a platform that is capable of serving dozens or even hundreds of projects (i.e., “tenants”) with a minimal developer overhead. SCINCO executes notebook containers across a shared Kubernetes cluster running on bare-metal that uses namespaces for isolating different projects from each other. SCINCO inherits all the advantages of Kubernetes including scalability, reproducibility and portability while significantly reducing the administrative overhead associated with managing servers. For example, TACC’s original JupyterHub utilized thirty 16GB virtual machines arranged into 5 different clusters; SCINCO can provide the same computational power with better load balancing using just 2 bare metal, physical servers with 256 GB of RAM each. Customizations (Figure 1) made within SCINCO support user-ID and group-ID lookup for an associated username, when launching a user’s notebook server. Additionally, the SCINCO design dynamically determines which persistent volumes the user has access to at runtime. SCINCO also allows for dynamic configuration changes at runtime, such as adding or removing volume mounts for a specific user group, notebook images, admin users, etc. The SCINCO project is currently developing an administrative portal, which will empower project administrators to manage their own JupyterHub clusters without direct support from the project staff. Project admins will get access to a dashboard where they can manage users, custom images, and volume mounts for individual users. They will be able to start, stop and start a shell session on the user's notebook servers. The SCINCO project made its initial production release in July of 2021 and the administrative portal will be released for early adopters in Fall 2021.

The explosion of IoT devices and sensors in recent years has led to a demand for efficiently storing, processing and analyzing time-series data. Geoscience researchers use time-series data stores such as Hydroserver, VOEIS and CHORDS. Many of these tools require a great deal of infrastructure to deploy and expertise to manage and scale. Tapis's (formerly known as Agave) platform as a service provides a way to support researchers in a way that they are not responsible for the infrastructure and can focus on the science. The University of Hawai‘i (UH) and Texas Advanced Computing Center (TACC) have collaborated to develop a new API integration that combines Tapis with the CHORDS time series data service to support projects at both institutions for storing, annotating and querying time-series data. This new Streams API leverages the strengths of both the Tapis platform and CHORDS service to enable capabilities for supporting time-series data streams not available in either tool alone. These new capabilities may be leveraged by Tapis powered science gateways with needs for handling spatially indexed time-series data-sets for their researchers as they have been at UH and TACC.