Search

The DesignSafe [1] portal is a web-based application created to facilitate researchers in the natural hazards community, providing data storage, computational resources, and publication pipelines to its users. With partners such as the RAPID response field reconnaissance teams who collect data immediately after a disaster event, the geospatial component of the data is essential information needed in subsequent analysis. Storing, searching, and visualizing this geospatial information has subsequently become a need for the DesignSafe and natural hazards community. A typical field reconnaissance mission can generate many gigabytes of geotagged images, videos and point cloud datasets. To integrate these data with the DesignSafe platform, we have developed a new set of RESTful APIs and a single page application to store, query and visualize these datasets. As one of the requirements for these tools is tight integration with the existing DesignSafe platform, which is built using Tapis [2] APIs, we have implemented the GeoAPI as a separate microservice in the Tapis v2 platform. This design allows us to easily utilize the existing data storage and user management tools built into DesignSafe. Users can access their existing data that is shared with their colleagues and already stored on the DesignSafe platform. Similarly, by utilizing the authentication services in Tapis, users can access these services with a single account. The backend has been developed using Python and the Flask framework. Application data is stored in a Postgres database with the PostGIS extension providing geospatial indexing. We have leveraged open-source libraries in the application such as GDAL, PDAL, Laspy, GeoPandas, and Shapely. Reconnaissance teams often use Lidar to collect point cloud data of structures damaged during a natural hazard event. These datasets alone can grow to many hundreds of GB and require additional processing. We use Celery to asynchronously process these data with Potree [3] to visualize them in the browser. Static assets such as images and videos are saved into a Ceph filesystem and are served directly by a Nginx proxy for performance. Maps can be automatically generated from data already in DesignSafe and asynchronously updated when new data is added to the storage system. The application is deployed into a Kubernetes cluster at TACC. A general architecture diagram is shown in figure 1. To visualize these datasets, we have created Hazmapper 2.0, a single page web application. This application was written in TypeScript using the Angular framework and Leaflet libraries. Additionally, we use an auto generated client library Ng-Tapis [4] to interact with the Tapis APIs. By doing so, we can utilize the API directly from the browser without the need for a separate backend application. External map tiles can be imported through a searchable interface using QuickMapServices (QMS) APIs [5]. The decoupled backend architecture allows us to have multiple user interfaces that are specialized for different end-user needs. One such example is the Taggit [6] application from Designsafe researchers. Separate from Hazmapper, this application allows for tagging and classification of georeferenced images.

The challenges in building science gateways are as varied as the problems they seek to solve, and there is no guarantee that the obstacles of tomorrow will look anything like the challenges of today. To meet the ever-changing needs of researchers, gateways must be rapidly deployable, easy to customize, and yet consistent enough that researchers can intuitively understand and use them. At Texas Advanced Computing Center (TACC), a variety of science gateways are used by researchers to access TACC’s computing resources, manage data and collaborate with other researchers. First generation portals were deployed directly to a virtual machine and suffered from a slew of issues over time such as package conflicts, difficulties in allowing for customization and brittle deployments caused by VM upgrades (e.g security updates). To address these challenges, the services of science gateways - such as a user guide, content management system (CMS), dashboard and other applications - were separated into Docker containers hosting distinct services. We then developed a Docker container-based deployment framework called Camino to facilitate the deployment [1]. The Camino framework employs configuration settings stored in a repository to facilitate easy, transparent and reproducible deployments. With this approach, developers can easily customize and deploy upgrades at-will to individual containers, meeting the rapidly evolving needs of PIs and stakeholders. Camino is currently used to deploy and manage a multitude of scientific gateways at TACC. As an example, these gateways are typically centered around our core science gateway, a Django/React application called Core-Portal [2] which provides a dashboard to HPC job access, data access, and collaboration tools, and runs in its own isolated container. Join us for this lightning talk where we will share the tools, strategies, and real-world solutions we are building now for the challenges of tomorrow. Whether the needs require a science gateway for documentation, or one that can interface with storage and HPC resources, come learn about the tooling and strategies we use to deliver customizable on-demand gateways. We will present the tech we use and describe the strategies and software design philosophies we are implementing to generate composable, extendable gateways that can deliver everything from full-fledged multi-service solutions to a-la-carte style service selection.