Search

Scientists in geospatial data-driven fields often spend significant efforts in “wrangling data”, i.e., accessing and processing data to make them usable in modeling and analysis tools. This project has created GeoEDF, an extensible geospatial data framework, to reduce this barrier by creating seamless connections among platforms, data and tools, making large distributed geospatial datasets directly usable in models and tools. Through an extensible set of community-contributed, modular and reusable data connectors and processors, GeoEDF abstracts away the complexity of acquiring and utilizing remote datasets. Researchers can string them together into a workflow for execution in various environments including a well-established science gateway MyGeoHub, JupyterHub-based deployments, and as a Docker container on laptops. By bringing data to the science, GeoEDF helps accelerate data-driven discovery and improve FAIR science practices.

Additional file 2.

Additional file 1.

While there is no shortage of research teams developing complex modeling code across the spectrum of scientific domains, most are not prepared to transition their simulations from private labs into publicly accessible gateway platforms. Research software engineers (RSEs) can bridge this vital knowledge gap, and in doing so help researchers both reach a greater audience for their work and build a greater value returned on their research funding. This presentation details the case of Purdue University RSEs helping build a gateway to the A-RESCUE simulation[1], a realistic vehicle traffic model for studying how efficiently an entire population will evacuate through a city-scale road network. Each project has unique challenges that will place constraints on what solutions are viable. RSEs for this project balanced what is technically the best direction for the architecture against the future roles of the researchers. They will be maintaining the gateway independently after the collaboration period, operating within the computing environment they have already created, and staying within the bounds of their grant deliverables. This project began by evaluating the existing simulation interface. The research team had developed a robust model which operated solely through a framework vendor provided GUI. This ran in a standard workstation but required each end user to setup a development environment and run the model through a specialized tool. For potential research collaborators this may be viable, but it would set too high a bar for any potential lay-user such as emergency personnel wanting to plan city evacuations. RSEs were able to raise these concerns and propose the creation of a new server interface built around an alternate batch-mode API found within the framework’s documentation. This allowed the simulation to be made available to new classes of remote users through a simple network socket with almost no modification of the underlying simulation code and no local software installation necessary. A-RESCUE is written in the Java language using the Repast Simphony agent-model framework[2]. This factored into the solution platform choice as the researchers would need to self-maintain the gateway after development concluded. NodeJS [3] was selected as the best platform for the gateway as it has excellent community support to help the team with debugging future issues, is written in JavaScript at which they were already proficient, and meshed well with their model Java code. With the server solution determined, the RSEs aided the team with refining a prototype web client and composing all the systems together within servers they had already purchased for the grant. Working with campus computing staff, they gained access to additional resources for server management and professional UI development for their web client prototype. Research teams utilizing campus RSEs to build scientific gateways adds value for all involved. Rather than merely optimizing model code, RSEs can create interfaces between applications, introduce new technologies, and streamline workflows for easier maintainability. These collaborations generate new audiences for both the research team’s work and the institution’s computing center. Promoting these collaborations should be a priority to all organizations.

A minimal set of reusable software packages and methodologies is described which have enabled the efficient creation of multiple, self-contained, data-centric web applications ("tools"). The tools themselves are dedicated to the analysis, sharing, and visualization of data produced by researchers from diverse domains. The development template described is intended to support a range of development contexts, developer proficiencies, and hosting environments. Specifically, we organized the tools' codebases within a simplified Model–View–Controller pattern and used Anaconda, Python, Jupyter, Docker, and Voilà [1] to develop and deploy the tools. The basis of the tool code is Python running as a Jupyter notebook. However, we forgo traditional, cell-based notebook organization and use Jupyter only to enable the use of ipywidgets for user interface controls. Further, this development context allowed us to leverage the use of libraries such as Pandas for data access, Ipyleaflet for geospatial visualization, and Matplotlib and Plotly for plots, charts, and graphs. A portion of the tools produced were published as stand-alone, containerized, independently hosted [2] web applications. We relied on Docker and Voilà to create containerized applications which include their own web servers. Others were published as tools embedded within a HUBzero [3] based science gateway. In this context, we relied on the gateway's Jupyter server and its ability to render the tool using Jupyter's "Appmode" extension. Thus far six tools have been developed using this template with subsequent tools only taking approximately one third as long as the initial effort employed in developing the template. The development template emerged in order to easily enhance code created by researchers and supplement it with a user interface (UI) in order to expose a tool allowing others to explore the researchers' data through a variety of rich UI elements that provide data-specific filtering, query, and visualization capabilities. One requirement was that the researchers themselves, not dedicated web developers, should be able to enhance and maintain the tool code. The template then evolved as we used it as the basis for the development of other tools and then as the basis for student interns to quickly build data-centric applications. We describe the steps we use to create these types of tools as well as the pitfalls we encountered as we assembled the template over time.

Collaborative cyberinfrastructure (CI) is rapidly becoming a key integral component in multidisciplinary research in several domains. However, research software groups that often create these CIs face major challenges in sustaining the CIs in the long run, including uncertainties in funding and rapid technology updates. Such issues often dissuade researchers from using these CIs for initial data and software sharing, leading to roadblocks to future reproducibility and collaboration. In this poster, we illustrate the evolution of both the software architecture and interoperability model of our science gateways that has led to their sustained use and increasing adoption by various researchers in support of the FAIR (Findable, Accessible, Interoperable, Reusable) enabled research and learning. Inspired by these use cases, our methods in turn have produced low-barrier, turn-key solutions for initial data and code sharing, extensible and scalable solutions for resource-intensive data analysis workflows, as well as a novel sustainability model for tiding over funding uncertainties. A poster presented at the SIAM CSE21 conference, minisymposterium PP7: Software Productivity and Sustainability for CSE.

Collaborative cyberinfrastructure (CI) is rapidly becoming a key component in multidisciplinary research of complex systems as scientists race to understand and create innovative solutions to the 21st century’s grand challenges such as resource sustainability and resilience in the face of population growth, food and water security, and ever more frequent and damaging natural disasters. Responding to the need to share data and software tools, collaborative CIs, connected via shared platforms or common communication protocols, help make research more accessible and reproducible, and have brought significant impacts to research communities. However, research software groups that often create these CIs face major challenges in sustaining the CIs in the long run, including uncertainties in funding and rapid technology updates. Such issues often dissuade researchers from using CIs for initial data and software sharing, leading to roadblocks to reproducibility and collaboration. In this presentation, we will describe how we evolved the software architecture and interoperability model of our science gateways in support of the FAIR (Findable, Accessible, Interoperable, Reusable) enabled research and learning. Our methods have produced low-barrier, turn-key solutions for initial data and code sharing, extensible and scalable solutions for resource-intensive data analysis workflows, as well as a novel sustainability model for tiding over funding uncertainties.

Disasters are sudden, calamitous events that bring great damage, loss or destruction to large populations and regions. Natural hazards pose a great threat and challenge to all human societies. The diversity in hazards requires a multi-disciplinary approach, integrating scientific research and application of the findings with pre-disaster prediction, decision making and assessment based on complete, scientific and reliable data. This White Paper aims to identify the gaps in technology and relevant policies that prevent effective interconnection of disaster-related data and information for use in research, education and public engagement. It presentsset of recommendations summarized for a better understanding of disasters and more effective ways of mitigating and reducing the impact of disasters on lives and properties.

This poster presents the GeoEDF project that is developing an extensible geospatial data framework to make scientific datasets directly usable in computational tools. Through an extensible set of modular and reusable data connectors and processors, GeoEDF is designed to abstract away the complexity of acquiring and utilizing data from remote sources. Researchers can string them together into a workflow that can be executed in various environments including HUBzero tools, HPC resources, or Jupyter Notebooks. By bringing data to the science, GeoEDF will help accelerate data-driven discovery, while ensuring that data is not siloed and improving the practice of FAIR science.

A lightning talk to present the NSF CSSI Framework project GeoEDF: an extensible geospatial data framework towards FAIR (Findable, Accessible, Interoperable, Reusable) science

A lightning talk to present the NSF CSSI Framework project GeoEDF: an extensible geospatial data framework towards FAIR (Findable, Accessible, Interoperable, Reusable) science

This poster presents the GeoEDF project that is developing an extensible geospatial data framework to make scientific datasets directly usable in computational tools. Through an extensible set of modular and reusable data connectors and processors, GeoEDF is designed to abstract away the complexity of acquiring and utilizing data from remote sources. Researchers can string them together into a workflow that can be executed in various environments including HUBzero tools, HPC resources, or Jupyter Notebooks. By bringing data to the science, GeoEDF will help accelerate data-driven discovery, while ensuring that data is not siloed and improving the practice of FAIR science.

Collaborative science has popularized the use of web-based research platforms or “science gateways”. With the growing emphasis on FAIR (Findable, Accessible, Interoperable, Reusable) science, gateways need to enable scientific workflows conducted on them to comply with the FAIR principles. In practice, scientific workflows still involve a mix of non-reusable code, desktop tools, and intermediate data handling; presenting significant challenges. In earth sciences, the complexity of using diverse, large quantities of data from remote repositories compounds these challenges. For example, hydrologists and agriculture economists in interdisciplinary research each need to use diverse data sets for their specialty domain but also need to connect their computational models through exchange of data. Data sources can range from repositories managed by NASA, USGS, etc., to sensor arrays in smart cities, and crowdsourcing. Due to the inherent massive volume, high dimensionality, heterogeneous formats, and variability in access protocols, researchers often spend a lot of time manually collecting and processing data using custom code, instead of focusing on scientific questions.We are developing a data framework (GeoEDF) that abstracts away the complexity of acquiring and utilizing data from data repositories. Extensible data connectors will implement common data query and access protocols such as HTTP, OPeNDAP, FTP, Globus, etc. supporting both static and streaming data. Data sources can be configured by simply specifying the data location, authentication, access protocol, etc. Connectors are parameterizable, allowing reuse for subdataset, temporal, and, spatial choices. Extensible data processors will implement common and domain-specific geospatial data processing such as resampling, format conversion, or a scientific simulation model. A plug-and-play workflow composer will allow users to string together data connectors and processors into reproducible workflows that can be executed in heterogeneous environments. Automated metadata extraction and annotation will be integrated into such workflows, supporting FAIR science through ease of data discovery. By bringing data to the science, GeoEDF will accelerate data-driven discovery.

Date: October 25th, 2019Time: 1:00-2:00 pm ETLocation: GoToMeetingWith the growing emphasis on FAIR (Findable, Accessible, Interoperable, Reusable) science, science gateways need to enable scientific workflows conducted on them to be compliant with the FAIR principles. In practice, scientific workflows still involve a mix of non-reusable code, desktop tools, and intermediate data handling; presenting significant challenges to ensuring this compliance. In Earth sciences, the complexity of using diverse, large quantities of data from remote repositories compounds these challenges. For example, hydrologists and agricultural economists in interdisciplinary research need to not only use diverse data sets for their specialty domain but also connect their computational models through exchange of data. Data sources can range from repositories managed by NASA, USGS, etc.,to sensor arrays in smart cities, and crowdsourcing. Due to the inherent massive volume, high dimensionality, heterogeneous formats, and variability in access protocols, researchers often spend a lot of time manually collecting and processing data using custom code, instead of focusing on scientific questions.GeoEDF, an extensible geospatial data framework, aims to lessen and possibly remove such barriers by creating seamless connections among platforms, data and tools, making large scientific and social geospatial datasets directly usable in scientific models and tools. GeoEDF is designed to abstract away the complexity of acquiring and utilizing data from diverse data providers. Extensible data connectors will implement common data query and access a variety of protocols supporting both static and streaming data. Extensible data processors will implement common and domain-specific geospatial data processing such as resampling, format conversion, or a scientific simulation model. A plug-and-play workflow composer will allow users to string together data connectors and processors into reproducible workflows that can be executed in heterogeneous environments. Automated metadata extraction and annotation will be integrated into such workflows, supporting FAIR science through ease of data discovery and reproduction. By bringing data to the science, GeoEDF will accelerate data-driven discovery. This presentation will provide an overview of the project, including its objectives, system design, scientific use cases, and anticipated outcomes.The recording of this webinar can be found at https://youtu.be/arWK-zGAkOg.

Geospatial data are present everywhere today with the proliferation of location-aware computing devices. This is especially true in the scientific community where large amounts of data are driving research and education activities in many domains. Collaboration over geospatial data, for example, in modeling, data analysis and visualization, must still overcome the barriers of specialized software and expertise among other challenges. Built using the HUBzero software, MyGeoHUB tries to address these needs by providing an open platform for geospatial tool development and sharing. It creates a geospatial data architecture that integrates spatial data management, mapping and visualization, and interfaces in the HUBzero platform for scientific collaborations. In addition to HUBzero’s built-in mechanism that supports self development and contribution of scientific tools, it also provides software building blocks including the geo-rendering enabled Rappture toolkit, a generic Python mapping library, geospatial data exploration and publication tools, and an integrated online geospatial data management solution. This poster will showcase how MyGeoHUB enable researchers and educators to self-manage their scientific data, rapidly create GIS-enabled tools, share geospatial data and tools on the web, and build dynamic workflows connecting data and tools, all without requiring significant software development skills, GIS expertise or IT administrative privileges. Team members: Larry Biehl, Rajesh Kalyanam, Rob Campbell, Betsy Hillery, Carolyn Ellis, Derrick Kearney, Leif Delgass, Kevin Wojkovich, Erich Huebner, Hou-Jen Ko, Nelson Villoria, Venkatesh Merwade, Adnan Rajib, Jaewoo Shin, I Luk Kim

Geospatial Data Analysis Building Blocks (GABBs) is an NSF-funded Data Infrastructure Building Blocks (DIBBS) project to create a powerful yet easy-to-use web-based system that provides the software building blocks allowing researchers. who are typically not a computer expert, to self-manage, curate, share, analyze, and visualize geospatial data for their research. This opens the way for rapid development of a variety of web-enabled interactive tools for probing and presenting geospatial data. The development of GABBS building blocks is driven by the requirements from several user communities, including hydrologic modeling and data sharing, applied economics modeling workflow, meteorological data management and visualization, K-12 education tools, and high education course development with online modeling support. GABBs also integrate XSEDE resources, Globus, iRODS and other cyberinfrastructure support, enabling large scale computation and data capabilities for the user community. This poster will present the design and implementation of the GABBS software consisting of geospatial data management functions that are integrated into HUBzero’s Project data space, software and hardware based mapping libraries and map viewer widgets for geospatial tool development, and dynamic invocation of data analysis tools from the Hub Project data space.Team members include: Larry Biehl, Rajesh Kalyanam, Rob Campbell, Betsy Hillery, Carolyn Ellis, Derrick Kearney, Leif Delgass, Kevin Wojkovich, Erich Huebner, Hou-Jen Ko, Nelson Villoria, Venkatesh Merwarde, Adnan Rajib, Jaewoo Shin, I Luk Kim

The HUBzero cyberinfrastructure framework has been used to build science gateways for various domains ranging from nanotechnology and biomedical sciences to high performance computing and engineering education to name a few. Several HUBzero-based science gateways also work with geospatial data in the context of environmental modeling, drought prediction, and the effects of climate change on agriculture. While these diverse research fields may work with varied geospatial data content and formats, they share common needs of managing, visualizing and processing the data. Geospatial data utilizes highly structured file formats, often including metadata that is conducive to automatic extraction and subsequent search. Similarly, irrespective of the actual file format employed, geospatial data at its core includes uniformly visualizable features. Developed in response to these common needs, Geospatial Data Analysis Building Blocks (GABBs) add these capabilities to the HUBzero software framework, enabling their availability out of the box for science gateways working with geospatial data. GABBs is integrated into the HUBzero framework, extending standard file management with iData, a value-added file management system that provides automated metadata extraction and geospatial preview support; various tools for visualizing and processing geospatial data; software and hardware visualization building blocks that simplify building new visualization tools with standard map elements; and finally a service API to iData, enabling remote data transfer from field devices and third party software. GABBs simplifies research workflows via a seamless interface between data and tools on HUBzero science gateways, with iData functioning as a uniform file management interface for data ingested remotely via the service API, uploaded manually via a web interface or simulation output saved from tools. GABBs-enabled HUBzero gateways are available out of the box, and individual GABBs capabilities can be added to any HUBzero science gateway. The publicly accessible MyGeoHub gateway is the central demonstration platform for all GABBs capabilities; a GABBs-enabled HUBzero gateway can be deployed to Amazon Web Services (AWS) completely automatically, in under an hour; and open-source Linux packages can be used to add individual GABBs components to any existing HUBzero gateway. Our poster will depict these various GABBs software building blocks and typical research workflows that they enable and simplify. It will also depict the various ways in which interested users can access or deploy a GABBs-enabled science gateway.

Science gateways have gained popularity and adoption in the past few years as an efficient and user friendly way to share scientific data, models, tools, and workflows in a community. However, all science gateways face a major hurdle of sustainability in the long term. When funding ends, the gateway developed for the project is likely to become obsolete due to non-trivial maintenance costs. We describe here a cost-sharing solution based on the HUBzero gateway framework. Instead of dedicating one hub for each project, this solution hosts multiple projects with synergies on one physical hub while at the same time providing branded individual web space for each project by leveraging HUBzero’s support of super groups. This reduces the cost in maintaining the underlying infrastructure, which is further shared among multiple projects, keeping the hosted gateways alive in the long term. We applied this model to MyGeoHub and will discuss our implementation experience.

Responding to the need for geospatial tool and data capabilities in HUBzero from various communities, a Purdue team is developing and integrating geospatial capabilities into the HUBzero software. Funded by the National Science Foundation’s Data Infrastructure Building Blocks (DIBBs) initiative, the follow-on program to the DataNet program, the GABBs project is a four-year software development effort aiming at enabling researchers, students and citizens to share geospatial data and tools online without requiring significant amounts of software expertise or IT administrative privileges. The new geospatial functions are being integrated into the HUBzero framework such as RAPPTURE Toolkit for rapid tool development, web component interface, resource management and publishing, and so on, and will be available via the open source HUBzero software releases. The GABBs project is building on past and ongoing projects such as DRINET, GeoShare, and WaterHUB, as well as integrating with other community efforts including iRODS distributed data management and Globus data transfer. This presentation provides an overview of the project, usage examples, and expected outcome. The project seeks input and requirements from the broader user community to guide its development.

HydroShare poster presented at SI2 PI meeting in Arlington, Virginia, February 24, 2014

DRINET is an NSF funded data interoperability project. It is creating a research environment for collecting and disseminating local to regional scale drought information while interoperating with other resources and tools. This presentation will provide an overview of the project, including the DRINET cyberinfrastructure framework, interactive online tools and datasets developed and integrated under this projects, and ongoing work in key research areas (metadata standardization, drought characterization, drought impacts on air quality, water quality, agriculture, urban planning and resilience, etc). This will be a team presentation with multiple presenters: Drs. Carol Song, Jake Carlson, Rao Govindaraju, Indrajeet Chaubey and Dev Niyogi.

DRINET is a research environment for collecting and disseminating local to regional scale drought information while interoperating with other resources and tools. The disseminated information via the DRINET will be based on a comprehensive evaluation of causal factors for short and long term droughts, as well as on a standardization of data formats and collection practices. It thus lays the foundation for investigating and providing improved drought risk and trigger indicators.