Search

In the ever-expanding landscape of data management, navigating the diverse array of metadata catalogs such as SciCat, data publications on Invenio derivatives, and internal archives presents a formidable challenge. However, with the right strategies, this mosaic of data can be effectively combined and represented to unlock its full potential. In this talk, we delve into the intricacies of data fusion, exploring innovative approaches to harmonize metadata catalogs, data publications, and archives seamlessly. We will discuss the importance of interoperability and standardization in facilitating the integration process, enabling disparate data sources to coalesce into a cohesive ecosystem. Through conceptional examples and case studies, we will provide insights into the practical application of strategies.

An essential challenge by creating FAIR datasets is the often underestimated I, which stands for interoperability. Especially for a dataset that is meant to be exported from its ecosystem, it is important to store the metadata and data in the appropriate exchangeable format based on standards. One possible source for metadata is an electronic lab notebook that stores it in a structured manner. In many cases the internal structure does not match any established metadata scheme and a mapping is required for a meaningful export. This talk presents a concept of what is necessary to make a generic export from an electronic Labbook based on semantic Mediawiki for ingestion into SciCat or interoperable Nexus files. First an existing scheme (e.g. Nexus application class) needs to be imported and the class dependencies is stored. Reference to the origin of created classes and properties are essential. Within Mediawiki, templates are used to assign a set of properties to a page. Together with a configuration which describes through which properties the templates are interconnected a generic export is possible. Now by selecting a certain measurement, the export script can extract all essential metadata to create e.g. a SciCat export with project, instrument, sample and dataset information or create a nexus file and knows which groups needs to be created and under which path a property is stored. Since the information is bound to the property, the application of a mapping can be optimized in an iterative manner. This makes it a flexible procedure that is perfectly usable for existing documentation where metadata schemes are applied at a later stage or need to be updated. In addition, the reference to the original metadata scheme is known in the whole pipeline and could be included in the export.

The connection between metadata, data, and software and the integration in an overall lifecycle is crucial for effective data management in research. The generic data management lifecycle, developed at HZDR, bridges these critical components, ensuring seamless data discovery, accessibility, and reproducibility. The approach emphasises the planning of experiments, the role of metadata, data storage, as well as software versioning, and the final publication of digital research artefacts, which enables comprehensive traceability from data creation to long-term archiving. By aligning these elements in a unified procedure, we recommend a uniform lifecycle that can be adapted to different research areas, with a particular focus on photon science and community services such as SciCat that improve data integrity and promote collaborative research.

The HZDR team has harnessed the power of Python, Flask, Dash, ZeroMQ and Kafka in tandem with MongoDB, to create a suite of web-based applications that simplify the extraction of laser shot data and metadata from various distinct, heterogeneous, semi-automated data acquisition systems. The Shotsheet apps play a central role in manual logging, especially in a facility with highly flexible but manual operation modes. These apps also connect to other data sources like the Mediawiki ELN system. To enable further automation, we’ve developed the Experimental Shot Counter and enrichment app (escape), which handles incoming ZeroMQ messages from the Draco Laser system (experiment driver) and provides tailored ZeroMQ and Kafka messages within the Lab intranet. These messages can automate the readout and processing of measurements, as well as trigger mechanical actions. In the subsequent landscape of data management, navigating the diverse array of metadata catalogs – such as SciCat, data publications on Invenio derivatives, and internal archives – presents a formidable challenge. However, with the right strategies, this mosaic of data can be effectively combined and represented to unlock its full potential. In the HZDR data management ecosystem , we delve into the intricacies of data fusion, exploring innovative approaches to seamlessly harmonize metadata catalogs, data publications, and archives.

HELIPORT is a data management solution that aims at making the components and steps of the entire research experiment’s life cycle discoverable, accessible, interoperable and reusable according to the FAIR principles. Among other information, HELIPORT integrates documentation, scientific workflows, and the final publication of the research results - all via already established solutions for proposal management, electronic lab notebooks, software development and devops tools, and other additional data sources. The integration is accomplished by presenting the researchers with a high-level overview to keep all aspects of the experiment in mind, and automatically exchanging relevant metadata between the experiment’s life cycle steps. Computational agents can interact with HELIPORT via a REST API that allows access to all components, and landing pages that allow for export of digital objects in various standardized formats and schemas. An overall digital object graph combining the metadata harvested from all sources provides scientists with a visual representation of interactions and relations between their digital objects, as well as their existence in the first place. Through the integrated computational workflow systems, HELIPORT can automate calculations using the collected metadata. By visualizing all aspects of large-scale research experiments, HELIPORT enables deeper insights into a comprehensible data provenance with the chance of raising awareness for data management.

The connection between metadata, data, and software and the integration in an overall lifecycle is crucial for effective data management in research. The generic data management lifecycle, developed at HZDR, bridges these critical components, ensuring seamless data discovery, accessibility, and reproducibility. The approach emphasises the planning of experiments, the role of metadata, data storage, as well as software versioning, and the final publication of digital research artefacts, which enables comprehensive traceability from data creation to long-term archiving. By aligning these elements in a unified procedure, we recommend a uniform lifecycle that can be adapted to different research areas, with a particular focus on community services, such as SciCat, that improves data integrity and promote collaborative research.

Researchers at the Helmholtz-Zentrum Dresden-Rossendorf rely on a variety of systems and tools when it comes to administer their research data. Processes involving research data management include the project planning phase (proposal submission to the beamtime proposal management system, the creation of data management plans and data policies), the documentation during the experiment or simulation campaign (electronic laboratory notebooks, wiki pages), backup- and archival systems and the final journal and data publications (collaborative authoring tools, meta-data catalogs, software and data repositories, publication systems). In addition, modern research projects are often required to interact with a variety of software stacks and workflow management systems to allow reproducibility on the underlying IT infrastructure. The "HELmholtz ScIentific Project WORkflow PlaTform" (HELIPORT), which is currently developed by researchers at HZDR and their collaborators, tries to facilitate the management of research data and metadata by providing an overarching guidance system which combines all the information by interfacing the underlying processes and even includes a workflow engine which can be used to automate processes like data analysis or data retrieval.

In our HELIPORT workshop, we provided insights into our project and share our results. In addition, we would like to provide a platform for the presentation of similar projects, as well as extensions or integrations from the surrounding research areas. The overall goal of the workshop is bringing together different institutions with similar challenges and establishing a community around our HELIPORT project. We therefore encouraged our community to submit an abstract for a talk or poster. The submissions are dedicated to four thematic points: • HELIPORTuse-cases, • Scientificprojectandmetadatamanagement, • Experiment-specificandoverallmetadataand • Scientificworkflows. HELIPORT (Helmholtz ScIentific Project WORkflow PlaTform) itself is a project funded by the Helmholtz Metadata Collaboration, and runs from July 2021 until June 2023. HELIPORT aims to make the entire life cycle of a scientific project findable, accessible, interoperable and reusable according to the FAIR principles. In particular, our data management solution deals with the areas from the generation of the data to the publication of primary research data, the workflows carried out and the actual research results. For this purpose, a concept was developed which shows the various essential components and their connections.

When dealing with research data management, researchers at Helmholtz- Zentrum Dresden – Rossendorf (HZDR) face a variety of systems and tools. These range from the project planning phase (proposal management, data management plans and policies), over documentation during the experiment or simulation campaign, to the publication (collaborative authoring tools, metadata catalogs, publication systems, data repositories). In addition, modern research projects usually are required to interact with a variety of software stacks and workflow management systems to allow comprehensi- ble and FAIR science on the underlying IT infrastructure (HPC, data storage, network file systems, archival). This article first demonstrates the data management systems and services provided at HZDR, followed by an overview of a self-developed guidance system. It is concluded by a real-world example.

HELIPORT is a data management solution that aims at making the components and steps of the entire research experiment’s life cycle discoverable, accessible, interoperable and reusable according to the FAIR principles. Among other information, HELIPORT integrates documentation, scientific workflows, and the final publication of the research results - all via already established solutions for proposal management, electronic lab notebooks, software development and devops tools, and other additional data sources. The integration is accomplished by presenting the researchers with a high-level overview to keep all aspects of the experiment in mind, and automatically exchanging relevant metadata between the experiment’s life cycle steps. Computational agents can interact with HELIPORT via a REST API that allows access to all components, and landing pages that allow for export of digital objects in various standardized formats and schemas. An overall digital object graph combining the metadata harvested from all sources provides scientists with a visual representation of interactions and relations between their digital objects, as well as their existence in the first place. Through the integrated computational workflow systems, HELIPORT can automate calculations using the collected metadata. By visualizing all aspects of large-scale research experiments, HELIPORT enables deeper insights into a comprehensible data provenance with the chance of raising awareness for data management.

Data repositories like Zenodo have a limited list of metadata to search for. Metadata catalogues are designed to provide a community-specific parameters search, but their deployment has just started. These catalogues require metadata standards for interoperability, which in turn are often in development for many communities. To support publications with a metadata standard in the future, a two-step concept is presented in this article. It discusses how the electronic documentation should be constructed, in order to convert this later into a standardised schema for publication. We will present examples from the laser-plasma community for both steps, firstly how we deal with the complex challenges of metadata management and secondly for methods for developing metadata schemas.

At the High-Field High-Repetition-Rate Terahertz facility (TELBE) [1] at the Helmholtz-Zentrum Dresden-Rossendorf, ultrafast terahertz-induced dynamics can be probed in various states of matter with highest precision. The TELBE source is available for external users who are not familiar with the experiments's data management pipelines. This is especially a challenge as the measurements at TELBE are data intensive, producing as much as 20GB per experiment over few minutes. In this contribution, we present the guidance system HELIPORT [3] which manages the metadata of the associated project proposal and integrates systems for workflow management, electronic lab documentation and others. We showcase the integrated workflow for post-processing of the experimental data at TELBE with in-built exchange of metadata between our proposal submission service, the experiment control software and workflows managed and executed using UNICORE [2]. Among other information, HELIPORT integrates documentation, scientific workflows, and the final publication of the research results - all via already established solutions for proposal management, electronic lab notebooks, software development and devops tools, and other additional data sources. The integration is accomplished by presenting the researchers with a high-level overview to keep all aspects of the experiment in mind, and automatically exchanging relevant metadata between the experiment's life cycle steps. By visualizing all aspects of large-scale research experiments, HELIPORT enables deeper insights into a comprehensible data provenance with the chance of raising awareness for FAIR data management. [1] https://doi.org/10.1063/1.4978042 [2] https://doi.org/10.1109/HPCSim.2016.7568392 [3] https://doi.org/10.1145/3456287.3465477

HELIPORT is a data management solution that aims at making the components and steps of the entire research experiment’s life cycle discoverable, accessible, interoperable and reusable according to the FAIR principles. Among other information, HELIPORT integrates documentation, scientific workflows, and the final publication of the research results - all via already established solutions for proposal management, electronic lab notebooks, software development and devops tools, and other additional data sources. The integration is accomplished by presenting the researchers with a high-level overview to keep all aspects of the experiment in mind, and automatically exchanging relevant metadata between the experiment’s life cycle steps. Computational agents can interact with HELIPORT via a REST API that allows access to all components, and landing pages that allow for export of digital objects in various standardized formats and schemas. An overall digital object graph combining the metadata harvested from all sources provides scientists with a visual representation of interactions and relations between their digital objects, as well as their existence in the first place. Through the integrated computational workflow systems, HELIPORT can automate calculations using the collected metadata. By visualizing all aspects of large-scale research experiments, HELIPORT enables deeper insights into a comprehensible data provenance with the chance of raising awareness for data management.

This presentation provides insights into the development of the HZDR's in-house laboratory notebook for the documentation of scientific experiments. A special focus is placed on the integration of a variety of third-party systems in order to optimise the automated enrichment of data in the laboratory notebook and to facilitate documentation for scientists.

This data publication contains an overview to the Top-Level Architecture of the proposed HZDR Data Management Strategy with additional description of the various systems and services.  The Helmholtz-Zentrum Dresden-Rossendorf (HZDR) pursues a comprehensive data management strategy that is designed as an architecture of services to describe and manage scientific experiments in a sustainable manner. This strategy is based on the Helmholtz Codebase): For archiving, version control and provision of software, special data sets and workflows.   Proposal Management System (

The guidance system HELIPORT aims to make the entire life cycle of a project at the HZDR searchable, accessible, complete and reusable according to the FAIR principles, mentioned below. In particular, our data management solution deals with the areas from the generation of the data to the publication of primary research data, the workflows carried out and the actual research results. For this purpose, a concept was developed which shows the various essential components and their connections. Descriptions of the individual components can be found in our RODARE publication: 10.14278/rodare.193