Search

This document presents the progress of the ExPaNDS project between March 2021 and February 2023. It reproduces the explanation of the work carried out by our partners as provided to the European Commission in the second and last periodic report of the project.

Poster describing the Open EM Data Network (OpEM), a consortium of Swiss electron microscopy facilities to improve open research data practices. Abstract The Open EM Data Network (OpEM) is a consortium of Swiss electron microscopy facilities working together to: Improve Open Research Data (ORD) practices in the Swiss EM community Provide an open and FAIR repository for Swiss EM data not hosted elsewhere Standardize EM metadata & automate collection at Swiss EM facilities Follow a consistent data lifecycle when collecting data at different facilities Streamline publication of EM data into open repositories like EMPIAR, EMDB, and the PDB OpEM will target both researchers producing EM data and consumers of open data for additional science. Data producers benefit from more streamlined data collection, standardized facilities, easier deposition for publication, and adherence to data management policies. The wider availability of open EM data brings numerous benefits, including reproducing results, applying new techniques to old data, training AI & other new methods, and mining data for new insights. &nbsp

Over 115 participants, including researchers, managers and IT professionals from the photon and neutron communities and representatives of the European Commission, attended the 2nd online Photon and Neutron EOSC Symposium, jointly organised by PaNOSC and ExPaNDS on 26 October 2021. Find in this booklet all presentations, the recordings will also be made available shortly.

This document presents the milestone achieved for a metadata catalogue release in the domain of photon and neutron (PaN) science. With the primary goal of supporting PaN FAIR data catalogue services, we have developed a self-contained, stand-alone metadata catalogue release that facilities can download to test/try and play with. The work represents milestone MS12 of the Horizon 2020 ExPaNDS project.

This document presents the progress of the ExPaNDS project after its first 18 months of activities, spanning from September 2019 to February 2021. It reproduces the explanation of the work carried out by the ExPaNDS partners as provided to the European Commission in the first periodic report of the project.

We present ontologies for the domain of photon and neutron (PaN) science. With the primary goal of supporting PaN FAIR data catalogue services, we have developed three ontologies: PaN experimental techniques (PaNET), an ontology of NeXus definitions (NeXusOntology), and a semantic integration ontology for the PaN domain (PaNmapping). The ontologies are presented as initial versions, supported by community development workflows. The work represents deliverable D3.2 of the Horizon 2020 ExPaNDS project.

An operational framework document describing the sustainable, coordinating organisational structure spanning national Photon and Neutron Research Infrastructures and EOSC. Common rules and best practices for implementing the national RI’s analysis services, and future data analysis services, within the EOSC.

This document summarises work in ExPaNDS Data Analysis Services work package (WP4) for the preparation and publication of Photon and Neutron reference data sets to be used to adapt, align and validate the project’s Prototype Data Analysis Services.

The ExPaNDS project aims at deploying into EOSC Data Catalogues and data analysis services. This document describes the status, a gap analysis, and a roadmap required to achieve harmonised and federated (meta)data catalogues within EOSC of the participating national Photon and Neutron (PaN) Research Infrastructures (RIs).

Research Infrastructures within the Photon and Neutron science community have a significant track record in publishing and implementing data policies to support the access and use of experimental data generated by the user. Recent developments within European science policy, in particular the drive towards supporting FAIR data in the European Open Science Cloud has meant that it is timely to review and revise these data policies. The PaNOSC project has developed a new PaNOSC Data Policy Framework with the contribution of ExPaNDS partners. The aim of this Deliverable 2.1: Draft extended data policy framework for photon and neutron research infrastructures is to reflect upon and extend that work. We review the FAIR data policy landscape at European and national levels, consider the current state of data policy adoption and implementation at ExPaNDS partner facilities, and examine existing FAIR ecosystem data policy recommendations, in particular, from the Turning FAIR into reality report and the recent FAIRsFAIR Deliverable 3.3: Policy enhancement recommendations In response, we make twenty-six recommendations of our own that serve to translate these recommendations to the local level of photon and neutron research infrastructures. As the report’s key output, we present thirty elements that should inform a data policy framework for photon and neutron research infrastructures. These focus on enabling FAIR data and are those elements about which RIs need to make choices — both on the level of commitments that they themselves are prepared to make as well as with regards to the obligations that they will place on users. We conclude with some next steps that will inform the final version of this deliverable: D2.3 Final data policy for photon and neutron research.&nbsp

The ExPaNDS project aims at deploying into EOSC data catalogues and data analysis services. This document describes the proposed architecture for these services and how they are planned to integrate into EOSC. The first part elaborates on our plan to abstract the data and analysis services with common APIs in order to make them available to portals and processing pipelines without knowing where the actual facility data, analysis stacks and ICT infrastructure resides. Then, as this is one of ExPaNDS high level objectives, the document describes the relationship between services provided by ExPaNDS RIs and the EOSC as well as their interdependencies.

The Diamond Light Source data analysis infrastructure, Zocalo, is built on a messaging framework. Analysis tasks are processed by a scalable pool of workers running on cluster nodes. Results can be written to a common file system, sent to another worker for further downstream processing and/or streamed to a LIMS. Zocalo allows increased parallelization of computationally expensive tasks and makes the use of computational resources more efficient. The infrastructure is low-latency, fault-tolerant, and allows for highly dynamic data processing. Moving away from static workflows expressed in shell scripts we can easily re-trigger processing tasks in the event that an issue is found. It allows users to re-run tasks with additional input and ensures that automatically and manually triggered processing results are treated equally. Zocalo was originally conceived to cope with the additional demand on infrastructure by the introduction of Eiger detectors with up to 18 Mpixels and running at up to 560 Hz framerate on single crystal diffraction beamlines. We are now adapting Zocalo to manage processing tasks for ptychography, tomography, cryo-EM, and serial crystallography workloads., Proceedings of the 17th International Conference on Accelerator and Large Experimental Physics Control Systems, ICALEPCS2019, New York, NY, USA

The ExPaNDS project is a collaboration between 10 national Photon and Neutron Research Infrastructures (PaN RIs) as well as EGI. The project aims to deliver standardised, interoperable, and integrated data sources and data analysis services for Photon and Neutron facilities, within the context of the European Open Science Cloud (EOSC). As part of its programme, it is considering how to enable and promote the generation and use of FAIR data within facilities’ processes (work package 2), and the harmonisation of tools and processes for data cataloguing and management (work package 3). WP2 and WP3 are conducting a landscape analysis in order to establish a baseline on the current state of the 10 participating institutions on FAIR data policies and data management practises. This activity can be compared with a similar activity which was undertaken within the PaNOSC project, which covers the transnational (ESFRI) facilities in the same domain. This baseline can then be used to steer the subsequent activity of the project and to assess its impact. This report summarises the results of a survey of the 10 participating institutes which was undertaken in December 2019.

We report on the tests undertaken to establish the technical requirements of a data archive/retrieval system to preserve and share scientific data across Instruct-ERIC facilities. Network monitoring tests were undertaken to assess the data transfer capabilities of various research facilities, highlighting the importance of network topology on transfer reliability, especially at the local level. The process used to archive and retrieve large data sets has evolved from a prototype process to a proven solution that could be adopted to centralise and share data between Instruct-ERIC Centres.

The high data rates produced by the latest generation of detectors, more efficient sample handling hardware and ever more remote users of the beamlines at Diamond Light Source require improved data reduction and data analysis techniques to maximize their benefit to scientists. In this paper some of the experiment data reduction and analysis steps are described, including real time image analysis with DIALS, our Fast DP and xia2-based data reduction pipelines, and Fast EP phasing and Dimple difference map calculation pipelines that aim to rapidly provide feedback about the recently completed experiment. SynchWeb, an interface to an open source laboratory information management system called ISPyB (co-developed at Diamond and the ESRF), provides a modern, flexible framework for managing samples and visualizing the data from all of these experiments and analyses, including plots, images, and tables of the analysed and reduced data, as well as showing experimental metadata, sample information., Proceedings of the 15th Int. Conf. on Accelerator and Large Experimental Physics Control Systems, ICALEPCS2015, Melbourne, Australia

Many synchrotron beamlines offer some form of continuous scanning for either energy scanning or sample mapping. However, this is normally done on an ad-hoc beamline by beamline basis. Diamond has recently embarked on an ambitious project to define how to implement continuous scanning as the standard way of doing virtually all mapping tasks on beamlines. The project is split into four main areas: 1) User interfaces to describe the mapping process in a scientifically relevant way, generating a scan description that can be used later; 2) The physical process of scanning and coordinating hardware motion and detector data capture across the beamline; 3) Capture of the detector data and all the associated meta-data to disk. Deciding and describing the layout of the file (or files) for the main use cases; 4) Display and analysis of live data and display of processed data. In order to achieve this common approach across beamlines, the standard software used throughout the facility (Delta Tau motor controllers, EPICS, GDA and DAWN), has been built on., Proceedings of the 15th Int. Conf. on Accelerator and Large Experimental Physics Control Systems, ICALEPCS2015, Melbourne, Australia

This paper presents the involvement of Diamond Light Source (DLS) with the open source community, the Eclipse Science Working Group and how DLS is changing to share software development effort better between groups. The paper explains moving from product-based to bundle-based software development process which lowers reinvention, increases reuse and reduces software development and support costs. This paper details specific ways in which DLS are engaging with the open source community and changing the way that research institutions deliver open source code., Proceedings of the 15th Int. Conf. on Accelerator and Large Experimental Physics Control Systems, ICALEPCS2015, Melbourne, Australia