Search

High Performance Compute (HPC) clusters often produce intermediate files as part of code execution and message passing is not always possible to supply data to these cluster jobs. In these cases, I/O goes back to central distributed storage to allow cross node data sharing. These systems are often high performance and characterised by their high cost per TB and sensitivity to workload type such as being tuned to small or large file I/O. However, compute nodes often have large amounts of RAM, so when dealing with intermediate files where longevity or reliability of the system is not as important, local RAM disks can be used to obtain performance benefits. In this paper we show how this problem was tackled by creating a RAM block that could interact with the object storage system Ceph, as well as creating a deployment tool to deploy Ceph on HPC infrastructure effectively. This work resulted in a system that was more performant than the central high performance distributed storage system used at Diamond reducing I/O overhead and processing time for Savu, a tomography data processing application, by 81.04% and 8.32% respectively., Comment: 5 pages, 4 figures

Managing Research Data, and making it available for use/reuse by others in line with the UKRI Concordat on Open Research Data and FAIR principles, is a major issue for research-intensive organisations. In this case study we outline an institute-wide review of data management in practice, carried out at the Rosalind Franklin Institute (The Franklin) in partnership with external consultants, Curlew Research, in March 2022. We aim to describe the processes involved in undertaking a review of the services already in place that support good practice in managing research data, and their uptake, with an emphasis on the methodology used. We conducted interviews with scientific Theme Leads which set the scope for the Data Management Workshops subsequently held with Researchers. Workshops were valuable in both providing actionable insights for data management and in priming the audience for future discussions. The final report produced for The Franklin, summarising results of the analysis, provides a snapshot of current practice for the Institute, highlighting what is working well and where improvements might be made, and provides a benchmark against which development can be measured in the coming years. The Review will continue to be conducted on an annual basis, reflecting changes in a fast-moving area and enabling an agile approach to research data management., Comment: 17 pages, 6 figures

In this work we present affinity-VAE: a framework for automatic clustering and classification of objects in multidimensional image data based on their similarity. The method expands on the concept of $\beta$-VAEs with an informed similarity-based loss component driven by an affinity matrix. The affinity-VAE is able to create rotationally-invariant, morphologically homogeneous clusters in the latent representation, with improved cluster separation compared with a standard $\beta$-VAE. We explore the extent of latent disentanglement and continuity of the latent spaces on both 2D and 3D image data, including simulated biological electron cryo-tomography (cryo-ET) volumes as an example of a scientific application.

Diamond Light Source is the UK's National Synchrotron Facility and as such provides access to world class experimental services for UK and international researchers. As a user facility, that is one that focuses on providing a good user experience to our varied visitors, Diamond invests heavily in software infrastructure and staff. Over 100 members of the 600 strong workforce consider software development as a significant tool to help them achieve their primary role. These staff work on a diverse number of different software packages, providing support for installation and configuration, maintenance and bug fixing, as well as additional research and development of software when required. This talk focuses on one of the software projects undertaken to unify and improve the user experience of several experiments. The "mapping project" is a large 2 year, multi group project targeting the collection and processing experiments which involve scanning an X-ray beam over a sample and building up an image of that sample, similar to the way that google maps bring together small pieces of information to produce a full map of the world. The project itself is divided into several work packages, ranging from teams of one to 5 or 6 in size, with varying levels of time commitment to the project. This paper aims to explore one of these work packages as a case study, highlighting the experiences of the project team, the methodologies employed, their outcomes, and the lessons learnt from the experience., Comment: 7 Pages, 5 figures, conferece

Diamond Light Source (DLS), the UK synchrotron facility, attracts scientists from across the world to perform ground-breaking x-ray experiments. With over 3000 scientific users per year, vast amounts of data are collected across the experimental beamlines, with the highest volume of data collected during tomographic imaging experiments. A growing interest in tomography as an imaging technique, has led to an expansion in the range of experiments performed, in addition to a growth in the size of the data per experiment. Savu is a portable, flexible, scientific processing pipeline capable of processing multiple, n-dimensional datasets in serial on a PC, or in parallel across a cluster. Developed at DLS, and successfully deployed across the beamlines, it uses a modular plugin format to enable experiment-specific processing and utilises parallel HDF5 to remove RAM restrictions. The Savu design, described throughout this paper, focuses on easy integration of existing and new functionality, flexibility and ease of use for users and developers alike., Comment: 10 pages, 10 figures, 1 table

Recent applications in computer vision have come to heavily rely on superpixel over-segmentation as a pre-processing step for higher level vision tasks, such as object recognition, image labelling or image segmentation. Here we present a new superpixel algorithm called Hierarchical Piecewise-Constant Super-regions (HPCS), which not only obtains superpixels comparable to the state-of-the-art, but can also be applied hierarchically to form what we call n-th order super-regions. In essence, a Markov Random Field (MRF)-based anisotropic denoising formulation over the quantized feature space is adopted to form piecewise-constant image regions, which are then combined with a graph-based split & merge post-processing step to form superpixels. The graph and quantized feature based formulation of the problem allows us to generalize it hierarchically to preserve boundary adherence with fewer superpixels. Experimental results show that, despite the simplicity of our framework, it is able to provide high quality superpixels, and to hierarchically apply them to form layers of over-segmentation, each with a decreasing number of superpixels, while maintaining the same desired properties (such as adherence to strong image edges). The algorithm is also memory efficient and has a low computational cost.