Search

The CERN IT Storage group operates multiple distributed storage systems to support all CERN data storage requirements. The storage and distribution of physics data generated by LHC and non-LHC experiments is one of the biggest challenges the group has to take on during LHC Run-3.EOS [1], the CERN distributed disk storage system is playing a key role in LHC data-taking. During the first ten months of 2022, more than 440PB have been written by the experiments and 2.9EB have been read out. The data storage requirements of LHC Run-3 are higher than what was previously delivered. The storage operations team has started investigating multiple areas to upgrade and optimize the current storage resources. A new, dedicated and redundant EOS infrastructure based on 100Gbit servers was installed, commissioned and deployed for the ALICE Online and Offline (O2) project. This cluster can sustain high-throughput data transfer between the ALICE Event Processing Nodes (EPN) and the CERN’s data center.This paper will present the architecture, techniques and workflows in place allowing EOS to deliver fast, reliable and scalable data storage to meet experiment needs during LHC Run-3 and beyond.

During the LHC era the XRootD framework has proven to be a critical component of numerous data management and software defined storage solutions (most importantly EOS, the CERN storage technology used for the LHC experiments), and as such grew into one of the most strategic storage technologies in the High Energy Physics (HEP) community. Over the last year significant developments in the area of the XRootD client have been introduced, making it even more reliable and robust, as well as easier to debug. Here, we present an overview of the new XRootD client and its main features, namely, support for erasure coding, in-flight data integrity checks, and the new record plug-in and replay tool that allow to record an I/O pattern and then replay it for debugging or benchmarking purposes.

EOS has been the main storage system at CERN for more than a decade, continuously improving in order to meet the ever evolving requirements of the LHC experiments and the whole physics user community. In order to satisfy the demands of LHC Run-3, in terms of storage performance and tradeoff between cost and capacity, EOS was enhanced with a set of new functionalities and features that we will detail in this paper. First of all, we describe the use of erasure coded layouts in a large-scale deployment which enables an efficient use of available storage capacity, while at the same time providing end-users with better throughput when accessing their data. This new operating model implies more coupling between the machines in a cluster, which in turn leads to the next set of EOS improvements that we discuss, targeting I/O traffic shaping, better I/O scheduling policies and tagged traffic prioritization. Increasing the size of the EOS clusters to cope with experiment demands, means stringent constraints on the data integrity and durability that we addressed by a re-designed consistency check engine. Another focus area of EOS development was to minimize the operational load by making the internal operational procedures (draining, balancing or conversions) more robust and efficient, to allow managing easily multiple clusters and avoid possible scaling issues. All these improvements available in the EOS 5 release series, are coupled with the new XRootD 5 framework which brings additional security features like TLS support and optimizations for large data transfers like page read and page write functionalities. Last but not least, the area of authentication/authorization methods has seen important developments by adding support for different types of bearer tokens that we will describe along with EOS specific token extensions. We conclude by highlighting potential areas of the EOS architecture that might require further developments or re-design in order to cope with the ever-increasing demands of our end-users.

Particle physics has an ambitious and broad experimental programme for the coming decades. This programme requires large investments in detector hardware, either to build new facilities and experiments, or to upgrade existing ones. Similarly, it requires commensurate investment in the R&D of software to acquire, manage, process, and analyse the shear amounts of data to be recorded. In planning for the HL-LHC in particular, it is critical that all of the collaborating stakeholders agree on the software goals and priorities, and that the efforts complement each other. In this spirit, this white paper describes the R&D activities required to prepare for this software upgrade.

Gentoo Prefix is explored to manage sophisticated physics software stacks. It will be shown that Gentoo Prefix is an advantageous package management solution for big physics experiments, for its reusability on heterogeneous host environments, its vast collection of ebuild recipes, its extensibility for the future computing architectures and its deep root in an open diverse community inside and outside science.

SIMD acceleration can potentially boost by factors the application throughput. Achieving efficient SIMD vectorization for scalar code with complex data flow and branching logic, goes however way beyond breaking some loop dependencies and relying on the compiler. Since the refactoring effort scales with the number of lines of code, it is important to understand what kind of performance gains can be expected in such complex cases. We started to investigate a couple of years ago a top to bottom vectorization approach to particle transport simulation. Percolating vector data to algorithms was mandatory since not all the components can internally vectorize. Vectorizing low-level algorithms is certainly necessary, but not sufficient to achieve relevant SIMD gains. In addition, the overheads for maintaining the concurrent vector data flow and copy data have to be minimized. In the context of a vectorization R&D for simulation we developed a framework to allow different categories of scalar and vectorized components to co-exist, dealing with data flow management and real-time heuristic optimizations. The paper describes our approach on coordinating SIMD vectorization at framework level, making a detailed quantitative analysis of the SIMD gain versus overheads, with a breakdown by components in terms of geometry, physics and magnetic field propagation. We also present the more general context of this R&D work and goals for 2018.

The ROOT Mathematical and Statistical libraries have been recently improved both to increase their performance and to facilitate the modelling of parametric functions that can be used for performing maximum likelihood fits to data sets to estimate parameters and their uncertainties. First, we report on the new functionalities introduced in ROOT’s TFormula and TF1 classes to build these models in a convenient way for the users. We show how function objects, represented in ROOT by TF1 classes, can be used as probability density functions and how they can be combined together—via an addition operator—to perform extended likelihood fit of several normalized components. We also describe the new operators introduced to perform the convolution of two functions. Finally, we report on the improvements in the performance of the ROOT fitting algorithm, by using SIMD vectorization when evaluating the model function on large data sets and by exploiting multi-thread parallelization when computing the likelihood function.

In the coming years, HEP data processing will need to exploit parallelism on present and future hardware resources to sustain the bandwidth requirements. As one of the cornerstones of the HEP software ecosystem, ROOT embraced an ambitious parallelisation plan which delivered compelling results. In this contribution the strategy is characterised as well as its evolution in the medium term. The units of the ROOT framework are discussed where task and data parallelism have been introduced, with runtime and scaling measurements. We will give an overview of concurrent operations in ROOT, for instance in the areas of I/O (reading and writing of data), fitting / minimization, and data analysis. This paper introduces the programming model and use cases for explicit and implicit parallelism, where the former is explicit in user code and the latter is implicitly managed by ROOT internally.

Experiments at the Large Hadron Collider (LHC) produce tens of petabytes of new data in ROOT format per year that need to be processed and analysed. In the next decade, following the planned upgrades of the LHC and its detectors, this data production rate is expected to increase at least ten-fold. Therefore, optimizing the ROOT I/O subsystem is of critical importance to the success of the LHC physics programme. This contribution presents ROOT’s approach for writing data from multiple threads to a single output file in an efficient way. Technical aspects of the implementation—the TBufferMerger class—and programming model examples are described. Measurements of runtime performance and the overall improvement relative to the case of serial data writing are also discussed.

The Physics programmes of LHC Run III and HL-LHC challenge the HEP community. The volume of data to be handled is unprecedented at every step of the data processing chain: analysis is no exception. Physicists must be provided with first-class analysis tools which are easy to use, exploit bleeding edge hardware technologies and allow to seamlessly express parallelism. This document discusses the declarative analysis engine of ROOT, RDataFrame, and gives details about how it allows to profitably exploit commodity hardware as well as high-end servers and manycore accelerators thanks to the synergy with the existing parallelised ROOT components. Real-life analyses of LHC experiments’ data expressed in terms of RDataFrame are presented, highlighting the programming model provided to express them in a concise and powerful way. The recent developments which make RDataFrame a lightweight data processing framework are described, such as callbacks and I/O capabilities. Finally, the flexibility of RDataFrame and its ability to read data formats other than ROOT’s are characterised, as an example it is discussed how RDataFrame can directly read and analyse LHCb’s raw data format MDF.

The development of the GeantV Electromagnetic (EM) physics package has evolved following two necessary paths towards code modernization. A first phase required the revision of the main electromagnetic physics models and their implementation. The main objectives were to improve their accuracy, extend them to the new high-energy frontier posed by the Future Circular Collider (FCC) programme and allow a better adaptation to a multi-particle flow. Most of the EM physics models in GeantV have been reviewed from theoretical perspective and rewritten with vector-friendly implementations, being now available in scalar mode in the alpha release. The second phase consists of a thorough investigation on the possibility to vectorise the most CPU-intensive physics code parts, such as final state sampling. We have shown the feasibility of implementing electromagnetic physics models that take advantage of SIMD/SIMT architectures, thus obtaining gains in performance. After this phase, the time has come for the GeantV project to take a step forward towards the final proof of concept. This takes shape through the testing of the full simulation chain (transport + physics + geometry) running in vectorized mode. In this paper we will present the first benchmark results obtained after vectorizing a full set of electromagnetic physics models.

The performance gap between CPU and memory widens continuously. Choosing the best memory layout for each hardware architecture is increasingly important as more and more programs become memory bound. For portable codes that run across heterogeneous hardware architectures, the choice of the memory layout for data structures is ideally decoupled from the rest of a program. This can be accomplished via a zero‐runtime‐overhead abstraction layer, underneath which memory layouts can be freely exchanged. We present the low‐level abstraction of memory access (LLAMA), a C++ library that provides such a data structure abstraction layer with example implementations for multidimensional arrays of nested, structured data. LLAMA provides fully C++ compliant methods for defining and switching custom memory layouts for user‐defined data types. The library is extensible with third‐party allocators. Providing two close‐to‐life examples, we show that the LLAMA‐generated array of structs and struct of arrays layouts produce identical code with the same performance characteristics as manually written data structures. Integrations into the SPEC CPU® lbm benchmark and the particle‐in‐cell simulation PIConGPU demonstrate LLAMA's abilities in real‐world applications. LLAMA's layout‐aware copy routines can significantly speed up transfer and reshuffling of data between layouts compared with naive element‐wise copying. LLAMA provides a novel tool for the development of high‐performance C++ applications in a heterogeneous environment. [ABSTRACT FROM AUTHOR]

In this work we present multi-dimensional mesoscale simulations of detonation initiation in energetic materials. We solve the reactive Euler equations, with the energy equation augmented by a power deposition term. The reaction rate at the mesoscale is modelled using density-based kinetics, while the deposition term is based on simulations of void collapse at the microscale, modelled at the mesoscale as hot spots. We carry out two- and three-dimensional mesoscale simulations of random packs of HMX crystals in a binder, and show that transition between no-detonation and detonation depends on the number density of the hot spots, the packing fraction, and the post-shock pressure of an imposed shock. In particular, we show that, for a fixed post-shock pressure, there exists a critical value of the number density of hot spots, such that when the number density is below this value a detonation wave will not develop. We highlight the importance of morphology to initiation by comparing with a homogeneous counterpart, and we compare relevant length scales by examining their corresponding power spectra. We also examine the effect of packing fraction and show that at low post-shock pressures there is significant variation in the initiation times, but that this variation disappears as the post-shock pressure is increased. Finally, we compare three-dimensional simulations with the experimental data, and show that the model is capable of qualitatively reproducing the trends shown in the data. [ABSTRACT FROM AUTHOR]

The development of a unique long cylindrical neutron source for broad area neutron activation analysis (NAA) is presented. This source uses inertial electrostatic confinement (IEC) to produce 2.54 MeV D-D or 14.1 MeV D-T fusion neutrons. The IEC offers a variety of applications ranging from security inspection stations to driven-subcritical research assemblies. This IEC design uses a biased grid to initial in a unique "star" mode plasma discharge creating beam-background gas (target) fusion reactions. They routinely produce ~108 2.54 MeV D-D neutrons/s at steady state. Pulsed operation has achieved up to 109 n/s. (~1011 n/s if a deuterium mixture is used). Indeed, a version of the IEC has been commercially produced as a portable neutron source for industrial NAA applications. While that IEC source used spherical geometry, the present work uses a cylindrical version which provides a unique long "linelike" neutron source for use in the present broad area NAA system. [ABSTRACT FROM AUTHOR]