Search

The new LHCb Vertex Locator (VELO) for LHCb, comprising a new pixel detector and readout electronics, will be installed in 2021 for data taking in Run 3 at the LHC. The electronics centers around the "VeloPix" ASIC at the front-end operating in a trigger-less readout at 40MHz. A custom serializer, called gigabit wireline transmitter (GWT), and associated custom protocol have been designed for the VeloPix. The GWT data are sent from the serializers of the VeloPix at a line rate of 5.12 Gb/s, reaching a total data rate of 2-3 Tb/s for the full VELO detector. Data are sent over 300-m optic-fiber links to the control and readout electronics cards for deserialization and processing in Intel Arria 10 FPGAs. Because of the VeloPix trigger-less design, latency variances up to 12 mu s can occur between adjacent datagrams. It is therefore essential to buffer and synchronize the data in firmware prior to onward propagation or suffer a huge CPU-processing penalty. This article will describe the architecture of the readout firmware in detail with focus given to the resynchronization mechanism and techniques for cauterization. Issues found during readout commissioning, and scaling resource utilization, along with the their solutions, will be illustrated. The latest results of the firmware data-processing chain can be presented as well as the verification procedures employed in simulation. Challenges for the next generation of the detector will also be presented with ideas for a readout processing solution. The new LHCb Vertex Locator (VELO) for LHCb, comprising a new pixel detector and readout electronics, will be installed in 2021 for data taking in Run 3 at the LHC. The electronics centers around the 'VeloPix' ASIC at the front-end operating in a trigger-less readout at 40MHz. A custom serializer, called gigabit wireline transmitter (GWT), and associated custom protocol have been designed for the VeloPix. The GWT data are sent from the serializers of the VeloPix at a line rate of 5.12 Gb/s, reaching a total data rate of 2-3 Tb/s for the full VELO detector. Data are sent over 300-m optic-fiber links to the control and readout electronics cards for deserialization and processing in Intel Arria 10 FPGAs. Because of the VeloPix trigger-less design, latency variances up to 12 $\mu \text{s}$ can occur between adjacent datagrams. It is therefore essential to buffer and synchronize the data in firmware prior to onward propagation or suffer a huge CPU-processing penalty. This article will describe the architecture of the readout firmware in detail with focus given to the resynchronization mechanism and techniques for cauterization. Issues found during readout commissioning, and scaling resource utilization, along with the their solutions, will be illustrated. The latest results of the firmware data-processing chain can be presented as well as the verification procedures employed in simulation. Challenges for the next generation of the detector will also be presented with ideas for a readout processing solution.

A front-end board (FEB) configuration test board is developed for examining the configuration interfaces of various application-specific integrated circuits (ASICs) developed for the ATLAS New Small Wheel project. In this paper, we present the development of a FEB configuration test board, and some functions are developed in terms of configurations of the key chips on the FEB-VMM3 and trigger data serializer (TDS2) using giga-bit transceiver-slow control adapter (GBT-SCA). Additionally, the protocols provided by GBT-SCA and the front-end board ASICs are used to verify the whole data link. The board provides technical reference for prototyping the FEB key chip configuration and data readout, as well as the final system configuration. The FEB(front end board) configuration test board is developed aiming at meeting the requirement of testing the new generation ASIC(application-specific integrated circuit) chips and its configuration system for ATLAS NSW(New Small Wheel) upgrade, In this paper, some functions are developed in terms of the configurations of the key chips on the FEB, VMM3 and TDS2 using GBT-SCA. Additionally, a flexible communication protocol is designed, verifying the whole data link. It provides technical reference for prototype FEB key chip configuration and data readout, as well as the final system configuration.

RapidIO ( http://rapidio.org/ ) technology is a packet-switched high-performance fabric, which has been under active development since 1997. The technology is used in all 4G/LTE base stations worldwide. RapidIO is also used in embedded systems that require high reliability, low latency, and deterministic operations in a heterogeneous environment. RapidIO has several offloading features in hardware, therefore relieving the CPUs from time- and power-consuming work. Most importantly, it allows for remote direct memory access and thus zero-copy data transfer. In addition, it lends itself readily to integration with field-programmable gate arrays. In this paper, we investigate RapidIO as a technology for high-speed data acquisition (DAQ) networks, in particular the DAQ system of an LHC experiment. We present measurements using a generic multiprotocol event-building emulation tool that was developed for the LHCb experiment. Event building using a local area network, such as the one foreseen for the future LHCb DAQ, puts heavy requirements on the underlying network as all data sources from the collider will want to send to the same destinations at the same time. This may lead to an instantaneous overcommitment of the output buffers of the switches. We will present results from implementing an event building cluster based on RapidIO interconnect, focusing on the bandwidth capabilities of the technology as well as its scalability.

TCP and the socket abstraction have barely changed over the last two decades, but at the network layer there has been a giant leap from a few megabits to 100 gigabits in bandwidth. At the same time, CPU architectures have evolved into the multi-core era and applications are expected to make full use of all available resources. Applications in the data acquisition domain based on the standard socket library running in a Non-Uniform Memory Access (NUMA) architecture are unable to reach full efficiency and scalability without the software being adequately aware about the IRQ (Interrupt Request), CPU and memory affinities. During the first long shutdown of LHC, the CMS DAQ system is going to be upgraded for operation from 2015 onwards and a new software component has been designed and developed in the CMS online framework for transferring data with sockets. This software attempts to wrap the low-level socket library to ease higher-level programming with an API based on an asynchronous event driven model similar to the DAT uDAPL API. It is an event-based application with NUMA optimizations, that allows for a high throughput of data across a large distributed system. This paper describes the architecture, the technologies involved and the performance measurements of the software in the context of the CMS distributed event building.

Numerical analyses of machine induced background at the LHC are needed to evaluate the complete running environment of an experiment. In order to have a comprehensive view of the machine background in an experiment all of its sources, ranging from collimators’ cleaning inefficiency to distant and local beam-gas interactions need to be estimated; particles showering from the losses are then to be transported all the way to the experimental setup and the response of the detector evaluated. In this paper we describe a novel methodology implemented for the LHCb experiment to achieve this. Each step in the chain is simulated with software specific to the task and provides input to the subsequent step through a well-defined and clear interface. Further, we will discuss in detail the various steps in the chain together with the advantages such a modular method allows in evaluating operational conditions where scaling of the initial sources can be applied. We will also give some examples of the results obtained.

LHCb aims to use its O(20000) CPU cores in the high level trigger (HLT) and its 120 TB Online storage system for data reprocessing during LHC shutdown periods. These periods can last a few days for technical maintenance or only a few hours during beam interfill gaps. These jobs run on files which are staged in from tape storage to the local storage buffer. The result are again one or more files. Efficient file writing and reading is essential for the performance of the system. Rather than using a traditional shared file-system such as NFS or CIFS we have implemented a custom, light-weight, non-Posix network file-system for the handling of these files. Streaming this file-system for the data-access allows to obtain high performance, while at the same time keep the resource consumption low and add nice features not found in NFS such as high-availability, transparent fail-over of the read and write service. The writing part of this streaming service is in successful use for the Online, real-time writing of the data during normal data acquisition operation. The network file system relies on a commercial file system which manages the data on persistent disk storage. The implementation together with performance figures are presented.

The LHCb experiment uses a single, high performance storage system to serve all kinds of storage needs: home directories, shared areas, raw data storage and buffer storage for event reconstruction. All these applications are concurrent and require careful optimisation. In particular for accessing the raw data in read and write mode, a custom light weight non- POSIX compliant file system has been developed. File serving is achieved by running several redundant file servers in an active active configuration with high availability capabilities and good performance. In this paper we describe the design and current architecture of this storage system. We discuss implementation issues and problems we had to overcome during the hitherto 18 months run-in period. Based on our experience we will also discuss the relative advantages and disadvantages of such a system over a system composed of several smaller storage systems. We also present performance measurements.

This paper focuses on the management and control of raw event data files containing electronic response to particle collisions in the LHCb detector at the LHC storage ring at CERN as well as their associated metadata. The typical foreseen file life cycle is presented, starting from the file creation to its transfer to the CERN tape storage system and the offline processing GRID system. To answer the management needs a dedicated database is implemented (Online Run Database) together with the necessary management software. The solution is integrated with the rest of the data acquisition software system and with the experiment control system. The software and hardware configuration is explained together with the testing methods, current status and future plans.

International audience; The complexity of today's experiments in High Energy Physics results in a large amount of readout channels which can count up to a million and above. The experiments in general consist of various subsystems which themselves comprise a large amount of detectors requiring sophisticated DAQ and readout elec- tronics. We report here on the structured software layers to control such a data acquisition system for the case of LHCb which is one of the four experiments for LHC. Additional focus is given on the protocols in use as well as the required hardware. An abstraction layer was implemented to allow access on the different and dis- tinct hardware types in a coherent and generic manner. The hierar- chical structure which allows propagating commands down to the subsystems is explained. Via finite state machines an expert system with auto-recovery abilities can be modeled.

This paper describes the new dynamic recovery mechanisms in the ATLAS Trigger and DataAcQuisition (TDAQ) system. The purpose of the new recovery mechanism is to minimize the impact certain errors and failures have on the system. The new recovery mechanisms are capable of analyzing and recovering from a variety of errors, both software and hardware, without stopping the data-gathering operations. An expert system is incorporated to perform the analysis of the errors and to decide what measures are needed. Due to the wide array of sub-systems there is also a need to optimize the way similar errors are handled for the different sub-systems. The main focus of the paper is to consider the design and implementation of the new recovery mechanisms and how expert knowledge is gathered from the different sub-systems and implemented in the recovery procedures.

New physical processes specific for microdosimetry simulation are under development in the Geant4 Low Energy Electromagnetic package. The first set of models implemented for this purpose cover the interactions of electrons, protons and light ions in liquid water; they address a physics domain relevant to the simulation of radiation effects in biological systems, where water represents an important component. The design developed for effectively handling particle interactions down to a low energy scale and the physics models implemented in the first public release of the software are described.

ASAP is a job creation and management framework used by CMS physicists to perform data analysis using the Grid. ASAP hides many of the low-level details of the Grid so that physicists can distribute their work across the Grid to take advantage of the available CPU power and access the stored data. There is also a server side component to, which users can delegate responsibility for their jobs. The server monitors the status of jobs and resubmits in the case of Grid or application failure The Grid provides the computing and storage resources necessary for the analysis and storage of large volumes of High Energy Physics data. ASAP provides a layer over the standard grid job management tools, which means that users do not have to be concerned with the details of job creation, submission and monitoring. The presence of the server side component increases the chances of the users’ jobs being successfully completed. The main concerns are the performance and reliability of jobs. In addition documentation is often missing or inadequate.

In 2004, a full slice of the ATLAS detector was tested for 6 months in the H8 experimental area of the CERN SPS, in the so-called Combined Test Beam, with beams of muons, pions, electrons and photons in the range 1 to 350 GeV. Approximately 90 million events were collected, corresponding to a data volume of 4.5 TB. The importance of this exercise was two-fold: for the first time the whole ATLAS software suite was used on fully combined real data. Besides, a novel production infrastructure was employed for the reconstruction of the real data as well as for a massive production of simulated events. This paper is a report on distributed Combined Test Beam Monte Carlo production with and without grid tools. In 2004, Monte Carlo production was started on the CERN LSF batch system and later it was, for the first time, performed on the LCG, with the simulation of about 4 Million events. In 2005, a more light-weight and user friendly system was tested on NorduGrid for the quick production of 210'000 photon events and 680'000 pion, muon and electron events. New Monte Carlo productions are already planed for the year 2006, using the new ATLAS production system

Developing systems for the ATLAS (A Toroidal LHC ApparatuS) detector at LHC poses a new set of challenges like the storage of enormous amounts of data with different structures. This data is not only traditional Conditions Data which is slowly evolving data but also data from the online systems which cannot be stored with the event data but must be accessible from each event, like run information. Following the requirements from such systems we have developed a system which provides the facility to store structural data in a time order and with the possibility to classify hierarchically. We will also present a case study of use in the online systems of the ATLAS experiment where this system was used to store online data. Furthermore, we will show the different interfaces developed for this system and its use in different areas of application

A single-ended to differential linear current driver has been designed and implemented in a 0.25 mum CMOS technology. This full-custom cell is intended for analogue data transmission in large-scale High-Energy Physics (HEP) experiments in the future CERN's Large Hadron Collider (LHC). Intrinsic technology radiation tolerance and specific design methodologies qualify this device to operate over 10 years in the LHC high radiation environment. High linearity and low power-overhead are achieved through a local feedback input stage and a current push-pull output stage. Linearity error is less than 0.1% in the specified operating range and 1% in close to full-swing operation. Static, dynamic and noise performance have been proven to be stable after a total ionising dose of 20 Mrad (SiO2) and after accelerated annealing

To test the feasibility of using remote farms to perform real-time event selection in the trigger/data acquisition system for the ATLAS experiment at CERN, a proof of concept was set up during 2004. The behaviour of the request-response protocol to move application data has been measured for remote farms connected with different wide area networks including a dedicated lightpath, a virtual private network, and the standard production network. The dynamics and effect of using TCP/IP as the transport protocol has also been investigated for this realtime application. These data are compared with conventional bulk data transfers and used to validate the observed performance of the online farms and estimate the effect of this traffic pattern on the wide area network

This paper describes the activities performed by a working group formed with the objective to support and develop mathematical software for the high energy physics experiments of the large hadron collider (LHC). The goal is to share common mathematical libraries between the experiments and the LHC computing grid application area projects. We present the various library components and their design, with particular emphasis on fitting and minimization, describing the new object oriented implementation in C++ of the MINUIT minimization package

Remote Procedure Call is a flexible technique for building distributed systems. Experience with systems including a variety of processor types and communication media has demonstrated some of its advantages, and pointed out special facilities which are requited and simplifying restrictions which are acceptable.

Despite advances in the programmable logic capabilities of modern trigger systems, a significant bottleneck remains in the amount of data to be transported from the detector to off-detector logic where trigger decisions are made. We demonstrate that a neural network autoencoder model can be implemented in a radiation tolerant ASIC to perform lossy data compression alleviating the data transmission problem while preserving critical information of the detector energy profile. For our application, we consider the high-granularity calorimeter from the CMS experiment at the CERN Large Hadron Collider. The advantage of the machine learning approach is in the flexibility and configurability of the algorithm. By changing the neural network weights, a unique data compression algorithm can be deployed for each sensor in different detector regions, and changing detector or collider conditions. To meet area, performance, and power constraints, we perform a quantization-aware training to create an optimized neural network hardware implementation. The design is achieved through the use of high-level synthesis tools and the hls4ml framework, and was processed through synthesis and physical layout flows based on a LP CMOS 65 nm technology node. The flow anticipates 200 Mrad of ionizing radiation to select gates, and reports a total area of 3.6 mm^2 and consumes 95 mW of power. The simulated energy consumption per inference is 2.4 nJ. This is the first radiation tolerant on-detector ASIC implementation of a neural network that has been designed for particle physics applications., 9 pages, 8 figures, 3 tables

The authors describe an optical fiber link that is used in conjunction with the CERN host interface family to connect high-performance VAXes to VMEbus or FASTBUS over distances up to 1000 m. The modular construction allows other permutations, including VAX/VAX, FASTBUS/FASTBUS, VMEbus/VMEbus, or FASTBUS/VMEbus connections, over similar distances. The link comprises two identical optical data interconnects (ODI) connected by two unidirectional multimode fibers. The ODI at each end of the link can be connected to any device conforming to the DRB32 user interface specification. The authors discuss the operation of an ODI, the testing of the optical link, and present and future ODI implementations. >