Search

In order to enable Exascale computing, next generation interconnection networks must scale to hundreds of thousands of nodes, and must provide features to also allow the HPC, HPDA, and AI applications to reach Exascale, while benefiting from new hardware and software trends. RED-SEA will pave the way to the next generation of European Exascale interconnects, including the next generation of BXI, as follows: (i) specify the new architecture using hardware-software co-design and a set of applications representative of the new terrain of converging HPC, HPDA, and AI; (ii) test, evaluate, and/or implement the new architectural features at multiple levels, according to the nature of each of them, ranging from mathematical analysis and modeling, to simulation, or to emulation or implementation on FPGA testbeds; (iii) enable seamless communication within and between resource clusters, and therefore development of a high-performance low latency gateway, bridging seamlessly with Ethernet; (iv) add efficient network resource management, thus improving congestion resiliency, virtualization, adaptive routing, collective operations; (v) open the interconnect to new kinds of applications and hardware, with enhancements for end-to-end network services - from programming models to reliability, security, low- latency, and new processors; (vi) leverage open standards and compatible APIs to develop innovative reusable libraries and Fabrics management solutions. ISSN:1089-6503

The ExaNeSt project started on December 2015 and is funded by EU H2020 research framework (call H2020-FETHPC-2014, n. 671553) to study the adoption of low-cost, Linux-based power-efficient 64-bit ARM processors clusters for Exascale-class systems. The ExaNeSt consortium pools partners with industrial and academic research expertise in storage, interconnects and applications that share a vision of an Euro-pean Exascale-class supercomputer. Their goal is designing and implementing a physical rack prototype together with its cooling system, the storage non-volatile memory (NVM) architecture and a low-latency interconnect able to test different options for interconnection and storage. Furthermore, the consortium is to provide real HPC applications to validate the system. Herein we provide a status report of the project initial developments.

A commercial Graphics Processing Unit (GPU) is used to build a fast Level 0 (L0) trigger system for the NA62 experiment at CERN. In particular, the parallel computing power of the GPU is exploited to perform real-time fitting in the Ring Imaging CHerenkov (RICH) detector for the L0 trigger of the NA62 experiment. Direct GPU communication using a FPGA-based board has been used to reduce the data transmission latency. The first result of multi-ring Cherenkov reconstrunction obtained during the NA62 physics run will be presented.

QUonG is a parallel computing platform developed at INFN and equipped with commodity multi-core CPUs coupled with last generation NVIDIA GPUs. Computing nodes communicate through a point-to-point, high performance, low latency 3D torus network implemented by the APEnet+ FPGA-based interconnect. Scaling of this cluster towards peta-and possibly exascale is a prominent investigation point and in this context fault tolerance issues are structural. Typical fault tolerance solutions for HPC systems (e.g. checkpoint/restart) need to be triggered to be applied in an automated and transparent way, or at least knowledge about occurring faults needs propagating in order to prompt a readjustment: an effective tool to detect faults and make the system aware of them is required. Thus, as a first step towards a fault tolerant QUonG we designed the Local Fault Monitor (LO|FA|MO), an HW/SW solution aimed at providing systemic fault awareness. LO|FA|MO allows the detection of node faults thanks to a mutual watchdog mechanism between the host and the APEnet+ NIC, moreover, diagnostic messages can be delivered to neighbour nodes through both the 3D network and a secondary connection for service communication. The double path ensures that no fault remains unknown at the global level, guaranteeing systemic fault awareness with no single point of failure. In this paper we describe our LO|FA|MO implementation, reporting preliminary measures that show scalability and its next to nil impact on system performance.

We describe the hands-on experience in developing a network-centric IP core supporting the RDMA protocol which is the engine of an FPGA-based PCIe NIC targeted for GPU-accelerated HPC clusters with a 3D-toroidal network topology. We report on different development areas related to our IP: the optimizations required to evolve the NIC to the current performance level (highlights of this work include the development of a RDMA engine with a dedicated translation-lookaside-buffer and a first-of-its-kind IP module that exploits the peer-to-peer protocol of NVIDIA GPUs); the addition of a component called LO|FA|MO IP that provides systemic fault-awareness to the network; the modifications to the core IP to turn it into low-latency interface called NaNet between a read-out board and a GPU farm in the data acquisition system of the low level trigger of a particle-physics experiment. Taking into account the forecast evolution of the FPGA platform (28 nm, PCIe Gen3, etc.), we conclude with future directions we envision for our IP.

We developed a custom FPGA-based Network Interface Controller named APEnet+ aimed at GPU accelerated clusters for High Performance Computing. The card exploits peer-to-peer capabilities (GPU-Direct RDMA) for latest NVIDIA GPGPU devices and the RDMA paradigm to perform fast direct communication between computing nodes, offloading the host CPU from network tasks execution. In this work we focus on the implementation of a Virtual to Physical address translation mechanism, using the FPGA embedded soft-processor. Address management is the most demanding task - we estimated up to 70% of the μC load - for the NIC receiving side, resulting being the main culprit for data bottleneck. To improve the performance of this task and hence improve data transfer over the network, we added a specialized hardware logic block acting as a Translation Lookaside Buffer. This block makes use of a peculiar Content Address Memory implementation designed for scalability and speed. We present detailed measurements to demonstrate the benefits coming from the introduction of such custom logic: a substantial address translation latency reduction (from a measured value of 1.9 μs to 124 ns) and a performance enhancement of both host-bound and GPU-bound data transfers (up to ~ 60% of bandwidth increase) in given message size ranges.

One of the most important issues that particle physics experiments at hadron colliders have to solve is realtime selection of the most interesting events. Typical collision frequencies do not allow all events to be written to tape for offline analysis, and in most cases, only a small fraction can be saved. The most commonly used strategy is based on two or three selection levels, with the low level ones usually exploiting dedicated hardware to decide within a few to ten microseconds if the event should be kept or not. This strict time requirement has made the usage of commercial devices inadequate, but recent improvements to Graphics Processing Units (GPUs) have substantially changed the conditions. Thanks to their highly parallel, multi-threaded, multicore architecture with remarkable computational power and high memory bandwidth, these commercial devices may be used in scientific applications, among which the event selection system (trigger) in particular may benefit, even at low levels. This paper describes the results of an R&D project to study the performance of GPU technology for low latency applications, such as HEP fast tracking trigger algorithms. On two different setups, we measure the latency to transfer data to/from the GPU, exploring the timing of different I/O technologies on different GPU models. We then describe the implementation and the performance of a track fitting algorithm which mimics the CDF Silicon Vertex Tracker. These studies provide performance benchmarks to investigate the potential and limitations of GPUs for future real-time applications in HEP experiments.

APEnet+ is our custom developed PCIe gen2 board based on an Altera Stratix IV FPGA. We demonstrate reliable usage of Altera's embedded transceivers coupled with QSFP+ (Quad Small Form Pluggable) technology. QSFP+ standard defines a hot-pluggable transceiver available in copper or optical cable assemblies for an aggregated bandwidth of up to 40 Gbps. We use embedded transceivers in a 4 lane configuration, each one capable of 8.5 Gbps, for an aggregate bandwidth of 34 Gpbs per link. On Stratix IV 290 we can place up to 6 bidirectional links, together with a PCIe gen2 ×8 hard IP. We describe design and implementation of this data transmission system.

QUonG is an INFN (Istituto Nazionale di Fisica Nucleare) initiative targeted to develop a high performance computing system dedicated to Lattice QCD computations. QUonG is a massively parallel computing platform that lever-ages on commodity multi-core processors coupled with last generation GPUs. Its network mesh exploits the characteristics of LQCD algorithm for the design of a point-to-point, high performance, low latency 3-d torus network to interconnect the computing nodes. The network is built upon the APE net+ project: it consists of an FPGA-based PCI Express board exposing six full bidirectional off-board links running at 34 Gbps each, and implementing RDMA protocol and an experimental direct network-to-GPU interface, enabling significant access latency reduction for inter-node data transfers. The final shape of a complete QUonG deployment is an assembly of standard 42U racks, each one capable of 60 TFlops/rack of peak performance, at a cost of 5 Ke/TFlops and for an estimated power consumption of 25 KW/rack. A first QUonG system prototype is expected to be delivered at the end of the year 2011.

We present test results and characterization of a data transmission system based on a last generation FPGA and a commercial QSFP+ module. QSFP+ standard defines a hot-pluggable transceiver available in copper or optical cable assemblies for an aggregated bandwidth of up to 40 Gbps. We implemented a complete testbench based on a commercial development card mounting an Altera Stratix IV FPGA with 24 serial transceivers at 8.5 Gbps, together with a custom mezzanine hosting three QSFP+ modules. We present test results and signal integrity measurements up to an aggregated bandwidth of 12 Gbps.