Search

Software to analyse very large sets of experimental data often relies on a pipeline of irregular computational tasks with decisions to remove irrelevant data from further processing. A user-centred framework was designed and deployed, HEP-Frame, which aids domain experts to develop applications for scientific data analyses and to monitor and control their efficient execution. The key feature of HEP-Frame is the performance portability of the code across different heterogeneous platforms, due to a novel adaptive multi-layer scheduler, seamlessly integrated into the tool, an approach not available in competing frameworks. The multi-layer scheduler transparently allocates parallel data/tasks across the available heterogeneous resources, dynamically balances threads among data input and computational tasks, adaptively reorders in run-time the parallel execution of the pipeline stages for each data stream, respecting data dependencies, and efficiently manages the execution of library functions in accelerators. Each layer implements a specific scheduling strategy: one balances the execution of the computational stages of the pipeline, distributing the execution of the stages of the same or different dataset elements among the available computing threads; another controls the order of the pipeline stages execution, so that most data is filtered out earlier and later stages execute the computationally heavy tasks; yet another adaptively balances the automatically created threads among data input and the computational tasks, taking into account the requirements of each application. Simulated data analyses from sensors in the ATLAS Experiment at CERN evaluated the scheduler efficiency, on dual multicore Xeon servers with and without accelerators, and on servers with the many-core Intel KNL. Experimental results show significant improved performance of these data analyses due to HEP-Frame features and the codes scaled well on multiple servers. Results also show the improved HEP-Frame scheduler performance over the key competitor, the HEFT list scheduler. The best overall performance improvement over a real fine tuned sequential data analysis was impressive in both homogeneous and heterogeneous multicore servers and in many-core servers: 81x faster in the homogeneous 24+24 core Skylake server, 86x faster in the heterogeneous 12+12 core Ivy Bridge server with the Kepler GPU, and 252x faster in the 64-core KNL server. Program Title: HEP-Frame CPC Library link to program files: https://doi.org/10.17632/m2jwxshtfz.1 Licencing provisions: GPLv3 Programming language: C++. Supplementary material: The current HEP-Frame public release available at https://bitbucket.org/ampereira/hep-frame/wiki/Home. Nature of problem: Scientific data analysis applications are often developed to process large amounts of data obtained through experimental measurements or Monte Carlo simulations, aiming to identify patterns in the data or to test and/or validate theories. These large inputs are usually processed by a pipeline of computational tasks that may filter out irrelevant data (a task and its filter is addressed as a proposition in this communication), preventing it from being processed by subsequent tasks in the pipeline. This data filtering, coupled with the fact that propositions may have different computational intensities, contribute to the irregularity of the pipeline execution. This can lead to scientific data analyses I/O-, memory-, or compute-bound performance limitations, depending on the implemented algorithms and input data. To allow scientists to process more data with more accurate results their code and data structures should be optimized for the computing resources they can access. Since the main goal of most scientists is to obtain results relevant to their scientific fields, often within strict deadlines, optimizing the performance of their applications is very time consuming and is usually overlooked. Scientists require a software framework to aid the design and development of efficient applications and to control their parallel execution on distinct computing platforms. Solution method: This work proposes HEP-Frame, a framework to aid the development and efficient execution of pipelined scientific analysis applications on homogeneous and heterogeneous servers. HEP-Frame is a user-centred framework to aid scientists to develop applications to analyse data from a large number of dataset elements, with a flexible pipeline of propositions. It not only stresses the interface to domain experts so that code is more robust and is developed faster, but it also aims high-performance portability across different types of parallel computing platforms and desirable sustainability features. This framework aims to provide efficient parallel code execution without requiring user expertise in parallel computing. Frameworks to aid the design and deployment of scientific code usually fall into two categories: (i) resource-centred, closer to the computing platforms, where execution efficiency and performance portability are the main goals, but forces developers to adapt their code to strict framework constraints; (ii) user-centred, which stresses the interface to domain experts to improve their code development speed and robustness, aiming to provide desirable sustainability features but disregarding the execution performance. There are also a set of frameworks that merge these two categories (Liu et al., 2015 [1] ; Deelman et al., 2015 [2]) for scientific computing. While they do not have steep learning curves, concessions have to be made to their ease of use to allow for their broader scope of targeted applications. HEP-Frame attempts to merge this gap, placing itself between a fully user- or resource-centred framework, so that users develop code quickly and do not have to worry about the computational efficiency of the code It handles (i) by ensuring efficient execution of applications according to their computational requirements and the available resources on the server through a multi-layer scheduler, while (ii) is addressed by automatically generating code skeletons and transparently managing the data structure and automating repetitive tasks. Additional comments: An early stage proof-of-concept was published in a conference proceedings (Pereira et al., 2015). However, the HEP-Frame version presented in this communication only shares a very small portion of the code related to the skeleton generation (less than 5% of the overall code), while the rest of the user interface, multi-layer scheduler, and parallelization strategies were completely redesigned and re-implemented. • A multi-layer scheduler to efficiently execute irregular pipelined data analyses. • Run-time balance #threads for simultaneous data read/setup and pipeline processing. • Scheduling data analyses on single or multiple homogeneous/heterogeneous servers. • Adaptive run-time reordering of parallel pipeline stages on multiple dataset values. • Impressive speedups (up to 252x) for compute/memory-bound real world applications. [ABSTRACT FROM AUTHOR]

Nonequispaced discrete Fourier transformation (NDFT) is widely applied in all aspects of computational science and engineering. The computational efficiency and accuracy of NDFT has always been a critical issue in hindering its comprehensive applications both in intensive and in extensive aspects of scientific computing. In our previous work Yang et al. (2018), a CUNFFT method was proposed and it shown outstanding performance in handling NDFT at intermediate scale based on CUDA (Compute Unified Device Architecture) technology. In the current work, we further improved the computational efficiency of the CUNTTF method using an efficient MPI-CUDA hybrid parallelization (HP) scheme of NFFT to achieve a cutting-edge treatment of NDFT at super extended scale. Within this HP-NFFT method, the spatial domain of NDFT is decomposed into several parts according to the accumulative feature of NDFT and the detailed number of CPU and GPU nodes. These decomposed NDFT subcells are independently calculated on different CPU nodes using a MPI process-level parallelization mode, and on different GPU nodes using a CUDA thread-level parallelization mode and CUNFFT algorithm. A massive benchmarking of the HP-NFFT method indicates that this method exhibits a dramatic improvement in computational efficiency for handling NDFT at super extended scale without loss of computational precision. Furthermore, the HP-NFFT method is validated via the calculation of Madelung constant of fluorite crystal structure, and thereafter verified that this method is robust for the calculation of electrostatic interactions between charged ions in molecular dynamics simulation systems. Program title: HP-NFFT CPC Library link to program files: http://dx.doi.org/10.17632/ys2y92jkwy.1 Licensing provisions: GNU General Public License 3 Programming language: MPI, C, and CUDA C Supplementary material: The program is designed for effective computation of large-scale nonequispaced discrete Fourier transformation (NDFT), which runs on particular computers equipped with NVIDIA GPUs. It has been tested on (a) one single computer node with Intel(R) Core(TM) i7-3770 @ 3.40 GHz (CPU) and GTX 980 Ti (GPU), and (b) MPI parallel computer nodes with the same configurations. Nature of problem: For NDFT, the computation is extremely time-consuming in many domains of computational physics due to the failure to utilize FFT directly, which often affects the computational efficiency of whole system seriously. Although the parallel method CUNFFT based on GPU has achieved a qualitative leap compared with previous methods in NDFT computation, the computation capability is limited to the throughput capacity of GPU when the size of NDFT system is big enough. Solution method: We constructed a hybrid parallel architecture, in which CPU and GPU are combined to accelerate the NDFT computation effectively. Firstly, the NDFT system is divided into several subcells via domain-decomposition method. Then MPI (Message Passing Interface) is used to implement the CPU-parallel computation with each computer node corresponding to a particular subcell, and furthermore each subcell in one computer node will be executed in GPU in parallel efficiently. In this hybrid parallel method, the most critical technical problem is how to parallelize a CUNFFT in the parallel strategy, which is conquered effectively by deep-seated research of basic principles and some algorithm skills. Restrictions: The HP-NFFT is mainly oriented to, and has shown significant computational efficiency to deal with large-scale NDFT computations. However, for a small NDFT system containing less than 10^6 particles, the mode of multiple computer nodes has no apparent efficiency advantage or even lower efficiency compared with the mode of a single computer node because of the serious network delay. References: [1] S.-C. Yang, H.-J. Qian, Z.-Y. Lu, Appl. Comput. Harmon. Anal. 44 , 273 (2018). [2] S.-C. Yang, Y.-L. Wang, G.-S. Jiao, H.-J. Qian, Z.-Y. Lu, J. Comput. Chem. 37 , 378 (2016). [3] S.-C. Yang, Z.-Y. Lu, H.-J. Qian, Y.-L. Wang, J.-P. Han, Comput. Phys. Commun. 220 , 376 (2017). [ABSTRACT FROM AUTHOR]

To celebrate the leading role Computer Physics Communications (CPC) has played in publishing open-source software in computational physics for over 50 years the editors are delighted to announce this Virtual Special Issue. Since 2018, coinciding with the 50th anniversary of the start of the CPC venture, thirty-two invited articles have been published. Each has been peer reviewed and each bears the header 'CPC 50th anniversary article'. The special issue is in keeping with CPC's ethos: it is focused on computational physics software and is accompanied by twenty-five software systems. The introduction to the collection also includes a personal reflection on Phil Burke, CPC's founder, by Alan Hibbert, a lifelong colleague, who joined Queen's University with Phil in the autumn of 1967. The distinctive feature of CPC is its Program Library which houses and distributes over 3500 open-source programs in computational physics. The introduction concludes with a description of key events in the history of the Program Library, its association with Queen's University Belfast and its transfer to Elsevier's Mendeley Data repository. [ABSTRACT FROM AUTHOR]