Search

Motivation Sequence simulation plays a vital role in phylogenetics with many applications, such as evaluating phylogenetic methods, testing hypotheses, and generating training data for machine-learning applications. We recently introduced a new simulator for multiple sequence alignments called AliSim, which outperformed existing tools. However, with the increasing demands of simulating large data sets, AliSim is still slow due to its sequential implementation; for example, to simulate millions of sequence alignments, AliSim took several days or weeks. Parallelization has been used for many phylogenetic inference methods but not yet for sequence simulation. Results This paper introduces AliSim-HPC, which, for the first time, employs high-performance computing for phylogenetic simulations. AliSim-HPC parallelizes the simulation process at both multi-core and multi-CPU levels using the OpenMP and message passing interface (MPI) libraries, respectively. AliSim-HPC is highly efficient and scalable, which reduces the runtime to simulate 100 large gap-free alignments (30 000 sequences of one million sites) from over one day to 11 min using 256 CPU cores from a cluster with six computing nodes, a 153-fold speedup. While the OpenMP version can only simulate gap-free alignments, the MPI version supports insertion–deletion models like the sequential AliSim. Availability and implementation AliSim-HPC is open-source and available as part of the new IQ-TREE version v2.2.3 at https://github.com/iqtree/iqtree2/releases with a user manual at http://www.iqtree.org/doc/AliSim. [ABSTRACT FROM AUTHOR]

Since simulation became the third pillar of scientific research, several forms of computers have become available to drive computer aided simulations, and nowadays, clusters are the most popular type of computers supporting these tasks. For instance, cluster settings, such as the so-called supercomputers, cluster of workstations (COW), cluster of desktops (COD), and cluster of virtual machines (COV) have been considered in literature to embrace a variety of scientific applications. However, those scientific applications categorized as high-performance computing (HPC) are conceptually restricted to be addressed only by supercomputers. In this aspect, we introduce the notions of cluster overhead and cluster coupling to assess the capacity of non-HPC systems to handle HPC applications. We also compare the cluster overhead with an existing measure of overhead in computing systems, the total parallel overhead, to explain the correctness of our methodology. The evaluation of capacity considers the seven dwarfs of scientific computing, which are well-known, scientific computing building blocks considered in the development of HPC applications. The evaluation of these building blocks provides insights regarding the strengths and weaknesses of non-HPC systems to deal with future HPC applications developed with one or a combination of these algorithmic building blocks. [ABSTRACT FROM AUTHOR]

Background: The Basic Local Alignment Search Tool (BLAST) is a suite of commonly used algorithms for identifying matches between biological sequences. The user supplies a database file and query file of sequences for BLAST to find identical sequences between the two. The typical millions of database and query sequences make BLAST computationally challenging but also well suited for parallelization on high-performance computing clusters. The efficacy of parallelization depends on the data partitioning, where the optimal data partitioning relies on an accurate performance model. In previous studies, a BLAST job was sped up by 27 times by partitioning the database and query among thousands of processor nodes. However, the optimality of the partitioning method was not studied. Unlike BLAST performance models proposed in the literature that usually have problem size and hardware configuration as the only variables, the execution time of a BLAST job is a function of database size, query size, and hardware capability. In this work, the nucleotide BLAST application BLASTN was profiled using three methods: shell-level profiling with the Unix "time" command, code-level profiling with the built-in "profiler" module, and system-level profiling with the Unix "gprof" program. The runtimes were measured for six node types, using six different database files and 15 query files, on a heterogeneous HPC cluster with 500+ nodes. The empirical measurement data were fitted with quadratic functions to develop performance models that were used to guide the data parallelization for BLASTN jobs. Results: Profiling results showed that BLASTN contains more than 34,500 different functions, but a single function, RunMTBySplitDB, takes 99.12% of the total runtime. Among its 53 child functions, five core functions were identified to make up 92.12% of the overall BLASTN runtime. Based on the performance models, static load balancing algorithms can be applied to the BLASTN input data to minimize the runtime of the longest job on an HPC cluster. Four test cases being run on homogeneous and heterogeneous clusters were tested. Experiment results showed that the runtime can be reduced by 81% on a homogeneous cluster and by 20% on a heterogeneous cluster by re-distributing the workload. Discussion: Optimal data partitioning can improve BLASTN's overall runtime 5.4-fold in comparison with dividing the database and query into the same number of fragments. The proposed methodology can be used in the other applications in the BLAST+ suite or any other application as long as source code is available. [ABSTRACT FROM AUTHOR]

Multiphase flow properties of fractures are important in engineering applications such as hydraulic fracturing, evaluating the sealing capacity of caprocks, and the productivity of hydrocarbon-bearing tight rocks. Due to the computational requirements of high fidelity simulations, investigations of flow and transport through fractures typically rely on simplified assumptions applied to large fracture networks. These simplifications ignore the effect of pore-scale capillary phenomena and 3D realistic fracture morphology (for instance, tortuosity, contact points, and crevasses) that lead to macro-scale effective transport properties. The effect of these properties can be studied through lattice Boltzmann simulations, but they require high performance computing clusters and are generally limited in their domain size. In this work, we develop a technique to represent 3D fracture geometries and fluid distributions in 2D without losing any information. Using this innovative approach, we present a specialized machine learning model which only requires a few simulations for training but still accurately predicts fluid flow through 3D fractures. We demonstrate our technique using simulations of a water filled fracture being displaced by supercritical CO 2 . By generating highly efficient simulations of micro-scale multiphase flow in fractures, we hope to investigate a wide range of fracture types and generalize our method to be incorporated into larger discrete fracture network simulations. [ABSTRACT FROM AUTHOR]

Background: Cell-based models are becoming increasingly popular for applications in developmental biology. However, the impact of numerical choices on the accuracy and efficiency of the simulation of these models is rarely meticulously tested. Without concrete studies to differentiate between solid model conclusions and numerical artifacts, modelers are at risk of being misled by their experiments' results. Most cell-based modeling frameworks offer a feature-rich environment, providing a wide range of biological components, but are less suitable for numerical studies. There is thus a need for software specifically targeted at this use case. Results: We present CBMOS, a Python framework for the simulation of the center-based or cell-centered model. Contrary to other implementations, CBMOS' focus is on facilitating numerical study of center-based models by providing access to multiple ordinary differential equation solvers and force functions through a flexible, user-friendly interface and by enabling rapid testing through graphics processing unit (GPU) acceleration. We show-case its potential by illustrating two common workflows: (1) comparison of the numerical properties of two solvers within a Jupyter notebook and (2) measuring average wall times of both solvers on a high performance computing cluster. More specifically, we confirm that although for moderate accuracy levels the backward Euler method allows for larger time step sizes than the commonly used forward Euler method, its additional computational cost due to being an implicit method prohibits its use for practical test cases. Conclusions: CBMOS is a flexible, easy-to-use Python implementation of the center-based model, exposing both basic model assumptions and numerical components to the user. It is available on GitHub and PyPI under an MIT license. CBMOS allows for fast prototyping on a central processing unit for small systems through the use of NumPy. Using CuPy on a GPU, cell populations of up to 10,000 cells can be simulated within a few seconds. As such, it will substantially lower the time investment for any modeler to check the crucial assumption that model conclusions are independent of numerical issues. [ABSTRACT FROM AUTHOR]

Process‐based models are becoming increasingly used tools for understanding how species are likely to respond to environmental changes and to potential management options. RangeShifter is one such modelling platform, which has been used to address a range of questions including identifying effective reintroduction strategies, understanding patterns of range expansion and assessing population viability of species across complex landscapes. Here we introduce a new version, RangeShifter 2.0, which incorporates important new functionality. It is now possible to simulate dynamics over user‐specified, temporally changing landscapes. Additionally, we integrated a new genetic module, notably introducing an explicit genetic modelling architecture, which allows for simulation of neutral and adaptive genetic processes. Furthermore, emigration, transfer and settlement traits can now all evolve, allowing for sophisticated simulation of the evolution of dispersal. We illustrate the potential application of RangeShifter 2.0's new functionality by two examples. The first illustrates the range expansion of a virtual species across a dynamically changing UK landscape. The second demonstrates how the software can be used to explore the concept of evolving connectivity in response to land‐use modification, by examining how movement rules come under selection over landscapes of different structure and composition. RangeShifter 2.0 is built using object‐oriented C++ providing computationally efficient simulation of complex individual‐based, eco‐evolutionary models. The code has been redeveloped to enable use across operating systems, including on high performance computing clusters, and the Windows graphical user interface has been enhanced. RangeShifter 2.0 will facilitate the development of in‐silico assessments of how species will respond to environmental changes and to potential management options for conserving or controlling them. By making the code available open source, we hope to inspire further collaborations and extensions by the ecological community. [ABSTRACT FROM AUTHOR]

• Showing the feasibility of constructing small, portable HPC systems for education. • Illustrating the benefits of this system for live interaction and application steering. • Providing enough material and information for others to reproduce and extend our work. Teaching parallel programming and HPC is a difficult task. There is a large number of sophisticated hardware and software components, each complex on their own and often showing non-intuitive interaction when used in combination. We consider education in HPC among the more difficult topics in computer science due to the fact that larger distributed memory systems are ubiquitous yet inaccessible and intangible to students. In this work, we present the Cluster Coffer, a miniature cluster computer based on 16 ARM compute boards that we believe is suitable for reducing the entry barrier to HPC in teaching and public outreach. We discuss our design goals for providing a portable, inexpensive system that is easy to maintain and repair. We outline the implementation path we took in terms of hardware and software, in order to provide others with the information required to reproduce and extend our work. Finally, we present two use cases for which the Cluster Coffer has been used multiple times, and will continue to be used in the upcoming years. [ABSTRACT FROM AUTHOR]

The large amount of biological data available in the current times, makes it necessary to use tools and applications based on sophisticated and efficient algorithms, developed in the area of bioinformatics. Further, access to high performance computing resources is necessary, to achieve results in reasonable time. To speed up applications and utilize available compute resources as efficient as possible, software developers make use of parallelization mechanisms, like multithreading. Many of the available tools in bioinformatics offer multithreading capabilities, but more compute power is not always helpful. In this study we investigated the behavior of well-known applications in bioinformatics, regarding their performance in the terms of scaling, different virtual environments and different datasets with our benchmarking tool suite BOOTABLE. The tool suite includes the tools BBMap, Bowtie2, BWA, Velvet, IDBA, SPAdes, Clustal Omega, MAFFT, SINA and GROMACS. In addition we added an application using the machine learning framework TensorFlow. Machine learning is not directly part of bioinformatics but applied to many biological problems, especially in the context of medical images (X-ray photographs). The mentioned tools have been analyzed in two different virtual environments, a virtual machine environment based on the OpenStack cloud software and in a Docker environment. The gained performance values were compared to a bare-metal setup and among each other. The study reveals, that the used virtual environments produce an overhead in the range of seven to twenty-five percent compared to the bare-metal environment. The scaling measurements showed, that some of the analyzed tools do not benefit from using larger amounts of computing resources, whereas others showed an almost linear scaling behavior. The findings of this study have been generalized as far as possible and should help users to find the best amount of resources for their analysis. Further, the results provide valuable information for resource providers to handle their resources as efficiently as possible and raise the user community's awareness of the efficient usage of computing resources. Author summary: The analysis of biological data increasingly makes more and more use of computational resources, or would not be possible at all without them. Besides classical high performance computing resources like computer clusters, the technology of cloud computing and its applications in biology and bioinformatics has strongly increased over the last years. With cloud computing, virtualization technologies are also increasingly used. However, computing resources are not endlessly available and should therefore be used as efficient as possible. To support the efficient development of multithreaded applications, we developed our benchmarking tool suite BOOTABLE and used it to study the scaling behavior of different bioinformatics tools, covering a wide range of application areas using different computing environments (virtualized and bare-metal). Our study showed that not every tool benefits from higher numbers of CPU cores, also linear scaling properties are not seen for all of them. With this study we want to create an awareness for the responsible usage of computing resources. It is not always better and faster to use more and more resources. Sometimes it is helpful to check, whether a tool or application is capable of using larger resources or not. [ABSTRACT FROM AUTHOR]

This work presents an efficient, flexible, and scalable strategy to implement density-based topology optimization formulation for fail-safe structural design. Such optimized designs can operate after faults modeled as loss of stiffness. However, the need for assessing the physical behavior in all the fault cases can make this problem unfeasible computationally. We combine many ingredients to exploit the different levels of parallelism of the formulation to deal successfully with this problem. We use a non-overlapping domain decomposition method to solve the failure cases using multi-core computing with distributed memory computation. These subdomains use adaptive mesh refinement (AMR) techniques to focus computing efforts on the regions of interest. A key point to evaluate the fault cases is the organization of the computing threads, especially in clusters of computers. We group the computing threads by physical computing nodes to reduce the inter-node communications, maximizing intra-node communications to mitigate bandwidth problems. Another ingredient of paramount importance is the use of computing buffers to adapt the evaluation of the fault cases to the computing resources. We test the scalability of the computational framework using a large cluster of computers. • Propose an efficient framework to address fail-safe continuum topology optimization. • Exploit the embarrassing level of parallelism using a hierarchical parallel scheme. • Adapt the computing resources to the requirements using computing buffers. • Efficient simulation of damage cases introducing adaptive mesh refinement techniques. • Strong scaling of the computing framework using more than 8000 computing cores. [ABSTRACT FROM AUTHOR]

Nowadays, increase in time complexity of applications and decrease in hardware costs are two major contributing drivers for the utilization of high‐performance architectures such as cluster computing systems. Actually, cluster computing environments, in the contemporary sophisticated data centres, provide the main infrastructure to process various data, where the biomedical one is not an exception. Indeed, optimized task scheduling is key to achieve high performance in such computing environments. The most distractive assumption about the problem of task scheduling, made by the state‐of‐the‐art approaches, is to assume the problem as a whole and try to enhance the overall performance, while the problem is actually consisted of two disparate‐in‐nature subproblems, that is, sequencing subproblem and assigning one, each of which needs some special considerations. In this paper, an efficient hybrid approach named ACO‐CLA is proposed to solve task scheduling problem in the mesh‐topology cluster computing environments. In the proposed approach, an enhanced ant colony optimization (ACO) is developed to solve the sequence subproblem, whereas a cellular learning automata (CLA) machine tackles the assigning subproblem. The utilization of background knowledge about the problem (i.e., tasks' priorities) has made the proposed approach very robust and efficient. A randomly generated data set consisting of 125 different random task graphs with various shape parameters, like the ones frequently encountered in the biomedicine, has been utilized for the evaluation of the proposed approach. The conducted comparison study clearly shows the efficiency and superiority of the proposed approach versus traditional counterparts in terms of the performance. From our first metric, that is, the NSL (normalized schedule length) point of view, the proposed ACO‐CLA is 2.48% and 5.55% better than the ETF (earliest time first), which is the second‐best approach, and the average performance of all other competing methods. On the other hand, from our second metric, that is, the speedup perspective, the proposed ACO‐CLA is 2.66% and 5.15% better than the ETF (the second‐best approach) and the average performance of all the other competitors. [ABSTRACT FROM AUTHOR]

Background Dimensionality reduction and visualization play vital roles in single-cell RNA sequencing (scRNA-seq) data analysis. While they have been extensively studied, state-of-the-art dimensionality reduction algorithms are often unable to preserve the global structures underlying data. Elastic embedding (EE), a nonlinear dimensionality reduction method, has shown promise in revealing low-dimensional intrinsic local and global data structure. However, the current implementation of the EE algorithm lacks scalability to large-scale scRNA-seq data. Results We present a distributed optimization implementation of the EE algorithm, termed distributed elastic embedding (D-EE). D-EE reveals the low-dimensional intrinsic structures of data with accuracy equal to that of elastic embedding, and it is scalable to large-scale scRNA-seq data. It leverages distributed storage and distributed computation, achieving memory efficiency and high-performance computing simultaneously. In addition, an extended version of D-EE, termed distributed optimization implementation of time-series elastic embedding (D-TSEE), enables the user to visualize large-scale time-series scRNA-seq data by incorporating experimentally temporal information. Results with large-scale scRNA-seq data indicate that D-TSEE can uncover oscillatory gene expression patterns by using experimentally temporal information. Conclusions D-EE is a distributed dimensionality reduction and visualization tool. Its distributed storage and distributed computation technique allow us to efficiently analyze large-scale single-cell data at the cost of constant time speedup. The source code for D-EE algorithm based on C and MPI tailored to a high-performance computing cluster is available at https://github.com/ShaokunAn/D-EE. [ABSTRACT FROM AUTHOR]

High-performance computing systems are the maximum expression in the field of processing for large amounts of data. However, their energy consumption is an aspect of great importance, which was not considered decades ago. Hence, software developers and hardware providers are obligated to approach new challenges to address energy consumption, and costs. Constructing a computational cluster with a large amount of systems on a chip can result in a powerful, ecologic platform, with the capacity to offer sufficient performance for different applications, as long as low costs and minimum energy consumption can be maintained. As a result, energy efficient hardware has an opportunity to impact upon the area of high-performance computing. This article presents a systematic review of the evaluations conducted on clusters of Systems on a Chip for High-Performance computing in the research setting. [ABSTRACT FROM AUTHOR]

As an alternative to traditional computing architecture, cloud computing now is rapidly growing. However, it is based on models like cluster computing in general. Now supercomputers are getting more and more powerful, helping scientists have more indepth understanding of the world. At the same time, clusters of commodity servers have been mainstream in the IT industry, powering not only large Internet services but also a growing number of data-intensive scientific applications, such as MPI based deep learning applications. In order to reduce the energy cost, more and more efforts are made to improve the energy consumption of HPC systems. Because I/O accesses account for a large portion of the execution time for data intensive applications, it is critical to design energy-aware parallel I/O functions for addressing challenges related to HPC energy efficiency. As the de facto standard for designing parallel applications in cluster environment, the Message Passing Interface has been widely used in high performance computing, therefore, getting the energy consumption information of MPI applications is critical for improving the energy efficiency of HPC systems. In this work we first present our energy measurement tool, a software framework that eases the energy collection in cluster environment. And then we present an approach which can optimise the parallel I/O operation's energy efficiency. The energy scheduling algorithm is evaluated in a cluster. We try to optimize the energy efficiency of BTIO. Before doing the optimization, we first modeled the execution time and energy consumption. In BTIO, a three-dimensional array is partitioned in a block-tridiagonal pattern and assigned across a square number of processes. Each process is responsible for many (the square root of the number of participating processors) subsets of the entire data set. All parameters of the elemental operations define a multidimensional configuration space. We first modeled the elemental operations. The configuration space of an elemental operation is the cross product of the individual values of the parameters. We sampled points which are evenly distributed in the configuration space, and tested the execution time and consumed energy with MEMT. We divided all the sampled points into training and test set. The majority (75%) of the points were used to iteratively train the execution time and energy models, and the left points were used for testing the models. The models of a collective I/O operation is the combination of these elemental models. The two figures show the modeled versus measured execution time and energy consumption of 250 BTIO operations. We can see that the models behave well. [ABSTRACT FROM AUTHOR]

This work presents an efficient approach for the generation of distributed Sparse Approximate Inverse preconditioners based on the near-field coupling information for the analysis of electromagnetic problems on large computing clusters. This scheme combines the Message Passing Interface and Open Multi-Processing paradigms in order to minimise the CPU time and memory footprint of the preconditioner, making use of specific algorithms tailored to balance the load and reduce the amount ot information shared between nodes. Some representative examples provide insight into the scalability and performance of the described approach addressing large and realistic scenarios. [ABSTRACT FROM AUTHOR]

Large-scale modeling and simulation (M&S) applications that do not require run-time inter-process communications can exhibit scaling problems when migrated to high-performance computing (HPC) clusters if traditional software parallelization techniques, such as POSIX multi-threading and the message passing interface, are used. A comprehensive approach for scaling M&S applications on HPC clusters has been developed and is called "computation segmentation." The computation segmentation is based on the built-in array job facility of job schedulers. If used correctly for appropriate applications, the array job approach provides significant benefits that are not obtainable using other methods. The parallelization illustrated in this paper becomes quite complex in its own right when applied to extremely large M&S tasks, particularly due to the need for nested loops. At the United States Food and Drug Administration, the approach has provided unsurpassed efficiency, flexibility, and scalability for work that can be performed using embarrassingly parallel algorithms. [ABSTRACT FROM AUTHOR]

The article is dedicated to optimizing the computing cluster performance and throughput of the storage with the Lustre file system to perform the reverse time migration task. Optimization is needed because increasing the number of processor cores at first helps to speed up the calculation process, but at some point the calculations start to take more time despite adding more cores with the same storage throughput. This behaviour is caused by the fact that performance gain achieved through parallel operation is offset by increasing delays in the storage that grow nonlinearly as the load increases. To take into account the particularities of this case study, the specific task complexity notion that defines the number of floating point operations per input/output byte was used. This notion and the queuing theory fundamentals were applied to produce a simple formula that links the Lustre file system storage throughput, quantity of storage nodes and the cluster performance in the optimal configuration. LINPACK and SPC-2 benchmark results were used as initial data. [ABSTRACT FROM AUTHOR]

The MOVES model was developed by the U.S. Environmental Protection Agency (U.S. EPA) to estimate emissions from on-road mobile sources and nonroad sources in the United States. Coupling high-resolution on-road vehicle activity data with appropriate MOVES emission rates further advances research efforts designed to assess the environmental impacts of transportation design and operation strategies. However, the complicated MOVES interface and slow performance makes it difficult to assess large, regional scale transportation networks and to undertake analyses of large-scale systems that are dynamic in nature. The MOVES-Matrix system develops an initial Large Matrix of MOVES outputs by running MOVES 146,853 times on the PACE high performance computing cluster to generate more than 90 billion emission rates to populate the matrix for a single area with one fuel regime and one inspection and maintenance program. A total of 117 such Large Matrices would be needed for the entire United States. The MOVES-Matrix system developed can be used to conduct the emissions modeling 200-times faster than using MOVES. The hypothetical case study shows that MOVES-Matrix is able to generate the exact same emission results as the MOVES model to ensure the validity for regulatory analysis. The resulting matrix allows users to link emission rates to big data projects and to evaluate changes in emissions for dynamic transportation systems in near-real-time. MOVES-Matrix does not currently estimate emissions from starts, hoteling or evaporative emissions, and the research team is working on MOVES-Matrix version 2 that supports incorporating off-network modeling. Implications: MOVES-Matrix should be of interest to a broad readership including those interested in vehicle emission modeling, near-road air quality modeling, transportation conformity analysis. The paper should also interest engineers who are involved in transportation regulatory and conformity analysis, state implementation plan, and who are seeking an efficient way of conducting regulatory emission modeling and air quality analysis in the United States. [ABSTRACT FROM AUTHOR]

Background: Analysing large and high-dimensional biological data sets poses significant computational difficulties for bioinformaticians due to lack of accessible tools that scale to hundreds of millions of data points. Results: We developed a novel machine learning command line tool called PyBDA for automated, distributed analysis of big biological data sets. By using Apache Spark in the backend, PyBDA scales to data sets beyond the size of current applications. It uses Snakemake in order to automatically schedule jobs to a high-performance computing cluster. We demonstrate the utility of the software by analyzing image-based RNA interference data of 150 million single cells. Conclusion: PyBDA allows automated, easy-to-use data analysis using common statistical methods and machine learning algorithms. It can be used with simple command line calls entirely making it accessible to a broad user base. PyBDA is available at https://pybda.rtfd.io. [ABSTRACT FROM AUTHOR]

The rapid growth of Internet has vast amounts of information over online. The correct information can be provided by the source only if the information is processed, analyzed and linked. The efficient store and manage model is required to access and to protect these large data. These data are structured and unstructured which is available in online in order to process such data an intense technology is required. The cloud computing satisfies the need of store and manage model. Whereas to access and protect data, many intense technologies like parallel and map reduce methods are available. But these methods face difficulties in large data processing. In this article these traditional difficulties are overcome by Hadoop data model to give a high performance computing of large data in cloud computing environment. In the experiment, the proposed method was compared with previous works. As a result, the proposed method achieved 0.51 Packet delivery ratio with 0.71s Elapsed Time/Word transfer at the Receiver throughput of 770kbps, which is much better than that of previous work. The single cluster and analytical hierarchy process (AHP) is used to compute data along with Hadoop to provide fault tolerance over failures, less processing time and communication errors. [ABSTRACT FROM AUTHOR]

This survey aims to present the state of the art in analytic communication performance models, providing sufficiently detailed descriptions of particularly noteworthy efforts. Modeling the cost of communications in computer clusters is an important and challenging problem. It provides insights into the design of the communication pattern of parallel scientific applications and mathematical kernels and sets a clear ground for optimization of their deployment in the increasingly complex high-performance computing infrastructure. The survey provides background information on how different performance models represent the underlying platform and shows the evolution of these models over time from early clusters of single-core processors to present-day multi-core and heterogeneous platforms. Prospective directions for future research in the area of analytic communication performance modeling conclude the survey. [ABSTRACT FROM AUTHOR]

Edge detection plays an important role in the field of image processing. Various edge detection techniques are obtained like Sobel, PSO Preweitt, Laplacian and Laplacian of Gaussian. These techniques consume some restrictions such as fixed edge thickness and some parameter like threshold is problematic to implement. Although many different edge-detection methods have been proposed for gray-scale, color, and multispectral images, they still face difficulties when extracting edge features from hyperspectral images (HSIs) that contain a large number of bands with very narrow gap in the spectral domain. High performance computing (HPC) is a technique used to assemble process and analyze more quantity of remote sensing (satellite) data which needs less processing time. HPC method is used in distributed & cluster Computing, parallel computing and following satellite data analyzing & processing approaches like geo-referencing, image edge detection, image classification, image mosaicking, Morphological/neural approach and image fusion for hyperspectral satellite data. Stimulated by means of the specific clustering of theory called gravitational theory, an original edge detection process. The conservative methods include the usage of linear time invariant filters. Recognition of edge is done by these filters as a These filters recognize an edge as a sudden conversion of grey scale pixel intensity. These techniques are well recognized and computationally effective. Canny, Sobel, Robert, Kirsch, Prewitt and LOG, on idea of spatial differential filters by using local grade. Within the less time these filters process the data and are inclined to noise. But in fuzzy technique, it does not include like this restriction, by altering the rules and output parameters, it is possible to change the thickness of the edge. In this paper, a new technique is created on fuzzy logic, which is suggested for detection of edge in digital images deprived of finding the threshold value. This method starts through segmenting the images into sections by means of fluctuating 3 × 3 binary matrixes. The pixels of edge are planned to a range of standards separated from each other. The results of this proposed technique is compared with the results obtained from PSO and neural network methods. This suggested technique provides a permanent effect in the lines smoothness and straightness for the straight lines and good roundness for the curved lines. And sharpness in the curves. [ABSTRACT FROM AUTHOR]

Cloud Computing has become the preferred choice of performing scientific applications over the cloud since the computing has become as a utility. Cloud-based services have evolved exponentially and also the sophistication of Cloud infrastructure supporting these services has grown. Running traditional applications such as scientific data processing on Cloud, needs to consider the suitability of the could for the compliance of the Cloud critical metrics. This article aims to study the performance of public clouds to support the scientific computing, which were performed on Grid computing, High performance computing or cluster computing. This article tries to address the issue by designing and developing a framework to measure the critical metrics. The performance of the public clouds was assessed through probing, simulation and historic data. The results indicate that only a few provide heterogeneous cloud services. Performance of some of the clouds services can be improved by tweaking the critical metrics. [ABSTRACT FROM AUTHOR]

Latest scientific application and simulation are requiring powerful computing resources. To get the sufficient resources, more and more computer experts are constructing high-end supercomputer, cluster computing or cloud computing. Although most corporations are starting to use high performance computing by organizing cluster, they rarely add some extra resources dynamically when computing power is not sufficient. In addition, they are also expensive to use insufficient resources as cloud computing resources. To overcome resource limitations when using cloud computing to conduct large computations, we exploit underutilized personal desktops by organizing a heterogeneous cluster environment. An open-source based scheduler, Son of Grid Engine (SGE) is provided in the integrated environment to manage or control the extra computing resources. Two kinds of groups are classified according to the individuals degree of utilization: a full-time group that provides resources at any time, and a custom-time group that provides resources only at designated times. We developed a resource manager that can turn on/off and schedule the SGE execution daemon; this manager performs as the core engine. The resource running time manager efficiently allocates and controls the members of affiliated computing resources. [ABSTRACT FROM AUTHOR]

Ever since the arrival of big data era, high performance computing technology which committed to solving largescale scientific computing problems, has been more and more extensively focused and widely applied. As the booming of high performance computer manufacturing industry, high performance computer cluster with float computing speed up to several quintillion times per second requires further optimization of the high performance computing environment middleware. In current high-performance computing environment middleware, the information transmission mode of resource information service cannot meet the requirement of rapid updating of massive information. In order to improve the performance of high performance computing environment middleware, this paper used ETL technology to optimize the information transmission mode to information synchronization mode. After optimizing the processing of 2 000 pieces of resource information, it reduced the latency by90%, reduced the system load by 98%, and reduced the number of network connections by 90%, which provided enhanced user experience to the system maintenance personnel and users. [ABSTRACT FROM AUTHOR]

Modeling and simulation (M&S) is a promising solution for manufacturers to verify and resolve errors in advance. Recently, M&S using high-performance computing has attracted attention as a technology trend for its efficiency in terms of the time and the cost. However, a huge amount of computing resources is still required to perform an M&S analysis workflow within a short time. For the resolution of this issue, we organize a heterogeneous cluster environment with individual nodes of the diverse performance and OS. An open source-based task scheduler named Son of Grid Engine is used for managing the heterogeneous resources for the experiment. To verify the efficiency and validity of the organization scheme, we have developed an environment for analyzing products and monitoring work through CalculiX, an open source-based finite element analysis solver. [ABSTRACT FROM AUTHOR]

Most courses on cluster computing in graduate and postgraduate studies are focused on parallel programming and high-performance/high-throughput computing. This is the typical usage of clusters in academia and research centres. However, nowadays, many companies are providing web, mail and, in general, Internet services using computer clusters. These services require a different “cluster flavour”: high-performance service and high availability. Despite the fact that computer clusters for each environment demand a different configuration, most university cluster computing courses keep focusing only on high-performance computing, ignoring other possibilities. In this paper, we propose several teaching strategies for a course on cluster computing that could fill this gap. The content developed here would be taught as a part of the course. The subject shows several strategies about how to configure, test and evaluate a high-availability/load-balanced Internet server. A virtualization-based platform is used to build a cluster prototype, using Linux as its operating system. Evaluation of the course shows that students knowledge and skills on the subject are improved at the end of the course. On the other hand, regarding the teaching methodology, the results obtained in the yearly survey of the University confirm student satisfaction. [ABSTRACT FROM AUTHOR]

Divide-and-conquer is one of the most important patterns of parallelism, being applicable to a large variety of problems. In addition, the most powerful parallel systems available nowadays are computer clusters composed of distributed-memory nodes that contain an increasing number of cores that share a common memory. The optimal exploitation of these systems often requires resorting to a hybrid model that mimics the underlying hardware by combining a distributed and a shared memory parallel programming model. This results in longer development times and increased maintenance costs. In this paper we present a very general skeleton library that allows to parallelize any divide-and-conquer problem in hybrid distributed-shared memory systems with little effort while providing much flexibility and good performance. Our proposal combines a message-passing paradigm at the process level and a threaded model inside each process, hiding the related complexity from the user. The evaluation shows that this skeleton provides performance comparable, and often better than that of manually optimized codes while requiring considerably less effort when parallelizing applications on multi-core clusters. [ABSTRACT FROM AUTHOR]

Computer clusters are today a cost-effective way of providing either high-performance and/or high-availability. The flexibility of their configuration aims to fit the needs of multiple environments, from small servers to SME and large Internet servers. For these reasons, their usage has expanded not only in academia but also in many companies. However, each environment needs a different “cluster flavour”. High-performance and high-throughput computing are required in universities and research centres while high-performance service and high-availability are usually reserved to use in companies. Despite this fact, most university cluster computing courses continue to cover only high-performance computing, usually ignoring other possibilities. In this paper, a master-level course which attempts to fill this gap is discussed. It explores the different types of cluster computing as well as their functional basis, from a very practical point of view. As part of the teaching methodology, each student builds from scratch a computer cluster based on a virtualization tool. The entire process is designed to be scalable. The goal is to be able to apply it to an actual computer cluster with a larger number of nodes, such as those the students may subsequently encounter in their professional life. [ABSTRACT FROM AUTHOR]

Cloud computing infrastructure is suitable for meeting computational needs of large task sizes. Optimal scheduling of tasks in cloud computing environment has been proved to be an NP-complete problem, hence the need for the application of heuristic methods. Several heuristic algorithms have been developed and used in addressing this problem, but choosing the appropriate algorithm for solving task assignment problem of a particular nature is difficult since the methods are developed under different assumptions. Therefore, six rule based heuristic algorithms are implemented and used to schedule autonomous tasks in homogeneous and heterogeneous environments with the aim of comparing their performance in terms of cost, degree of imbalance, makespan and throughput. First Come First Serve (FCFS), Minimum Completion Time (MCT), Minimum Execution Time (MET), Max-min, Min-min and Sufferage are the heuristic algorithms considered for the performance comparison and analysis of task scheduling in cloud computing. [ABSTRACT FROM AUTHOR]

This paper describes our new hybrid parallelization of the Finite Element Tearing and Interconnecting (FETI) method for the multi-socket and multi-core computer cluster. This is an essential step in our development of the Hybrid FETI solver were small number of neighboring subdomains is aggregated into clusters and each cluster is processed by a single compute node. In our previous work we have implemented FETI solver using MPI parallelization into our ESPRESO solver. The proposed hybrid implementation provides better utilization of resources of modern HPC machines using advanced shared memory runtime systems such as Cilk++ runtime. Cilk++ is an alternative to OpenMP which is used by ESPRESO for shared memory parallelization. We have compared the performance of the hybrid parallelization to MPI-only parallelization. The results show that we have reduced both solver runtime and memory utilization. This allows a solver to use a larger number of smaller sub-domains and in order to solve larger problems using a limited number of compute nodes. This feature is essential for users with smaller computer clusters. In addition, we have evaluated this approach with large-scale benchmarks of size up to 1.3 billion of unknowns to show that the hybrid parallelization also reduces runtime of the FETI solver for these types of problems. [ABSTRACT FROM AUTHOR]

Graphic Processing Units (GPUs) are widely used in high performance computing, due to their high computational power and high performance per Watt. However, one of the main bottlenecks of GPU-accelerated cluster computing is the data transfer between distributed GPUs. This not only affects performance, but also power consumption. The most common way to utilize a GPU cluster is a hybrid model, in which the GPU is used to accelerate the computation, while the CPU is responsible for the communication. This approach always requires a dedicated CPU thread, which consumes additional CPU cycles and therefore increases the power consumption of the complete application. In recent work we have shown that the GPU is able to control the communication independently of the CPU. However, there are several problems with GPU-controlled communication. The main problem is intra-GPU synchronization, since GPU blocks are non-preemptive. Therefore, the use of communication requests within a GPU can easily result in a deadlock. In this work we show how dynamic parallelism solves this problem. GPU-controlled communication in combination with dynamic parallelism allows keeping the control flow of multi-GPU applications on the GPU and bypassing the CPU completely. Using other in-kernel synchronization methods results in massive performance losses, due to the forced serialization of the GPU thread blocks. Although the performance of applications using GPU-controlled communication is still slightly worse than the performance of hybrid applications, we will show that performance per Watt increases by up to 10% while still using commodity hardware. [ABSTRACT FROM AUTHOR]

• This work presents BLMM, a Python tool for analysing mass-univariate LMMs. • BLMM utilizes vectorization speed-ups when working with multiple voxels. • BLMM accounts for the ǣvoxel-wise missingnessǥ ubiquitous in large-n fMRI analyses. • The correctness and performance of BLMM are assessed via extensive simulation. • The paper concludes by providing a real data example based on the UK Biobank. Within neuroimaging large-scale, shared datasets are becoming increasingly commonplace, challenging existing tools both in terms of overall scale and complexity of the study designs. As sample sizes grow, researchers are presented with new opportunities to detect and account for grouping factors and covariance structure present in large experimental designs. In particular, standard linear model methods cannot account for the covariance and grouping structures present in large datasets, and the existing linear mixed models (LMM) tools are neither scalable nor exploit the computational speed-ups afforded by vectorisation of computations over voxels. Further, nearly all existing tools for imaging (fixed or mixed effect) do not account for variability in the patterns of missing data near cortical boundaries and the edge of the brain, and instead omit any voxels with any missing data. Yet in the large- n setting, such a voxel-wise deletion missing data strategy leads to severe shrinkage of the final analysis mask. To counter these issues, we describe the "Big" Linear Mixed Models (BLMM) toolbox, an efficient Python package for large-scale fMRI LMM analyses. BLMM is designed for use on high performance computing clusters and utilizes a Fisher Scoring procedure made possible by derivations for the LMM Fisher information matrix and score vectors derived in our previous work, Maullin-Sapey and Nichols (2021). [ABSTRACT FROM AUTHOR]

High Performance Computing usually leverages messaging libraries such as MPI, GASNet, or OpenSHMEM, among others, in order to exchange data among processes in large-scale clusters. Furthermore, these libraries make use of specialized low-level network layers in order to achieve as much performance as possible from hardware interconnects such as InfiniBand or 40 Gb Ethernet, for example. EXTOLL is an emerging network targeted at high performance clusters. Specialized low-level network layers require some kind of flow control in order to prevent buffer overflows at the receiver side. In this paper we present a new end-to-end flow control mechanism that is able to dynamically adapt, at execution time, the buffer resources used by a process according to the communication pattern of the parallel application and the varying activity among communicating peers. The tests carried out on a 64-node 1024-core EXTOLL cluster show that our new dynamic flow control mechanism presents very low overhead with an extraordinarily high buffer efficiency, as overall buffer resources are reduced by 4× with respect to the amount of buffers required by a static flow control protocol achieving similar low overhead levels. [ABSTRACT FROM AUTHOR]

One of the major challenges in the high performance computing (HPC) clusters is intelligent power management to improve energy efficiency. The key contribution of the presented work is the modeling of a Power Aware Job Scheduler (PAJS) for HPC clusters, such that the: (a) threshold voltage is adjusted judiciously to achieve energy efficiency and (b) response time is minimized by scaling the supply voltage. The PAJS considers the symbiotic relationship between power and performance and caters the optimization of the both, simultaneously. The key novelty in our work is utilization of the dynamic threshold-voltage scaling (DTVS) for the reduction of cumulative power utilized by each node in the cluster. Moreover, to enhance the performance of the resource scheduling strategies in this work, independent tasks within a job are scheduled to most suitable computing nodes (CNs). This paper analyzes and compares eight scheduling techniques in terms of energy consumption and makespan. Primarily, the most suitable dynamic voltage scaling (DVS) level adhering to the deadline is identified for each of the CNs by the scheduling heuristics. Afterwards, the DTVS is employed to scale down the static, as well as dynamic power by regulating the supply and bias voltages. Finally, the per node threshold scaling is used attain power saving. Our simulation results affirm that the proposed methodology significantly reduces the energy consumption using the DTVS. [ABSTRACT FROM AUTHOR]

Energy preservation in computing systems is an important research topic nowadays. Clusters are usually composed of different hardware with different performance and energy consumption. Performance and efficiency are two metrics introduced in this paper that describe servers' computational power and energy efficiency, respectively. Based on these metrics, we propose three scheduling policies for hard real-time tasks that are executed on a heterogeneous cluster with power-aware dynamic voltage/frequency scaling processors. Simulation experiments show promising results as compared to those of other existing scheduling policies. In order to study the effects of processor failures, the impact of replacing high-performance processors with high-efficiency processors is studied. Furthermore, the load balancing mechanism used in the system is viewed from an energy perspective. [ABSTRACT FROM AUTHOR]

Cloud computing techniques take the form of distributed computing by utilizing multiple computers to execute computing simultaneously on the service side. To process the increasing quantity of multimedia data, numerous large-scale multimedia data storage computing techniques in the cloud computing have been developed. Of all the techniques, Hadoop plays a key role in the cloud computing. Hadoop, a computing cluster formed by low-priced hardware, can conduct the parallel computing of petabytes of multimedia data. Hadoop features high-reliability, high-efficiency, and high-scalability. The numerous large-scale multimedia data computing techniques include not only the key core techniques, Hadoop and MapReduce, but also the data collection techniques, such as File Transfer Protocol and Flume. In addition, distributed system configuration allocation, automatic installation, and monitoring platform building and management techniques are all included. As a result, only with the integration of all the techniques, a reliable large-scale multimedia data platform can be offered. In this paper, we introduce how cloud computing can make a breakthrough by proposing a multimedia social network dataset on Hadoop platform and implementing a prototype version. Detailed specifications and design issues are discussed as well. An important finding of this article is that we can save more time if we conduct the multimedia social networking analysis using Cloud Hadoop Platform rather than using a single computer. The advantages of cloud computing over the traditional data processing practices are fully demonstrated in this article. The applicable framework designs and the tools available for the large-scale data processing are also proposed. We show the experimental multimedia data including data sizes and processing time. [ABSTRACT FROM AUTHOR]

Complex eScience and other sophisticated applications in the field of HPC imply new demands that queuing based resource management systems cannot meet. To guarantee Quality of Service and co-allocation in the Grid, planning based resource management systems implementing advance reservation are needed. These systems face new challenges as a planning based management system has to keep track of the jobs and reservations in the future. Additionally, during the negotiation process of incoming reservations, a good overview of the remaining, not-yet reserved capacity is needed-not only for the current allocation, but also for the whole book-ahead time. Therefore, the resource management problem becomes a two dimensional problem for advance reservations in this field. In this paper different data structures are investigated and discussed in order to fit to planning based resource management. As a result the benefits of using lists of resource allocation or free blocks are exposed. This general idea widely used to manage continuous resources is extended to cover not only the resource dimension but also the time dimension. The list of blocks approach is evaluated in a Grid level and a local resource management system for a computing cluster. The extensive simulations showed a better runtime and higher reservation success rate compared with the currently favored approach of a slotted time and the more sophisticated approach based on AVL trees. [ABSTRACT FROM AUTHOR]

High Performance Cluster Computing Systems (HPCSs) represent the best performance because their configuration is customized regarding the features of the problem to be solved at design time. Therefore, if the problem has static nature and features, the best customized configuration can be done. New generations of scientific and industrial problems usually have dynamic nature and behavior. A drawback of this dynamicity is that the customized HPCSs face challenges at runtime, and consequently show the worse performance. The reason for this might be due to the fact that dynamic problems are not adapted to configuration of the HPCS. Hence, requests of the dynamic problem are not in the direction of the HPCS configuration. The main proposed solutions for this challenge are dynamic load balancing or using reconfigurable platforms. In this paper, a vector algebra-based model for HPCS reconfiguration at runtime is presented and named AMRC. This model determines the element causing the dynamic behavior and analyzes the reason regarding both software and hardware at runtime. Some results of the presented model show that by defining a general state vector whose direction is toward reaching high performance computing and whose weight is based on the initial features and explicit requirements of the problem, as well as by defining a vector for each process in the problem at runtime, we can trace changes in the directions and uncover the reason for them. [ABSTRACT FROM AUTHOR]

In high-performance computing on distributed-memory systems, communication often represents a significant part of the overall execution time. The relative cost of communication will certainly continue to rise as compute-density growth follows the current technology and industry trends. Design of lower-communication alternatives to fundamental computational algorithms has become an important field of research. For distributed 1-D FFT, communication cost has hitherto remained high as all industry-standard implementations perform three all-to-all internode data exchanges (also called global transposes). These communication steps indeed dominate execution time. In this paper, we present a mathematical framework from which many single-all-to-all and easy-to-implement 1-D FFT algorithms can be derived. For large-scale problems, our implementation can be twice as fast as leading FFT libraries on state-of-the-art computer clusters. Moreover, our framework allows tradeoff between accuracy and performance, further boosting performance if reduced accuracy is acceptable. [ABSTRACT FROM AUTHOR]

In recent years, High Performance Computing (HPC) systems have been shifting from expensive massively parallel architectures to clusters of commodity PCs to take advantage of cost and performance benefits. Fault tolerance in such systems is a growing concern for long-running applications. In this paper, we briefly review the failure rates of HPC systems and also survey the fault tolerance approaches for HPC systems and issues with these approaches. Rollback-recovery techniques which are most often used for long-running applications on HPC clusters are discussed because they are widely used for long-running applications on HPC systems. Specifically, the feature requirements of rollback-recovery are discussed and a taxonomy is developed for over twenty popular checkpoint/restart solutions. The intent of this paper is to aid researchers in the domain as well as to facilitate development of new checkpointing solutions. [ABSTRACT FROM AUTHOR]

• An orchestration aid system (OAS) was developed to support a gridded crop growth simulations. • The OAS requires minimum labor and expertise in management and operation of cluster computers. • Functionality of the OAS was demonstrated using single board computers. • The OAS would help researchers design site-specific options for climate change adaptation. Spatial simulations of crop growth under climate change have been limited to researchers who have access to the resources for high performance computing. The objective of this study was to develop an orchestration aid system for concurrent gridded simulations of crop growth, which would support the design of climate change adaptation options on crop production without expertise in distributed computing. The orchestration aid system was designed to help a user build a set of virtualized cluster computers using a simple input file, which would require little expertise in distributed computing, rather than manual configuration. The orchestration aid system, which was referred to as GROWLERS-kube, was implemented to launch multiple sets of gridded simulations using pods or containers managed by Kubernetes. As a case study, GROWLER-kube was executed using 16 Raspberry Pi 4 computers to perform 120 sets of the gridded simulations under diverse crop management options, including varying planting date and cultivar, for the period from 2001 to 2010. The wall time or the elapsed time for the given sets of the gridded simulation differed by configuration of virtualized cluster computers, such as the number of pods used for server and client nodes, although the total number of physical nodes were identical. For example, the wall time difference between virtualized cluster computer sets was about 28.9% when 15 worker nodes were used. In particular, the acceleration of the gridded simulations was at maximum using a large number of the virtualized cluster computers with a small number of nodes. It was found that the spatial distribution of planting dates and cultivars was similar to that of a previous study based on field experiments mostly in regions where rice is usually grown. These results suggest that GROWLERS-kube would facilitate the spatial assessment of climate change impact on crop production without considerable effort and expertise in distributed computing, which would aid a researcher to focus on the design of adaptation strategies. [ABSTRACT FROM AUTHOR]

• Patient-specific treatment planning is required in cancer treatment using ultrasound. • This needs efficient numerical methods to solve ultrasound field in the body accurately. • This paper presents a fast full-wave solver for solving ultrasound field in the body. • Numerical experiments demonstrate the great performance of the new formulation. Therapeutic ultrasound is a promising non-invasive method for inducing various beneficial biological effects in the human body. In cancer treatment applications, high-power ultrasound is focused at a target tissue volume to ablate the malignant tumour. The success of the procedure depends on the ability to accurately focus ultrasound and destroy the target tissue volume through coagulative necrosis whilst preserving the surrounding healthy tissue. Patient-specific treatment planning strategies are therefore being developed to increase the efficacy of such therapies, while reducing any damage to healthy tissue. These strategies require to use high-performance computing methods to solve ultrasound wave propagation in the body quickly and accurately. For realistic clinical scenarios, all numerical methods which employ volumetric meshes require several hours or days to solve the full-wave propagation on a computer cluster. The boundary element method (BEM) is an efficient approach for modelling the wave field because only the boundaries of the hard and soft tissue regions require discretisation. This paper presents a multiple-domain BEM formulation with a novel preconditioner for solving the Helmholtz transmission problem (HTP). This new formulation is efficient at high-frequencies and where high-contrast materials are present. Numerical experiments are performed to solve the HTP in multiple domains comprising: (i) human ribs, an idealised abdominal fat layer and liver tissue, (ii) a human kidney with a perinephric fat layer, exposed to the acoustic field generated by a high-intensity focused ultrasound (HIFU) array transducer. The time required to solve the equations associated with these problems on a single workstation is of the order of minutes. These results demonstrate the great potential of this new BEM formulation for accurately and quickly solving ultrasound wave propagation problems in large anatomical domains which is essential for developing treatment planning strategies. [ABSTRACT FROM AUTHOR]

Background Several prediction problems in computational biology and genomic medicine are characterized by both big data as well as a high imbalance between examples to be learned, whereby positive examples can represent a tiny minority with respect to negative examples. For instance, deleterious or pathogenic variants are overwhelmed by the sea of neutral variants in the non-coding regions of the genome: thus, the prediction of deleterious variants is a challenging, highly imbalanced classification problem, and classical prediction tools fail to detect the rare pathogenic examples among the huge amount of neutral variants or undergo severe restrictions in managing big genomic data. Results To overcome these limitations we propose parSMURF, a method that adopts a hyper-ensemble approach and oversampling and undersampling techniques to deal with imbalanced data, and parallel computational techniques to both manage big genomic data and substantially speed up the computation. The synergy between Bayesian optimization techniques and the parallel nature of parSMURF enables efficient and user-friendly automatic tuning of the hyper-parameters of the algorithm, and allows specific learning problems in genomic medicine to be easily fit. Moreover, by using MPI parallel and machine learning ensemble techniques, parSMURF can manage big data by partitioning them across the nodes of a high-performance computing cluster. Results with synthetic data and with single-nucleotide variants associated with Mendelian diseases and with genome-wide association study hits in the non-coding regions of the human genome, involhing millions of examples, show that parSMURF achieves state-of-the-art results and an 80-fold speed-up with respect to the sequential version. Conclusions parSMURF is a parallel machine learning tool that can be trained to learn different genomic problems, and its multiple levels of parallelization and high scalability allow us to efficiently fit problems characterized by big and imbalanced genomic data. The C++ OpenMP multi-core version tailored to a single workstation and the C++ MPI/OpenMP hybrid multi-core and multi-node parSMURF version tailored to a High Performance Computing cluster are both available at https://github.com/AnacletoLAB/parSMURF. [ABSTRACT FROM AUTHOR]

We propose a novel parallel numerical algorithm for calculating the smallest eigenvalues of highly ill-conditioned Hankel matrices. It is based on the LDLT decomposition and involves finding a k × k sub-matrix of the inverse of the original N × N Hankel matrix H N − 1. The computation involves extremely high precision arithmetic, message passing interface, and shared memory parallelisation. We demonstrate that this approach achieves good scalability on a high performance computing cluster (HPCC) which constitutes a major improvement of the earlier approaches. We use this method to study a family of Hankel matrices generated by the weight w (x) = e − x β , supported on [0, ∞) and β > 0. Such weight generates a Hankel determinant, a fundamental object in random matrix theory. In the situation where β > 1/2, the smallest eigenvalue tends to 0 exponentially fast. If β < 1/2, which is the situation where the classical moment problem is indeterminate, then the smallest eigenvalue is bounded from below by a positive number. If β = 1 / 2 , it is conjectured that the smallest eigenvalue tends to 0 algebraically, with a precise exponent. The algorithm run on the HPCC producing a fantastic match between the theoretical value of 2/ π and the numerical result. [ABSTRACT FROM AUTHOR]