Search

We present two multilingual LLMs designed to embrace Europe's linguistic diversity by supporting all 24 official languages of the European Union. Trained on a dataset comprising around 60% non-English data and utilizing a custom multilingual tokenizer, our models address the limitations of existing LLMs that predominantly focus on English or a few high-resource languages. We detail the models' development principles, i.e., data composition, tokenizer optimization, and training methodologies. The models demonstrate competitive performance across multilingual benchmarks, as evidenced by their performance on European versions of ARC, HellaSwag, MMLU, and TruthfulQA.

The rapid advancement of machine learning (ML) technologies has driven the development of specialized hardware accelerators designed to facilitate more efficient model training. This paper introduces the CARAML benchmark suite, which is employed to assess performance and energy consumption during the training of transformer-based large language models and computer vision models on a range of hardware accelerators, including systems from NVIDIA, AMD, and Graphcore. CARAML provides a compact, automated, extensible, and reproducible framework for assessing the performance and energy of ML workloads across various novel hardware architectures. The design and implementation of CARAML, along with a custom power measurement tool called jpwr, are discussed in detail., Comment: To be published in Workshop Proceedings of The International Conference for High Performance Computing Networking, Storage, and Analysis (SC-W '24) (2024)

Benchmarks are essential in the design of modern HPC installations, as they define key aspects of system components. Beyond synthetic workloads, it is crucial to include real applications that represent user requirements into benchmark suites, to guarantee high usability and widespread adoption of a new system. Given the significant investments in leadership-class supercomputers of the exascale era, this is even more important and necessitates alignment with a vision of Open Science and reproducibility. In this work, we present the JUPITER Benchmark Suite, which incorporates 16 applications from various domains. It was designed for and used in the procurement of JUPITER, the first European exascale supercomputer. We identify requirements and challenges and outline the project and software infrastructure setup. We provide descriptions and scalability studies of selected applications and a set of key takeaways. The JUPITER Benchmark Suite is released as open source software with this work at https://github.com/FZJ-JSC/jubench., Comment: To be published in Proceedings of The International Conference for High Performance Computing Networking, Storage, and Analysis (SC '24) (2024)

We devise a spectral divide-and-conquer scheme for matrices that are self-adjoint with respect to a given indefinite scalar product (i.e. pseudosymmetic matrices). The pseudosymmetric structure of the matrix is preserved in the spectral division, such that the method can be applied recursively to achieve full diagonalization. The method is well-suited for structured matrices that come up in computational quantum physics and chemistry. In this application context, additional definiteness properties guarantee a convergence of the matrix sign function iteration within two steps when Zolotarev functions are used. The steps are easily parallelizable. Furthermore, it is shown that the matrix decouples into symmetric definite eigenvalue problems after just one step of spectral division.

We present methods for computing the generalized polar decomposition of a matrix based on the dynamically weighted Halley (DWH) iteration. This method is well established for computing the standard polar decomposition. A stable implementation is available, where matrix inversion is avoided and QR decompositions are used instead. We establish a natural generalization of this approach for computing generalized polar decompositions with respect to signature matrices. Again the inverse can be avoided by using a generalized QR decomposition called hyperbolic QR decomposition. However, this decomposition does not show the same favorable stability properties as its orthogonal counterpart. We overcome the numerical difficulties by generalizing the CholeskyQR2 method. This method computes the standard QR decomposition in a stable way via two successive Cholesky factorizations. An even better numerical stability is achieved by employing permuted graph bases, yielding residuals of order $10^{-14}$ even for badly conditioned matrices, where other methods fail.

Optical properties of materials related to light absorption and scattering are explained by the excitation of electrons. The Bethe-Salpeter equation is the state-of-the-art approach to describe these processes from first principles (ab initio), i.e. without the need for empirical data in the model. To harness the predictive power of the equation, it is mapped to an eigenvalue problem via an appropriate discretization scheme. The eigenpairs of the resulting large, dense, structured matrix can be used to compute dielectric properties of the considered crystalline or molecular system. The matrix always shows a $2\times 2$ block structure. Additionally, certain definiteness properties typically hold. One form can be acquired for crystalline systems, another one is more general and can for example be used to study molecules. In this work, we present new theoretical results characterizing the structure of the two forms in the language of non-standard scalar products. These results enable us to develop a new perspective on the state-of-the-art solution approach for crystalline systems. This new viewpoint is used to develop two new methods for solving the eigenvalue problem. One requires less computational effort while providing the same degree of accuracy. The other one improves the expected accuracy, compared to methods currently in use, with a comparable performance. Both methods are well suited for high performance environments and only rely on basic numerical linear algebra building blocks.

For a given matrix, we are interested in computing GR decompositions $A=GR$, where $G$ is an isometry with respect to given scalar products. The orthogonal QR decomposition is the representative for the Euclidian scalar product. For a signature matrix, a respective factorization is given as the hyperbolic QR decomposition. Considering a skew-symmetric matrix leads to the symplectic QR decomposition. The standard approach for computing GR decompositions is based on the successive elimination of subdiagonal matrix entries. For the hyperbolic and symplectic case, this approach does in general not lead to a satisfying numerical accuracy. An alternative approach computes the QR decomposition via a Cholesky factorization, but also has bad stability. It is improved by repeating the procedure a second time. In the same way, the hyperbolic and the symplectic QR decomposition are related to the $LDL^T$ and a skew-symmetric Cholesky-like factorization. We show that methods exploiting this connection can provide better numerical stability than elimination-based approaches.

We present a high-performance solver for dense skew-symmetric matrix eigenvalue problems. Our work is motivated by applications in computational quantum physics, where one solution approach to solve the so-called Bethe-Salpeter equation involves the solution of a large, dense, skew-symmetric eigenvalue problem. The computed eigenpairs can be used to compute the optical absorption spectrum of molecules and crystalline systems. One state-of-the art high-performance solver package for symmetric matrices is the ELPA (Eigenvalue SoLvers for Petascale Applications) library. We extend the methods available in ELPA to skew-symmetric matrices. This way, the presented solution method can benefit from the optimizations available in ELPA that make it a well-established, efficient and scalable library, such as GPU support. We compare performance and scalability of our method to the only available high-performance approach for skew-symmetric matrices, an indirect route involving complex arithmetic. In total, we achieve a performance that is up to 3.67 higher than the reference method using Intel's ScaLAPACK implementation. The runtime to solve the Bethe-Salpeter-Eigenvalue problem can be improved by a factor of 10. Our method is freely available in the current release of the ELPA library.