Back to Search
Start Over
Simplifying non-contiguous data transfer with MPI for Python.
- Source :
-
Journal of Supercomputing . Nov2023, Vol. 79 Issue 17, p20019-20040. 22p. - Publication Year :
- 2023
-
Abstract
- Python is becoming increasingly popular in scientific computing. The package MPI for Python (mpi4py) allows writing efficient parallel programs that scale across multiple nodes. However, it does not support non-contiguous data via slices, which is a well-known feature of NumPy. In this work, we therefore evaluate several methods to support the direct transfer of non-contiguous arrays in mpi4py. This significantly simplifies the code, while the performance basically stays the same. In a PingPong-, Stencil- and Lattice-Boltzmann-Benchmark, we compare the common manual copying, a NumPy-Copy design and a design that is based on MPI derived datatypes. In one case, the MPI derived datatype design could achieve a speedup of 15% in a Stencil-Benchmark on four compute nodes. Our designs are superior to naive manual copies, but for maximum performance manual copies with pre-allocated buffers or MPI persistent communication will be a better choice. [ABSTRACT FROM AUTHOR]
- Subjects :
- *PYTHONS
*SCIENTIFIC computing
*PARALLEL programming
Subjects
Details
- Language :
- English
- ISSN :
- 09208542
- Volume :
- 79
- Issue :
- 17
- Database :
- Academic Search Index
- Journal :
- Journal of Supercomputing
- Publication Type :
- Academic Journal
- Accession number :
- 172445267
- Full Text :
- https://doi.org/10.1007/s11227-023-05398-7