Search

Your search keyword '"Dubois, Lionel"' showing total 5 results
Start Over Author "Dubois, Lionel" Publication Year Range Last 3 years Publisher ieee
5 results on '"Dubois, Lionel"'

4. MadPipe: Memory Aware Dynamic Programming Algorithm for Pipelined Model Parallelism

Author

Beaumont, Olivier, Eyraud-Dubois, Lionel, Shilova, Alena, Beaumont, Olivier, High-End Parallel Algorithms for Challenging Numerical Simulations (HiePACS), Laboratoire Bordelais de Recherche en Informatique (LaBRI), Université de Bordeaux (UB)-Centre National de la Recherche Scientifique (CNRS)-École Nationale Supérieure d'Électronique, Informatique et Radiocommunications de Bordeaux (ENSEIRB)-Université de Bordeaux (UB)-Centre National de la Recherche Scientifique (CNRS)-École Nationale Supérieure d'Électronique, Informatique et Radiocommunications de Bordeaux (ENSEIRB)-Inria Bordeaux - Sud-Ouest, Institut National de Recherche en Informatique et en Automatique (Inria)-Institut National de Recherche en Informatique et en Automatique (Inria), Université de Bordeaux (UB)-École Nationale Supérieure d'Électronique, Informatique et Radiocommunications de Bordeaux (ENSEIRB)-Centre National de la Recherche Scientifique (CNRS)-Université de Bordeaux (UB)-École Nationale Supérieure d'Électronique, Informatique et Radiocommunications de Bordeaux (ENSEIRB)-Centre National de la Recherche Scientifique (CNRS)-Inria Bordeaux - Sud-Ouest, Scool (Scool), Inria Lille - Nord Europe, Institut National de Recherche en Informatique et en Automatique (Inria)-Institut National de Recherche en Informatique et en Automatique (Inria)-Centre de Recherche en Informatique, Signal et Automatique de Lille - UMR 9189 (CRIStAL), and Centrale Lille-Université de Lille-Centre National de la Recherche Scientifique (CNRS)-Centrale Lille-Université de Lille-Centre National de la Recherche Scientifique (CNRS)

Subjects
[INFO.INFO-DC] Computer Science [cs]/Distributed, Parallel, and Cluster Computing [cs.DC], [INFO.INFO-DC]Computer Science [cs]/Distributed, Parallel, and Cluster Computing [cs.DC]
Abstract

International audience; The training phase in Deep Neural Networks (DNNs) is very computationally intensive and is nowadays often performed on parallel computing platforms, ranging from a few GPUs to several thousand GPUs. The strategy of choice for the parallelization of training is the so-called data parallel approach, based of the parallel training of the different inputs (typically images) and a the aggregation of network weights with collective communications (AllReduce). The scalability of this approach is limited both by the memory available on each node and the networking capacities for collective operations. Recently, a parallel model approach, in which the network weights are distributed and images are trained in a pipeline/stream manner over the computational nodes has been proposed (Pipedream, Gpipe). In this paper, we formalize in detail the optimization problem associated with the placement of DNN layers onto computation resources when using pipelined model parallelism, and we derive a dynamic programming based heuristic, MadPipe, that allows to significantly improve the performance of the parallel model approach compared to the literature.

Published
2022

5. TEXTAROSSA: Towards EXtreme scale Technologies and Accelerators for euROhpc hw/Sw Supercomputing Applications for exascale

Author

Agosta, Giovanni, primary, Cattaneo, Daniele, additional, Fornaciari, William, additional, Galimberti, Andrea, additional, Massari, Giuseppe, additional, Reghenzani, Federico, additional, Terraneo, Federico, additional, Zoni, Davide, additional, Brandolese, Carlo, additional, Celino, Massimo, additional, Iannone, Francesco, additional, Palazzari, Paolo, additional, Zummo, Giuseppe, additional, Bernaschi, Massimo, additional, D'Ambra, Pasqua, additional, Saponara, Sergio, additional, Danelutto, Marco, additional, Torquati, Massimo, additional, Aldinucci, Marco, additional, Arfat, Yasir, additional, Cantalupo, Barbara, additional, Colonnelli, Iacopo, additional, Esposito, Roberto, additional, Martinelli, Alberto R., additional, Mittone, Gianluca, additional, Beaumont, Olivier, additional, Bramas, Berenger, additional, Eyraud-Dubois, Lionel, additional, Goglin, Brice, additional, Guermouche, Abdou, additional, Namyst, Raymond, additional, Thibault, Samuel, additional, Filgueras, Antonio, additional, Vidal, Miquel, additional, Alvarez, Carlos, additional, Martorell, Xavier, additional, Oleksiak, Ariel, additional, Kulczewski, Michal, additional, Lonardo, Alessandro, additional, Vicini, Piero, additional, Cicero, Francesca Lo, additional, Simula, Francesco, additional, Biagioni, Andrea, additional, Cretaro, Paolo, additional, Frezza, Ottorino, additional, Paolucci, Pier Stanislao, additional, Turisini, Matteo, additional, Giacomini, Francesco, additional, Boccali, Tommaso, additional, Montangero, Simone, additional, and Ammendola, Roberto, additional

Published
2021
Full Text
View/download PDF

Catalog

Books, media, physical & digital resources
See catalog results