Search

Your search keyword '"Valerie Hayot-Sasson"' showing total 13 results
Start Over Author "Valerie Hayot-Sasson"
13 results on '"Valerie Hayot-Sasson"'

1. Planning for ecological connectivity across scales of governance in a multifunctional regional landscape

Author

Lael Parrott, Catherine Kyle, Valerie Hayot-Sasson, Charles Bouchard, and Jeffrey A. Cardille

Subjects
wildlife corridors, circuitscape, shrub-steppe ecosystems, functional connectivity, environmental planning, translational ecology, british columbia, Human ecology. Anthropogeography, GF1-900, Environmental sciences, GE1-350
Abstract

Although a landscape is a single environmental system, human systems of governance at the landscape scale are often fragmented across jurisdictions and diverse stakeholders, impeding coordinated planning to maintain ecological connectivity. We sought solutions to overcome this challenge for wildlife corridor conservation in a rapidly developing, multifunctional landscape in one of North America’s most endangered ecoregions. Circuitscape modelling was used to identify key wildlife movement corridors through our study area. We then describe how the results of this modelling have informed a collaborative multi-stakeholder process leading to shared conservation objectives across scales of governance and illustrate its success with our case study. We conclude that achieving landscape-scale conservation objectives requires ongoing and coordinated collaboration facilitated by a dedicated group of individuals and informed by science.EDITED BY Davide Geneletti

Published
2019
Full Text
View/download PDF

2. Proceedings of the OHBM Brainhack 2021

Author

Aki Nikolaidis, Matteo Manchini, Tibor Auer, Katherine L. Bottenhorn, Eva Alonso-Ortiz, Gabriel Gonzalez-Escamilla, Sofie Valk, Tristan Glatard, Melvin Selim Atay, Johanna M.M. Bayer, Janine Bijsterbosch, Johannes Algermissen, Natacha Beck, Patrick Bermudez, Isil Poyraz Bilgin, Steffen Bollmann, Claire Bradley, Megan E.J. Campbell, Bryan Caron, Oren Civier, Luis Pedro Coelho, Shady El Damaty, Samir Das, Mathieu Dugré, Eric Earl, Stefanie Evas, Nastassja Lopes Fischer, De Fu Yap, Kelly G. Garner, Remi Gau, Giorgio Ganis, Dylan G. E. Gomes, Martin Grignard, Samuel Guay, Omer Faruk Gulban, Sarah Hamburg, Yaroslav O. Halchenko, Valerie Hayot-Sasson, Dawn Liu Holford, Laurentius Huber, Manuel Illanes, Tom Johnstone, Avinash Kalyani, Kinshuk Kashyap, Han Ke, Ibrahim Khormi, Gregory Kiar, Vanja Ković, Tristan Kuehn, Achintya Kumar, Xavier Lecours-Boucher, Michael Lührs, Robert Luke, Cecile Madjar, Sina Mansour L., Chris Markeweicz, Paula Andrea Martinez, Alexandra McCarroll, Léa Michel, Stefano Moia, Aswin Narayanan, Guiomar Niso, Emmet A. O’Brien, Kendra Oudyk, François Paugam, Yuri G. Pavlov, Jean-Baptiste Poline, Benedikt A. Poser, Céline Provins, Pradeep Reddy Raamana, Pierre Rioux, David Romero-Bascones, Ekansh Sareen, Antonio Schettino, Alec Shaw, Thomas Shaw, Cooper A. Smout, Anđdela Šoškié, Jessica Stone, Suzy J Styles, Ryan Sullivan, Naoyuki Sunami, Shamala Sundaray, Jasmine Wei Rou, Dao Thanh Thuy, Sebastien Tourbier, Sebastián Urch, Alejandro de la Vega, Niruhan Viswarupan, Adina Wagner, Lennart Walger, Hao-Ting Wang, Fei Ting Woon, David White, Christopher Wiggins, Will Woods, Yu-Fang Yang, Ksenia Zaytseva, Judy D. Zhu, and Marcel P. Zwiers

Published
2023
Full Text
View/download PDF

3. GenSLMs: Genome-scale language models reveal SARS-CoV-2 evolutionary dynamics

Author

Maxim Zvyagin, Alexander Brace, Kyle Hippe, Yuntian Deng, Bin Zhang, Cindy Orozco Bohorquez, Austin Clyde, Bharat Kale, Danilo Perez-Rivera, Heng Ma, Carla M. Mann, Michael Irvin, J. Gregory Pauloski, Logan Ward, Valerie Hayot-Sasson, Murali Emani, Sam Foreman, Zhen Xie, Diangen Lin, Maulik Shukla, Weili Nie, Josh Romero, Christian Dallago, Arash Vahdat, Chaowei Xiao, Thomas Gibbs, Ian Foster, James J. Davis, Michael E. Papka, Thomas Brettin, Rick Stevens, Anima Anandkumar, Venkatram Vishwanath, and Arvind Ramanathan

Abstract

We seek to transform how new and emergent variants of pandemiccausing viruses, specifically SARS-CoV-2, are identified and classified. By adapting large language models (LLMs) for genomic data, we build genome-scale language models (GenSLMs) which can learn the evolutionary landscape of SARS-CoV-2 genomes. By pretraining on over 110 million prokaryotic gene sequences and finetuning a SARS-CoV-2-specific model on 1.5 million genomes, we show that GenSLMs can accurately and rapidly identify variants of concern. Thus, to our knowledge, GenSLMs represents one of the first whole genome scale foundation models which can generalize to other prediction tasks. We demonstrate scaling of GenSLMs on GPU-based supercomputers and AI-hardware accelerators utilizing 1.63 Zettaflops in training runs with a sustained performance of 121 PFLOPS in mixed precision and peak of 850 PFLOPS. We present initial scientific insights from examining GenSLMs in tracking evolutionary dynamics of SARS-CoV-2, paving the path to realizing this on large biological data.

Published
2022
Full Text
View/download PDF

4. Modeling the Linux page cache for accurate simulation of data-intensive applications

Author

Christopher J. Steele, Hoang-Dung Do, Rafael Ferreira da Silva, Henri Casanova, Valerie Hayot-Sasson, and Tristan Glatard

Subjects
FOS: Computer and information sciences, Data processing, Computer Science - Performance, Computer science, business.industry, Distributed computing, Big data, USable, Bottleneck, law.invention, Performance (cs.PF), Memory management, Workflow, Computer Science - Distributed, Parallel, and Cluster Computing, law, Page cache, Distributed, Parallel, and Cluster Computing (cs.DC), Wrench, business
Abstract

The emergence of Big Data in recent years has resulted in a growing need for efficient data processing solutions. While infrastructures with sufficient compute power are available, the I/O bottleneck remains. The Linux page cache is an efficient approach to reduce I/O overheads, but few experimental studies of its interactions with Big Data applications exist, partly due to limitations of real-world experiments. Simulation is a popular approach to address these issues, however, existing simulation frameworks do not simulate page caching fully, or even at all. As a result, simulation-based performance studies of data-intensive applications lead to inaccurate results. In this paper, we propose an I/O simulation model that includes the key features of the Linux page cache. We have implemented this model as part of the WRENCH workflow simulation framework, which itself builds on the popular SimGrid distributed systems simulation framework. Our model and its implementation enable the simulation of both single-threaded and multithreaded applications, and of both writeback and writethrough caches for local or network-based filesystems. We evaluate the accuracy of our model in different conditions, including sequential and concurrent applications, as well as local and remote I/Os. We find that our page cache model reduces the simulation error by up to an order of magnitude when compared to state-of-the-art, cacheless simulations., 10 pages, 8 figures, CCGrid

Published
2021

5. Performance benefits of Intel® Optane™ DC persistent memory for the parallel processing of large neuroimaging data

Author

Tristan Glatard, Valerie Hayot-Sasson, and Shawn T. Brown

Subjects
Computer science, business.industry, Locality, Big data, Petabyte, 02 engineering and technology, Visualization, 03 medical and health sciences, 0302 clinical medicine, Computer architecture, 020204 information systems, 0202 electrical engineering, electronic engineering, information engineering, Direct mode, Cache, Lazy evaluation, business, 030217 neurology & neurosurgery, Dram
Abstract

Open-access neuroimaging datasets have reached petabyte scale and continue to grow. The ability to leverage the entirety of these datasets is limited to a restricted number of research laboratories with both the capacity and infrastructure to process the data. Whereas Big Data engines have significantly reduced application performance penalties with respect to data movement, their applied strategies (e.g. data locality, in-memory computing and lazy evaluation) are not necessarily practical within neuroimaging workflows where intermediary results may need to be materialized to shared storage for post-processing analysis. In this paper we evaluate the performance advantage brought by Intel® Optane™ DC persistent memory for the processing of large neuroimaging datasets using the two available configurations modes: Memory mode and App Direct mode. We employ a synthetic algorithm on the 76 GiB and 603 GiB BigBrain, as well as apply a standard neuroimaging application on the Consortium for Reliability and Reproducibility (CoRR) dataset using 25 and 96 parallel processes in both cases. Our results show that the performance of applications leveraging persistent memory is superior to that of other storage devices, with the exception of DRAM. This is the case in both Memory and App Direct mode, irrespective of the amount of data and parallelism. Furthermore, persistent memory in App Direct mode is believed to benefit from the use of DRAM as a cache for writing when output data is significantly smaller than available memory. We believe the use of persistent memory will be beneficial to both neuroimaging applications running on HPC or visualization of large, high-resolution images.

Published
2020
Full Text
View/download PDF

6. Planning for ecological connectivity across scales of governance in a multifunctional regional landscape

Author

Charles Bouchard, Catherine Kyle, Jeffrey A. Cardille, Valerie Hayot-Sasson, and Lael Parrott

Subjects
Scale (ratio), Human systems engineering, british columbia, Management, Monitoring, Policy and Law, Wildlife corridor, circuitscape, Ecological connectivity, Environmental systems, lcsh:Human ecology. Anthropogeography, Ecology, Evolution, Behavior and Systematics, lcsh:Environmental sciences, Nature and Landscape Conservation, lcsh:GE1-350, business.industry, Functional connectivity, Corporate governance, Environmental resource management, functional connectivity, environmental planning, wildlife corridors, translational ecology, Geography, Automotive Engineering, lcsh:GF1-900, business, shrub-steppe ecosystems
Abstract

Although a landscape is a single environmental system, human systems of governance at the landscape scale are often fragmented across jurisdictions and diverse stakeholders, impeding coordinated planning to maintain ecological connectivity. We sought solutions to overcome this challenge for wildlife corridor conservation in a rapidly developing, multifunctional landscape in one of North America’s most endangered ecoregions. Circuitscape modelling was used to identify key wildlife movement corridors through our study area. We then describe how the results of this modelling have informed a collaborative multi-stakeholder process leading to shared conservation objectives across scales of governance and illustrate its success with our case study. We conclude that achieving landscape-scale conservation objectives requires ongoing and coordinated collaboration facilitated by a dedicated group of individuals and informed by science.EDITED BY Davide Geneletti

Published
2019

7. PyBIDS: Python tools for BIDS datasets

Author

Elizabeth DuPre, John A. Lee, Jean-Baptiste Poline, Dave F. Kleinschmidt, Taylor Salo, Yaroslav O. Halchenko, Michael Hanke, Dmitry Petrov, Adina Wagner, Valerie Hayot-Sasson, Christopher J. Markiewicz, Matteo Visconti di Oleggio Castello, Alejandro de la Vega, Johan D. Carlin, Krista DeStasio, Stefan Appelhoff, Tal Yarkoni, Chris Holdgraf, Oscar Esteban, Kirstie Whitaker, Dylan M. Nielson, Ross Blair, Lee S. Tirrell, Bertrand Thirion, Michael Notter, Gregory Kiar, Isla Staden, Mainak Jas, Krzysztof J. Gorgolewski, Russell A. Poldrack, and Quinten McNamara

Subjects
Computer science, Programming language, Neuroinformatics, Python (programming language), ddc:004, computer.software_genre, computer, Article, computer.programming_language
Abstract

Brain imaging researchers regularly work with large, heterogeneous, high-dimensional datasets. Historically, researchers have dealt with this complexity idiosyncratically, with every lab or individual implementing their own preprocessing and analysis procedures. The resulting lack of field-wide standards has severely limited reproducibility and data sharing and reuse.To address this problem, we and others recently introduced the Brain Imaging Data Standard (BIDS; (Gorgolewski et al., 2016)), a specification meant to standardize the process of representing brain imaging data. BIDS is deliberately designed with adoption in mind; it adheres to a user-focused philosophy that prioritizes common use cases and discourages complexity. By successfully encouraging a large and ever-growing subset of the community to adopt a common standard for naming and organizing files, BIDS has made it much easier for researchers to share, reuse, and process their data (Gorgolewski et al., 2017).The ability to efficiently develop high-quality spec-compliant applications itself depends to a large extent on the availability of good tooling. Because many operations recur widely across diverse contexts—for example, almost every tool designed to work with BIDS datasets involves regular file-filtering operations—there is a strong incentive to develop utility libraries that provide common functionality via a standardized, simple API.PyBIDS is a Python package that makes it easier to work with BIDS datasets. In principle, its scope includes virtually any functionality that is likely to be of general use when working with BIDS datasets (i.e., that is not specific to one narrow context). At present, its core and most widely used module supports simple and flexible querying and manipulation of BIDS datasets. PyBIDS makes it easy for researchers and developers working in Python to search for BIDS files by keywords and/or metadata; to consolidate and retrieve file-associated metadata spread out across multiple levels of a BIDS hierarchy; to construct BIDS-valid path names for new files; and to validate projects against the BIDS specification, among other applications.In addition to this core functionality, PyBIDS also contains an ever-growing set of modules that support additional capabilities meant to keep up with the evolution and expansion of the BIDS specification itself. Currently, PyBIDS includes tools for (1) reading and manipulating data contained in various BIDS-defined files (e.g., physiological recordings, event files, or participant-level variables); (2) constructing design matrices and contrasts that support the new BIDS-StatsModel specification (for machine-readable representation of fMRI statistical models); and (3) automated generation of partial Methods sections for inclusion in publications.PyBIDS can be easily installed on all platforms via pip (pip install pybids), though currently it is not officially supported on Windows. The package has few dependencies outside of standard Python numerical and image analysis libraries (i.e., numpy, scipy, pandas, and NiBabel). The core API is deliberately kept minimalistic: nearly all interactions with PyBIDS functionality occur through a core BIDSLayout object initialized by passing in a path to a BIDS dataset. For most applications, no custom configuration should be required.Although technically still in alpha release, PyBIDS is already being used both as a dependency in dozens of other open-source brain imaging packages –e.g., fMRIPrep (Esteban et al.,2019), MRIQC (Esteban et al., 2017), datalad-neuroimaging (https://github.com/datalad/datalad-neuroimaging), and fitlins (https://github.com/poldracklab/fitlins) – and directly in many researchers’ custom Python workflows. Development is extremely active, with bug fixes and new features continually being added (https://github.com/bids-standard/pybids), and major releases occurring approximately every 6 months. As of this writing, 29 people have contributed code to PyBIDS, and many more have provided feedback and testing. The API is relatively stable, and documentation and testing standards follow established norms for open-source scientific software. We encourage members of the brain imaging community currently working in Python to try using PyBIDS, and welcome new contributions.

Published
2019
Full Text
View/download PDF

8. A performance comparison of Dask and Apache Spark for data-intensive neuroimaging pipelines

Author

Mathieu Dugré, Valerie Hayot-Sasson, and Tristan Glatard

Subjects
FOS: Computer and information sciences, 0303 health sciences, Computer Science - Performance, Computer science, business.industry, Big data, Cloud computing, Python (programming language), computer.software_genre, Performance (cs.PF), Pipeline transport, Data sharing, 03 medical and health sciences, 0302 clinical medicine, Neuroimaging, Computer Science - Distributed, Parallel, and Cluster Computing, Computer cluster, Distributed, Parallel, and Cluster Computing (cs.DC), Data mining, business, computer, 030217 neurology & neurosurgery, 030304 developmental biology, computer.programming_language, Data transmission
Abstract

In the past few years, neuroimaging has entered the Big Data era due to the joint increase in image resolution, data sharing, and study sizes. However, no particular Big Data engines have emerged in this field, and several alternatives remain available. We compare two popular Big Data engines with Python APIs, Apache Spark and Dask, for their runtime performance in processing neuroimaging pipelines. Our evaluation uses two synthetic pipelines processing the 81GB BigBrain image, and a real pipeline processing anatomical data from more than 1,000 subjects. We benchmark these pipelines using various combinations of task durations, data sizes, and numbers of workers, deployed on an 8-node (8 cores ea.) compute cluster in Compute Canada's Arbutus cloud. We evaluate PySpark's RDD API against Dask's Bag, Delayed and Futures. Results show that despite slight differences between Spark and Dask, both engines perform comparably. However, Dask pipelines risk being limited by Python's GIL depending on task type and cluster configuration. In all cases, the major limiting factor was data transfer. While either engine is suitable for neuroimaging pipelines, more effort needs to be placed in reducing data transfer time., 10 pages, 15 figures, 1 tables. To appear in the proceeding of the 14th WORKS Workshop on Topics in Workflows in Support of Large-Scale Science, 17 November 2019, Denver, CO, USA

Published
2019

9. Brainhack: Developing a culture of open, inclusive, community-driven neuroscience

Author

Selim Melvin Atay, Iska Moxon-Emre, Anders Eklund, Paula P. Brooks, Nandita Vijayakumar, Anibal Sólon Heinsfeld, Jean-Baptiste Poline, Yaroslav O. Halchenko, David Meunier, Daniela M. Hohmann, Martin Szinte, Arshitha Basavaraj, Andrija Štajduhar, Link Tejavibulya, Michael Dayan, Anisha Keshavan, Chao Jiang, Felix Hoffstaedter, Michael P. Milham, Davide Momi, Hua Xie, Ai Wern Chung, Georg Langs, Hao-Ting Wang, Xenia Kobeleva, Robert Oostenveld, Thomas G. Close, Lena Dorfschmidt, Jesse A. Brown, Serge Koudoro, J.P. Manzano-Patron, Isil P. Bilgin, Shreyas Fadnavis, Sylvain Takerkart, R. Cameron Craddock, Stephan Heunis, Scott Peltier, Kathryn Berluti, Hayli Spence, Pablo F. Damasceno, David H. O’Connor, Eleftherios Garyfallidis, Eugene P. Duff, Rudolph Pienaar, Nicolas Traut, AmanPreet Badhwar, Ali R. Khan, Marissa Laws, Abigail S. Greene, Richard A. I. Bethlehem, Angela R. Laird, Krista DeStasio, Ina Thome, Camille Maumet, Alexandru D. Iordan, P. Christiaan Klink, Damion V. Demeter, John A. Onofrey, Cassandra D. Gould van Praag, Jakub Vohryzek, Suzanne T. Witt, Aysha Motala, Valerie Hayot-Sasson, Geetika Gupta, Alexandre Rosa Franco, John W. VanMeter, Michel Thiebaut de Schotten, James M. Shine, Lucina Q. Uddin, Thomas S. Hartmann, Guillaume Flandin, Clara Moreau, Tomislav Lipic, Claire Bradley, Fernando A. Barrios, Daniel S. Margulies, R. Austin Benn, Sofie Van Den Bossche, Lydia Riedl, Xihe Xie, Christopher R. Madan, Roberto Toro, Viviana Siless, Fabrizio De Vico Fallani, Nikoloz Sirmpilatze, Emily Olafson, Anqi Qiu, Theresa W. Cheng, Valentina Borghesani, Sidhant Chopra, Claire Cury, Giorgio Marinato, Horea-Ioan Ioanas, Giorgia Cona, Michael Joseph, Angela Tam, Mathias Scharinger, Daniel A. Handwerker, Katherine L. Bottenhorn, Sina Mansour L, Kathryn L. Mills, Christopher J. Markiewicz, Elizabeth Levitis, Cyril Pernet, Stephanie J. Forkel, Agah Karakuzu, Edwina R Orchard, Sarah L. Dziura, Saige Rutherford, Kamalaker Dadi, Enrico Glerean, Desiree Lussier, Davide Poggiali, Molly Simmonite, Jason Kai, Jessica Flannery, Ting Xu, Jon Haitz Legarreta, Nasim Anousheh, Marco Bedini, Tristan Glatard, Thomas E. Nichols, Joscelin Rocha-Hidalgo, Erin W. Dickie, Dipankar Bachar, Malin Sandström, Xi-Nian Zuo, Pedro Pinheiro-Chagas, Laura C. Rice, Jakša Vukojević, Javier Gonzalez-Castillo, José C. García Alanis, Lorenzo Pasquini, Yu-Fang Yang, Matteo Mancini, Deena Shariq, Chao-Gan Yan, Laura Tomaz da Silva, César Caballero-Gaudes, Yasmine Bassil, Aurina Arnatkeviciute, Dustin Scheinost, Satrajit S. Ghosh, Gaël Varoquaux, Etienne Combrisson, Bramsh Qamar Chandio, Kelly Garner, Tiago Quendera, Patrick Friedrich, Shawn A. Rhoads, Roxane Licandro, Elizabeth DuPre, Aki Nikolaidis, Simon Schwab, Stephanie Noble, Guillaume Auzias, Daniel J. Lurie, Mahboobeh Parsapoor, Eneko Uruñuela, Andrew Doyle, Peer Herholz, Saampras Ganesan, Vincent Koppelmans, Corey Horien, Samir Das, Junaid S. Merchant, Siyuan Gao, Matheus Marcon, Nathalia Bianchini Esper, B.T. Thomas Yeo, Katja Heuer, Caroline O’Brien, Micaela Y. Chan, Sook-Lei Liew, Lindsay D. Oliver, Kirstie Whitaker, Christoph Vogelbacher, Dylan M. Nielson, Krisanne Litinas, Dorien C. Huijser, Pierre Bellec, R. Todd Constable, David N. Kennedy, Julia Sprenger, Lea-Theresa Mais, Oscar Esteban, Patrick J. Park, Patrick Callahan, Christopher R. Nolan, Johanna Bayer, Guillaume Dumas, Elise Bannier, Elizabeth A. McDevitt, Ariel Rokem, Samuel A. Nastase, Olivia W. Stanley, Ruggero Basanisi, Daniele Marinazzo, Gregory Kiar, Lisa Novello, Samuel Guay, John C. Flournoy, Stefano Moia, Kendra Oudyk, Fang-Cheng Yeh, Gustav Nilsonne, Thomas B. Shaw, Steve Wideman, Saskia Bollmann, Steffen Bollmann, Julia M. Huntenburg, Augusto Buchweitz, Matteo Visconti di Oleggio Castello, Dimitra Maoutsa, Lucy B. Whitmore, Catherine Alice Hahn, Antonino Vallesi, Remi Gau, Felipe Meneguzzi, Université Catholique de Louvain = Catholic University of Louvain (UCL), Yale University [New Haven], Max Planck Institute for Human Cognitive and Brain Sciences [Leipzig] (IMPNSC), Max-Planck-Gesellschaft, Florida International University [Miami] (FIU), University of Reading (UOR), University of Würzburg, Champalimaud Centre for the Unknown [Lisbon], University of Melbourne, University of Cambridge [UK] (CAM), Georgetown University [Washington] (GU), Philipps Universität Marburg = Philipps University of Marburg, Université de Montréal (UdeM), University College of London [London] (UCL), University of Sussex, Universiteit Gent = Ghent University (UGENT), German Research Center for Neurodegenerative Diseases - Deutsches Zentrum für Neurodegenerative Erkrankungen (DZNE), Université de Sherbrooke (UdeS), Middle East Technical University [Ankara] (METU), McGill University = Université McGill [Montréal, Canada], Modelling brain structure, function and variability based on high-field MRI data (PARIETAL), Service NEUROSPIN (NEUROSPIN), Université Paris-Saclay-Direction de Recherche Fondamentale (CEA) (DRF (CEA)), Commissariat à l'énergie atomique et aux énergies alternatives (CEA)-Commissariat à l'énergie atomique et aux énergies alternatives (CEA)-Université Paris-Saclay-Direction de Recherche Fondamentale (CEA) (DRF (CEA)), Commissariat à l'énergie atomique et aux énergies alternatives (CEA)-Commissariat à l'énergie atomique et aux énergies alternatives (CEA)-Inria Saclay - Ile de France, Institut National de Recherche en Informatique et en Automatique (Inria)-Institut National de Recherche en Informatique et en Automatique (Inria), Universiteit Leiden, Karolinska Institutet [Stockholm], Princeton University, Universität Zürich [Zürich] = University of Zurich (UZH), University of the Basque Country/Euskal Herriko Unibertsitatea (UPV/EHU), University medical center and campus biotech Geneva, Emory University [Atlanta, GA], Cardiff University's Brain Research Imaging Centre [Cardiff] (CUBRIC), School of Psychology [Cardiff University], Cardiff University-Cardiff University, The University of Sydney, Weill Cornell Medicine [Cornell University], Cornell University [New York], Università degli Studi di Padova = University of Padua (Unipd), Jülich Research Centre, University of Texas at Austin [Austin], Institut Pasteur [Paris] (IP), Centre de Recherche Interdisciplinaire / Center for Research and Interdisciplinarity [Paris, France] (CRI), Institut National de la Santé et de la Recherche Médicale (INSERM)-Université Paris Cité (UPCité), Basque Center on Cognition Brain and Language [Gipuzkoa, Espagne] (BCBL), Linköping University (LIU), University of Queensland [Brisbane], University of New South Wales [Sydney] (UNSW), Universidad Nacional Autónoma de México = National Autonomous University of Mexico (UNAM), University of Maryland [College Park], University of Maryland System, University of Washington [Seattle], Harvard Medical School [Boston] (HMS), Child Mind Institute, University of Western Ontario (UWO), Indiana University [Bloomington], Indiana University System, Monash university, Institut de Neurosciences de la Timone (INT), Aix Marseille Université (AMU)-Centre National de la Recherche Scientifique (CNRS), Neuroimagerie: méthodes et applications (Empenn), Institut National de la Santé et de la Recherche Médicale (INSERM)-Inria Rennes – Bretagne Atlantique, Institut National de Recherche en Informatique et en Automatique (Inria)-Institut National de Recherche en Informatique et en Automatique (Inria)-SIGNAUX ET IMAGES NUMÉRIQUES, ROBOTIQUE (IRISA-D5), Institut de Recherche en Informatique et Systèmes Aléatoires (IRISA), Université de Rennes (UR)-Institut National des Sciences Appliquées - Rennes (INSA Rennes), Institut National des Sciences Appliquées (INSA)-Institut National des Sciences Appliquées (INSA)-Université de Bretagne Sud (UBS)-École normale supérieure - Rennes (ENS Rennes)-Institut National de Recherche en Informatique et en Automatique (Inria)-CentraleSupélec-Centre National de la Recherche Scientifique (CNRS)-IMT Atlantique (IMT Atlantique), Institut Mines-Télécom [Paris] (IMT)-Institut Mines-Télécom [Paris] (IMT)-Université de Rennes (UR)-Institut National des Sciences Appliquées - Rennes (INSA Rennes), Institut Mines-Télécom [Paris] (IMT)-Institut Mines-Télécom [Paris] (IMT)-Institut de Recherche en Informatique et Systèmes Aléatoires (IRISA), Institut National des Sciences Appliquées (INSA)-Institut National des Sciences Appliquées (INSA)-Université de Bretagne Sud (UBS)-École normale supérieure - Rennes (ENS Rennes)-CentraleSupélec-Centre National de la Recherche Scientifique (CNRS)-IMT Atlantique (IMT Atlantique), Institut Mines-Télécom [Paris] (IMT)-Institut Mines-Télécom [Paris] (IMT), National Institute of Mental Health (NIMH), Università degli Studi di Trento (UNITN), Centro Nacional de Investigaciones Cardiovasculares Carlos III [Madrid, Spain] (CNIC), Instituto de Salud Carlos III [Madrid] (ISC), Queensland University of Technology [Brisbane] (QUT), University of California [San Francisco] (UC San Francisco), University of California (UC), Universidade Federal do Rio Grande do Sul [Porto Alegre] (UFRGS), University of Texas at Dallas [Richardson] (UT Dallas), University of Oregon [Eugene], Monash University [Melbourne], Boston Children's Hospital, Algorithms, models and methods for images and signals of the human brain (ARAMIS), Sorbonne Université (SU)-Inria de Paris, Institut National de Recherche en Informatique et en Automatique (Inria)-Institut National de Recherche en Informatique et en Automatique (Inria)-Institut du Cerveau = Paris Brain Institute (ICM), Assistance publique - Hôpitaux de Paris (AP-HP) (AP-HP)-Institut National de la Santé et de la Recherche Médicale (INSERM)-CHU Pitié-Salpêtrière [AP-HP], Assistance publique - Hôpitaux de Paris (AP-HP) (AP-HP)-Sorbonne Université (SU)-Sorbonne Université (SU)-Sorbonne Université (SU)-Centre National de la Recherche Scientifique (CNRS)-Assistance publique - Hôpitaux de Paris (AP-HP) (AP-HP)-Institut National de la Santé et de la Recherche Médicale (INSERM)-CHU Pitié-Salpêtrière [AP-HP], Assistance publique - Hôpitaux de Paris (AP-HP) (AP-HP)-Sorbonne Université (SU)-Sorbonne Université (SU)-Centre National de la Recherche Scientifique (CNRS), Institut du Cerveau = Paris Brain Institute (ICM), Assistance publique - Hôpitaux de Paris (AP-HP) (AP-HP)-Sorbonne Université (SU)-Sorbonne Université (SU)-Sorbonne Université (SU)-Centre National de la Recherche Scientifique (CNRS), Institut des Maladies Neurodégénératives [Bordeaux] (IMN), Université de Bordeaux (UB)-Centre National de la Recherche Scientifique (CNRS), Neuroimagerie: méthodes et applications (EMPENN), Institut National de Recherche en Informatique et en Automatique (Inria)-Institut National de Recherche en Informatique et en Automatique (Inria)-SIGNAL, IMAGE ET LANGAGE (IRISA-D6), Chinese Academy of Sciences [Beijing] (CAS), Beijing Normal University (BNU), This project has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement no. 867458 awarded to Julia M. Huntenburg, from ANR-19-DATA-0025 NeuroWebLab for Katja Heuer, Roberto Toro, and Nicholas Traut, and from NIH K00MH122372 for Stephanie Noble., Brainhack Community: Nasim Anousheh, Aurina Arnatkeviciute, Guillaume Auzias, Dipankar Bachar, Elise Bannier, Ruggero Basanisi, Arshitha Basavaraj, Marco Bedini, Pierre Bellec, R Austin Benn, Kathryn Berluti, Steffen Bollmann, Saskia Bollmann, Claire Bradley, Jesse Brown, Augusto Buchweitz, Patrick Callahan, Micaela Y Chan, Bramsh Q Chandio, Theresa Cheng, Sidhant Chopra, Ai Wern Chung, Thomas G Close, Etienne Combrisson, Giorgia Cona, R Todd Constable, Claire Cury, Kamalaker Dadi, Pablo F Damasceno, Samir Das, Fabrizio De Vico Fallani, Krista DeStasio, Erin W Dickie, Lena Dorfschmidt, Eugene P Duff, Elizabeth DuPre, Sarah Dziura, Nathalia B Esper, Oscar Esteban, Shreyas Fadnavis, Guillaume Flandin, Jessica E Flannery, John Flournoy, Stephanie J Forkel, Alexandre R Franco, Saampras Ganesan, Siyuan Gao, José C García Alanis, Eleftherios Garyfallidis, Tristan Glatard, Enrico Glerean, Javier Gonzalez-Castillo, Cassandra D Gould van Praag, Abigail S Greene, Geetika Gupta, Catherine Alice Hahn, Yaroslav O Halchenko, Daniel Handwerker, Thomas S Hartmann, Valérie Hayot-Sasson, Stephan Heunis, Felix Hoffstaedter, Daniela M Hohmann, Corey Horien, Horea-Ioan Ioanas, Alexandru Iordan, Chao Jiang, Michael Joseph, Jason Kai, Agah Karakuzu, David N Kennedy, Anisha Keshavan, Ali R Khan, Gregory Kiar, P Christiaan Klink, Vincent Koppelmans, Serge Koudoro, Angela R Laird, Georg Langs, Marissa Laws, Roxane Licandro, Sook-Lei Liew, Tomislav Lipic, Krisanne Litinas, Daniel J Lurie, Désirée Lussier, Christopher R Madan, Lea-Theresa Mais, Sina Mansour L, J P Manzano-Patron, Dimitra Maoutsa, Matheus Marcon, Daniel S Margulies, Giorgio Marinato, Daniele Marinazzo, Christopher J Markiewicz, Camille Maumet, Felipe Meneguzzi, David Meunier, Michael P Milham, Kathryn L Mills, Davide Momi, Clara A Moreau, Aysha Motala, Iska Moxon-Emre, Thomas E Nichols, Dylan M Nielson, Gustav Nilsonne, Lisa Novello, Caroline O'Brien, Emily Olafson, Lindsay D Oliver, John A Onofrey, Edwina R Orchard, Kendra Oudyk, Patrick J Park, Mahboobeh Parsapoor, Lorenzo Pasquini, Scott Peltier, Cyril R Pernet, Rudolph Pienaar, Pedro Pinheiro-Chagas, Jean-Baptiste Poline, Anqi Qiu, Tiago Quendera, Laura C Rice, Joscelin Rocha-Hidalgo, Saige Rutherford, Mathias Scharinger, Dustin Scheinost, Deena Shariq, Thomas B Shaw, Viviana Siless, Molly Simmonite, Nikoloz Sirmpilatze, Hayli Spence, Julia Sprenger, Andrija Stajduhar, Martin Szinte, Sylvain Takerkart, Angela Tam, Link Tejavibulya, Michel Thiebaut de Schotten, Ina Thome, Laura Tomaz da Silva, Nicolas Traut, Lucina Q Uddin, Antonino Vallesi, John W VanMeter, Nandita Vijayakumar, Matteo Visconti di Oleggio Castello, Jakub Vohryzek, Jakša Vukojević, Kirstie Jane Whitaker, Lucy Whitmore, Steve Wideman, Suzanne T Witt, Hua Xie, Ting Xu, Chao-Gan Yan, Fang-Cheng Yeh, B T Thomas Yeo, Xi-Nian Zuo, ANR-19-DATA-0025,NeuroWebLab,Un laboratoire de neuroscience collectif: Au delà de FAIR(2019), European Project: 867458,LC-FMR, Maumet, Camille, Un laboratoire de neuroscience collectif: Au delà de FAIR - - NeuroWebLab2019 - ANR-19-DATA-0025 - DONNEES - VALID, Brainhack: Developing a culture of open, inclusive, community-driven neuroscience - LC-FMR - 867458 - INCOMING, Leiden University, Weill Cornell Medicine [New York], Institut National de la Santé et de la Recherche Médicale (INSERM)-Université Paris Cité (UPC), Universiteit Gent = Ghent University [Belgium] (UGENT), University Hospital Bonn, Sherbrooke Connectivity Imaging Lab [Sherbrooke] (SCIL), Département d'informatique [Sherbrooke] (UdeS), Faculté des sciences [Sherbrooke] (UdeS), Université de Sherbrooke (UdeS)-Université de Sherbrooke (UdeS)-Faculté des sciences [Sherbrooke] (UdeS), Université de Sherbrooke (UdeS)-Université de Sherbrooke (UdeS), Videos and Images Theory and Analytics Laboratory (VITAL), Inria Saclay - Ile de France, Institut National de Recherche en Informatique et en Automatique (Inria)-Institut National de Recherche en Informatique et en Automatique (Inria)-Service NEUROSPIN (NEUROSPIN), Commissariat à l'énergie atomique et aux énergies alternatives (CEA)-Commissariat à l'énergie atomique et aux énergies alternatives (CEA), Erasmus University Rotterdam, Origami (Origami), Laboratoire d'InfoRmatique en Image et Systèmes d'information (LIRIS), Institut National des Sciences Appliquées de Lyon (INSA Lyon), Université de Lyon-Institut National des Sciences Appliquées (INSA)-Université de Lyon-Institut National des Sciences Appliquées (INSA)-Centre National de la Recherche Scientifique (CNRS)-Université Claude Bernard Lyon 1 (UCBL), Université de Lyon-École Centrale de Lyon (ECL), Université de Lyon-Université Lumière - Lyon 2 (UL2)-Institut National des Sciences Appliquées de Lyon (INSA Lyon), Université de Lyon-Université Lumière - Lyon 2 (UL2), Princeton Neuroscience Institute [Princeton], Human Neuroscience Platform, Brighton and Sussex Medical School (BSMS), Azienda Ospedale Università di Padova = Hospital-University of Padua (AOUP), Institute of Neuroscience and Medicine [Jülich] (INM-1), Département de Neuroscience - Department of Neuroscience, Institut Pasteur [Paris]-Centre National de la Recherche Scientifique (CNRS)-Université Paris Cité (UPC), Queensland Brain Institute, Universidad Nacional Autónoma de México (UNAM), Radboud university [Nijmegen], McGovern Institute for Brain Research [Cambridge], Massachusetts Institute of Technology (MIT), The Brainhack Community - https://pubmed.ncbi.nlm.nih.gov/33932337, Philipps University of Marburg, University of the Basque Country [Bizkaia] (UPV/EHU), Universita degli Studi di Padova, Philipps Universität Marburg, Institut Pasteur [Paris], Empenn, Université de Bretagne Sud (UBS)-Institut National des Sciences Appliquées - Rennes (INSA Rennes), Institut National des Sciences Appliquées (INSA)-Université de Rennes (UNIV-RENNES)-Institut National des Sciences Appliquées (INSA)-Université de Rennes (UNIV-RENNES)-Institut National de Recherche en Informatique et en Automatique (Inria)-École normale supérieure - Rennes (ENS Rennes)-Centre National de la Recherche Scientifique (CNRS)-Université de Rennes 1 (UR1), Université de Rennes (UNIV-RENNES)-CentraleSupélec-IMT Atlantique Bretagne-Pays de la Loire (IMT Atlantique), Institut Mines-Télécom [Paris] (IMT)-Institut Mines-Télécom [Paris] (IMT)-Université de Bretagne Sud (UBS)-Institut National des Sciences Appliquées - Rennes (INSA Rennes), Institut National des Sciences Appliquées (INSA)-Université de Rennes (UNIV-RENNES)-Institut National des Sciences Appliquées (INSA)-Université de Rennes (UNIV-RENNES)-École normale supérieure - Rennes (ENS Rennes)-Centre National de la Recherche Scientifique (CNRS)-Université de Rennes 1 (UR1), University of California [San Francisco] (UCSF), University of California, Geochemistry, and University of Padova [Padova, Italy]

Subjects
0301 basic medicine, Open science, Community building, Best practice, Neuroscience(all), Brainhack, Article, neuroscience, 03 medical and health sciences, 0302 clinical medicine, best practices, collaboration, community building, hackathon, inclusivity, open science, reproducibility, training, Congresses as Topic, Neurosciences, Practice Guidelines as Topic, Communication, Internet, Research community, Medical Bioscience, Psychology, ddc:610, Sociology, Brainhack Community, Training, Neurology & Neurosurgery, Scientific progress, organization & administration [Neurosciences], General Neuroscience, [SCCO.NEUR]Cognitive science/Neuroscience, [SCCO.NEUR] Cognitive science/Neuroscience, Medicinsk biovetenskap, 030104 developmental biology, [SDV.IB]Life Sciences [q-bio]/Bioengineering, Cognitive Sciences, Engineering ethics, 030217 neurology & neurosurgery
Abstract

Available online 30 April 2021. Brainhack is an innovative meeting format that promotes scientific collaboration and education in an open, inclusive environment. This NeuroView describes the myriad benefits for participants and the research community and how Brainhacks complement conventional formats to augment scientific progress. The present manuscript is part of a growing community effort to collate Brainhack-related insights and expertise into a Jupyter Book (http://brainhack.org/brainhack_jupyter_book/) that will serve as a centralized set of resources for the community; we acknowledge all the individuals who contributed and will make ongoing contributions to these resources. A pre-print version of the present manuscript is available as part of the Jupyter Book. Moreover, we would like to acknowledge all Brainhack organizers, supporters, presenters, and participants for their contribution to growing and maintaining this community. The benefits described in this manuscript would not be possible without them. We also thank all institutions, labs, and organizations who have helped this community grow, meet in stimulating environments, and add an excellent educational resource pool and agenda. With an expanding community, Brainhack’s support network keeps growing, and we thank all labs and individual researchers for their dedication and expertise offered to this community (see http://brainhack.org/brainhack_jupyter_book/acknowledgments.html for a full list of individual acknowledgments; an updated list will be maintained in the Jupyter Book). Grants and funding bodies: This project has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement no. 867458 awarded to Julia M. Huntenburg; from ANR-19-DATA-0025 NeuroWebLab for Katja Heuer, Roberto Toro, and Nicholas Traut; and from NIH K00MH122372 for Stephanie Noble. The Brainhack Community member list and contributions of the different authors are detailed at http://brainhack.org/brainhack_jupyter_book/contributors.html. Our crediting system is described here: http://brainhack.org/brainhack_jupyter_book/neuroview_authorship-agreement.html.

Published
2021
Full Text
View/download PDF

10. Evaluation of pilot jobs for Apache Spark applications on HPC clusters

Author

Tristan Glatard and Valerie Hayot-Sasson

Subjects
Job scheduler, FOS: Computer and information sciences, Queueing theory, Computer Science - Performance, business.industry, Computer science, media_common.quotation_subject, Big data, Supercomputer, computer.software_genre, Scheduling (computing), Performance (cs.PF), Debugging, Computer Science - Distributed, Parallel, and Cluster Computing, Software deployment, Operating system, Distributed, Parallel, and Cluster Computing (cs.DC), business, computer, media_common
Abstract

Big Data has become prominent throughout many scientific fields, and as a result, scientific communities have sought out Big Data frameworks to accelerate the processing of their increasingly data-intensive pipelines. However, while scientific communities typically rely on High-Performance Computing (HPC) clusters for the parallelization of their pipelines, many popular Big Data frameworks such as Hadoop and Apache Spark were primarily designed to be executed on dedicated commodity infrastructures. This paper evaluates the benefits of pilot jobs over traditional batch submission for Apache Spark on HPC clusters. Surprisingly, our results show that the speed-up provided by pilot jobs over batch scheduling is moderate to non-existent (0.98 on average) despite the presence of long queuing times. In addition, pilot jobs provide an extra layer of scheduling that complicates debugging and deployment. We conclude that traditional batch scheduling should remain the default strategy to deploy Apache Spark applications on HPC clusters.

Published
2019
Full Text
View/download PDF

11. Performance Evaluation of Big Data Processing Strategies for Neuroimaging

Author

Shawn T. Brown, Valerie Hayot-Sasson, and Tristan Glatard

Subjects
FOS: Computer and information sciences, 0303 health sciences, Computer science, business.industry, Big data, Locality, Image processing, Machine learning, computer.software_genre, Data sharing, 03 medical and health sciences, 0302 clinical medicine, Computer Science - Distributed, Parallel, and Cluster Computing, In-Memory Processing, Page cache, Distributed, Parallel, and Cluster Computing (cs.DC), Cache, Artificial intelligence, Lazy evaluation, business, computer, 030217 neurology & neurosurgery, 030304 developmental biology
Abstract

Neuroimaging datasets are rapidly growing in size as a result of advancements in image acquisition methods, open-science and data sharing. However, the adoption of Big Data processing strategies by neuroimaging processing engines remains limited. Here, we evaluate three Big Data processing strategies (in-memory computing, data locality and lazy evaluation) on typical neuroimaging use cases, represented by the BigBrain dataset. We contrast these various strategies using Apache Spark and Nipype as our representative Big Data and neuroimaging processing engines, on Dell EMC's Top-500 cluster. Big Data thresholds were modelled by comparing the data-write rate of the application to the filesystem bandwidth and number of concurrent processes. This model acknowledges the fact that page caching provided by the Linux kernel is critical to the performance of Big Data applications. Results show that in-memory computing alone speeds-up executions by a factor of up to 1.6, whereas when combined with data locality, this factor reaches 5.3. Lazy evaluation strategies were found to increase the likelihood of cache hits, further improving processing time. Such important speed-up values are likely to be observed on typical image processing operations performed on images of size larger than 75GB. A ballpark speculation from our model showed that in-memory computing alone will not speed-up current functional MRI analyses unless coupled with data locality and processing around 280 subjects concurrently. Furthermore, we observe that emulating in-memory computing using in-memory file systems (tmpfs) does not reach the performance of an in-memory engine, presumably due to swapping to disk and the lack of data cleanup. We conclude that Big Data processing strategies are worth developing for neuroimaging applications.

Published
2018

12. Boutiques: a flexible framework to integrate command-line applications in computing platforms

Author

Natacha Beck, M. Rousseau, Rafael Ferreira da Silva, Valerie Hayot-Sasson, Guillaume Flandin, Alan C. Evans, Shawn T. Brown, Axel Bonnet, Tristan Aumentado-Armstrong, Charles R.G. Guttmann, Krzysztof J. Gorgolewski, Pierre Rioux, Pierre-Olivier Quirion, Samir Das, Tristan Glatard, Sorina Camarasu-Pop, Gregory Kiar, Frederic Cervenansky, Pierre Bellec, Pascal Girard, Rémi Bernard, Department of Computer Science and Software Engineering [Montreal] (CSE), Concordia University [Montreal], McConnell Brain Imaging Centre (MNI), Montreal Neurological Institute and Hospital, McGill University = Université McGill [Montréal, Canada]-McGill University = Université McGill [Montréal, Canada], Département d'Informatique et de Recherche Opérationnelle [Montreal] (DIRO), Université de Montréal (UdeM), Aix-Marseille Université - Département Sciences, Arts et Techniques des Images et du Son (AMU SATIS), Aix Marseille Université (AMU), Images et Modèles, Centre de Recherche en Acquisition et Traitement de l'Image pour la Santé (CREATIS), Université Jean Monnet [Saint-Étienne] (UJM)-Hospices Civils de Lyon (HCL)-Institut National des Sciences Appliquées de Lyon (INSA Lyon), Université de Lyon-Institut National des Sciences Appliquées (INSA)-Université de Lyon-Institut National des Sciences Appliquées (INSA)-Université Claude Bernard Lyon 1 (UCBL), Université de Lyon-Centre National de la Recherche Scientifique (CNRS)-Institut National de la Santé et de la Recherche Médicale (INSERM)-Université Jean Monnet [Saint-Étienne] (UJM)-Hospices Civils de Lyon (HCL)-Institut National des Sciences Appliquées de Lyon (INSA Lyon), Université de Lyon-Centre National de la Recherche Scientifique (CNRS)-Institut National de la Santé et de la Recherche Médicale (INSERM), Service Informatique et développements, Department of Developmental, Molecular and Chemical Biology, Tufts University School of Medicine [Boston], Wellcome Trust Centre for Neuroimaging, University College of London [London] (UCL), Laboratoire de l'intégration, du matériau au système (IMS), Université Sciences et Technologies - Bordeaux 1-Institut Polytechnique de Bordeaux-Centre National de la Recherche Scientifique (CNRS), Stanford University, Brigham and Women's Hospital [Boston], Université Claude Bernard Lyon 1 (UCBL), Université de Lyon-Université de Lyon-Institut National des Sciences Appliquées de Lyon (INSA Lyon), Université de Lyon-Institut National des Sciences Appliquées (INSA)-Institut National des Sciences Appliquées (INSA)-Université Jean Monnet - Saint-Étienne (UJM)-Institut National de la Santé et de la Recherche Médicale (INSERM)-Centre National de la Recherche Scientifique (CNRS)-Université Claude Bernard Lyon 1 (UCBL), Université de Lyon-Institut National des Sciences Appliquées (INSA)-Institut National des Sciences Appliquées (INSA)-Université Jean Monnet - Saint-Étienne (UJM)-Institut National de la Santé et de la Recherche Médicale (INSERM)-Centre National de la Recherche Scientifique (CNRS), Université Sciences et Technologies - Bordeaux 1 (UB)-Institut Polytechnique de Bordeaux-Centre National de la Recherche Scientifique (CNRS), and Centre National de la Recherche Scientifique (CNRS)-Institut Polytechnique de Bordeaux-Université Sciences et Technologies - Bordeaux 1

Subjects
0301 basic medicine, Computer science, Neuroimaging, Health Informatics, 03 medical and health sciences, 0302 clinical medicine, Software, [INFO.INFO-IM]Computer Science [cs]/Medical Imaging, Technical Note, Redundancy (engineering), Humans, Publication, ComputingMilieux_MISCELLANEOUS, computer.programming_language, business.industry, application integration, Brain, Computational Biology, Reproducibility of Results, Neuroinformatics, neuroinformatics, JSON, Computer Science Applications, 030104 developmental biology, containers, [INFO.INFO-ET]Computer Science [cs]/Emerging Technologies [cs.ET], Software engineering, business, computer, 030217 neurology & neurosurgery
Abstract

We present Boutiques, a system to automatically publish, integrate, and execute command-line applications across computational platforms. Boutiques applications are installed through software containers described in a rich and flexible JSON language. A set of core tools facilitates the construction, validation, import, execution, and publishing of applications. Boutiques is currently supported by several distinct virtual research platforms, and it has been used to describe dozens of applications in the neuroinformatics domain. We expect Boutiques to improve the quality of application integration in computational platforms, to reduce redundancy of effort, to contribute to computational reproducibility, and to foster Open Science.

Published
2018
Full Text
View/download PDF

13. Sequential algorithms to split and merge ultra-high resolution 3D images

Author

Tristan Glatard, Yuhong Yan, Yongping Gao, and Valerie Hayot-Sasson

Subjects
0301 basic medicine, Computer science, 02 engineering and technology, Iterative reconstruction, Hard disk drive performance characteristics, Ultra high resolution, 03 medical and health sciences, 030104 developmental biology, Memory management, 0202 electrical engineering, electronic engineering, information engineering, 020201 artificial intelligence & image processing, Cluster analysis, Image resolution, Merge (version control), Algorithm
Abstract

Splitting and merging data is a requirement for many parallel or distributed processing operations. Naive algorithms to split and merge 3D blocks from ultra-high resolution images perform very poorly, as a result of seek times. In contrast, naive algorithms to split and merge 3D slabs perform optimally as seek time is significantly minimized. We introduce and analyze sequential algorithms (Clustered reads, Multiple reads, Clustered writes, and Multiple writes) that leverage memory buffering to address this issue. Clustered reads and Clustered writes, access image chunks only once, but they have to seek in the reconstructed image. Multiple reads and Multiple writes minimize seeks in the reconstructed image, but they access image chunks multiple times. Evaluation on a 3850×3025×3500 brain image shows that our algorithms perform similarly to the optimal configuration provided that enough memory is available. Additionally, Multiple reads supports on-the-fly compression of the merged image transparently but Clustered reads does not, due to its use of negative seeking. We conclude that splitting and merging large 3D images can be done efficiently without relying on complex data formats.

Published
2017
Full Text
View/download PDF

Catalog

Books, media, physical & digital resources
See catalog results