1. HAL: Computer System for Scalable Deep Learning
- Author
-
Sayed Hadi Hashemi, Benjamin Rabe, Roy H. Campbell, Yan Zhan, Volodymyr Kindratenko, Ke Xu, Jian Peng, Dawei Mu, William Gropp, and John Maloney
- Subjects
Software ,Computer architecture ,Software deployment ,business.industry ,Computer science ,Server ,Deep learning ,InfiniBand ,Artificial intelligence ,IBM ,business ,Supercomputer ,Enterprise software - Abstract
We describe the design, deployment and operation of a computer system built to efficiently run deep learning frameworks. The system consists of 16 IBM POWER9 servers with 4 NVIDIA V100 GPUs each, interconnected with Mellanox EDR InfiniBand fabric, and a DDN all-flash storage array. The system is tailored towards efficient execution of the IBM Watson Machine Learning enterprise software stack that combines popular open-source deep learning frameworks. We build a custom management software stack to enable an efficient use of the system by a diverse community of users and provide guides and recipes for running deep learning workloads at scale utilizing all available GPUs. We demonstrate scaling of a PyTorch and TensorFlow based deep neural networks to produce state-of-the-art performance results.
- Published
- 2020