Agrawal, Ankur, Chia-Yu Chen, Jungwook Choi, Gopalakrishnan, Kailash, Jinwook Oh, Shukla, Sunil, Srinivasan, Viji, Venkataramani, Swagath, and Wei Zhang
Deep Neural Networks (DNNs) have emerged as a powerful and versatile set of techniques showing successes on challenging artificial intelligence (AI) problems. Applications in domains such as image/video processing, autonomous cars, natural language processing, speech synthesis and recognition, genomics and many others have embraced deep learning as the foundation. DNNs achieve superior accuracy for these applications with high computational complexity using very large models which require 100s of MBs of data storage, exaops of computation and high bandwidth for data movement. In spite of these impressive advances, it still takes days to weeks to train state of the art Deep Networks on large datasets - which directly limits the pace of innovation and adoption. In this paper, we present a multi-pronged approach to address the challenges in meeting both the throughput and the energy efficiency goals for DNN training. Although computationally intensive, DNNs are resilient to loss of numerical precision. Algorithm-level noise-tolerance of deep learning training may be leveraged to relax accuracy constraints in application specific hardware, resulting in systems that achieve better area- and power-efficiency than traditionally designed highly-accurate systems. As a first step, we explored the use of lowprecision fixed-point arithmetic for DNN training. Migrating from floating-point arithmetic to fixed-point arithmetic reduces the area, power and delay of compute units, resulting in gains in throughputper- unit-area and throughput-per-unit-power metrics. Representing numbers in low-precision also reduces memory-footprint and communication bandwidth requirements, allowing for training of larger models for a given on-chip memory and I/O budget. The key finding of our exploration is that deep neural networks can be trained using low-precision fixed-point arithmetic, provided thatthe stochastic rounding scheme is applied while operating on these numbers [1]. Our training experiments with deep neural networks for MNIST and CIFAR10 image classification tasks demonstrate that networks trained using 16-bit wide fixed-point and stochastic rounding achieve nearly the same performance as that obtained when trained using 32-bit floating-point computations. We also experimented with reduced precision floating point numbers as a way to further reduce bit-width requirements for number representations, since floating-point numbers span a much greater dynamic range than fixed-point numbers for a given total bit-width. We experimented with 16-, 12-, and 10-bit representation for floating-point numbers, where we fixed the exponent to be 6 bits, and used 9, 5, and 3 bits, respectively, to represent the mantissa (along with 1 sign bit) [2]. We observed little to no degradation in the training error (cross-entropy) or test error when we reduce the precision to 12b, when compared to our baseline training experiments with 32-bit computations for CIFAR10 and MNIST benchmarks. Since the logic complexity of floating point units (FPUs) is roughly proportional to the square of number of bits, we expect 12-b FPUs to be 7x-8x smaller and more power efficient than 32-b FPUs. Such large gains at the buildingblock level are critical to the success of hardware accelerators over general-purpose CPUs and GPUs. In order to exploit maximum parallelism, DNN training tasks are typically distributed across multiple accelerator nodes (or learners) [3] - by either partitioning the training minibatch data across multiple accelerator nodes (data parallelism) or by distributing the DNN model across multiple nodes (model parallelism). Although DNN training benefits from using multiple learners in parallel, traditional stochastic gradient descent techniques require the parameter server to wait for an update from each of the learning nodes. Using an in-house DNN framework (Rudra), the results of applying variations of the asynchronous stochastic gradient descent algorithm on model training time and accuracy is presented. This work demonstrated the need to reduce the mini-batch size per learner in proportion to the number of learners in order to maintain model accuracy in large-scale data-parallel DNN training tasks. To harness the compounded benefits of reduced precision computations and relaxed communication techniques, we explored the impact of applying multiple approximation techniques concurrently on DNN accuracy. These experiments showed that it is possible to simultaneously reduce the numerical precision (down to 12-bits) as well as apply relaxed synchronization schemes (for a 50% speedup in execution time) - without incurring any additional degradation in the quality of the DNN model Over the past few years, several prior works have demonstrated that DNN training and inference energy efficiencies can be improved significantly using specialized hardware and data-flows. However, a direct comparison between different implementations is not possible due to variations in the implementation technology, dataflow mapping, the hardware architecture and the available data storage. We set out to investigate whether it is possible to systematically explore the design space so as to estimate a given DNN's performance on any given shared memory architecture using a variety of data-flows. We propose DEEP MATRIX, a framework to systematically characterize and analyze the performance of DNNs on shared memory accelerator systems. DEEP MATRIX uses the following inputs: (i) a description of a DNN, (ii) a target hardware architecture containing multiple accelerator cores, an on-chip memory hierarchy with private and shared memories, and an on-chip interconnect network, and (iii) a dataflow specification which dictates how the compute is orchestrated within the cores, to systematically explore the design space of how compute and data can be partitioned and mapped to the different processor and memory elements, such that overall system-level performance is maximized, while meeting the constraints (on-chip storage capacity, bandwidth) imposed by the architecture [4]. Putting this all together, we provide recommendations for system design and chip microarchitectures in the Deep Learning era. Future DNN training systems will incorporate a number of key features including the ability to support large-scale (>> 8) distributed DNN training tasks running on specialized (ASIC) hardware. Dataflow accelerators that support reduced precision computations (with optimized precision and number formats) and maintain high accelerator utilizations (even while working with small minibatch sizes) look promising as the industry looks to specialize beyond GPUs for Deep Learning. [ABSTRACT FROM AUTHOR]