Search

The choice of data type has a major impact on speed, accuracy, and power consumption of deep learning accelerators. Quantizing the weights and activations of neural networks to integer based computation is an industry standard for reducing memory footprint and computation cost of inference in embedded systems. Uniform weight quantization can be used for tasks where accuracy drop can be tolerated. However, the drop in accuracy due to a uniform quantization might be non-negligible especially when performed on shallow networks, complex computer vision tasks, or with lower-bit integers. In this paper, we introduce a software and a hardware solution to improve on a baseline integer based uniform quantization so that it can be run on lower power systems and with even less bits. We also introduce a novel encoding technique on top of our software solution specific for partial sums to significantly reduce memory footprint, latency and energy consumption due to movement of partial sums. The proposed SW solution exploits non-uniform piece-wise linear quantization to improve accuracy by capturing bell shaped distribution of weights while still using INT-based computation units. The proposed partial sum encoding can be applied to the partial sums regardless of uniform or non-uniform quantization. The proposed HW solution can either combine integers to make larger integers or turn them into Floating-Point operations so that various levels can have various precisions or data types, if necessary. To do so, we studied upper limits of precision we need in our compute units to support floating point inner product operations. It turns out that we can improve upon integer IPUs to perform accurate floating-point operations without introducing large shift units or wide adder trees. Our proposed SW solution (PWLQ) achieves the state-of-the-art results on all cases and it outperforms all other methods with a large margin. The proposed partial sum encoding technique effectively compresses the partial sum of networks like Resnet-50 down to 12bits (from Int64/32) without loss in accuracy. The proposed HW architecture achieves area improvements of up to 46% in TOPS/mm2 with power efficiency improvements of up to 63% in TOPS/W when compared to state of the art mixed precision implementation. [ABSTRACT FROM AUTHOR]

This paper examines the mapping of algorithms encountered when solving dense linear systems and linear least-squares problems to a custom Linear Algebra Processor. Specifically, the focus is on Cholesky, LU (with partial pivoting), and QR factorizations and their blocked algorithms. As part of the study, we expose the benefits of redesigning floating point units and their surrounding data-paths to support these complicated operations. We show how adding moderate complexity to the architecture greatly alleviates complexities in the algorithm. We study design tradeoffs and the effectiveness of architectural modifications to demonstrate that we can improve power and performance efficiency to a level that can otherwise only be expected of full-custom ASIC designs. A feasibility study of inner kernels is extended to blocked level and shows that, at block level, the Linear Algebra Core (LAC) can achieve high efficiencies with up to 45 GFLOPS/W for both Cholesky and LU factorization, and over 35 GFLOPS/W for QR factorization. While maintaining such efficiencies, our extensions to the MAC units can achieve up to 10, 12, and 20 percent speedup for the blocked algorithms of Cholesky, LU, and QR factorization, respectively. [ABSTRACT FROM AUTHOR]

As technology is reaching physical limits, reducing power consumption is a key issue on our path to sustained performance. In this paper, we study fundamental tradeoffs and limits in efficiency (as measured in energy per operation) that can be achieved for an important class of kernels, namely the level-3 Basic Linear Algebra Subprograms (BLAS). It is well-accepted that specialization is the key to efficiency. This paper establishes a baseline by studying GEneral Matrix-matrix Multiplication (GEMM) on a variety of custom and general-purpose CPU and GPU architectures. Our analysis shows that orders of magnitude improvements in efficiency are possible with relatively simple customizations and fine-tuning of memory hierarchy configurations. We argue that these customizations can be generalized to perform other representative linear algebra operations. In addition to exposing the sources of inefficiencies in current CPUs and GPUs, our results show our prototype Linear Algebra Processor (LAP) implementing Double-precision GEMM (DGEMM) can achieve 600 GFLOPS while consuming less than 25 Watts in standard 45 nm technology, which is up to 50\times more energy efficient than cutting-edge CPUs. [ABSTRACT FROM AUTHOR]