Back to Search
Start Over
Transform Quantization for CNN Compression.
- Source :
-
IEEE transactions on pattern analysis and machine intelligence [IEEE Trans Pattern Anal Mach Intell] 2022 Sep; Vol. 44 (9), pp. 5700-5714. Date of Electronic Publication: 2022 Aug 04. - Publication Year :
- 2022
-
Abstract
- In this paper, we compress convolutional neural network (CNN) weights post-training via transform quantization. Previous CNN quantization techniques tend to ignore the joint statistics of weights and activations, producing sub-optimal CNN performance at a given quantization bit-rate, or consider their joint statistics during training only and do not facilitate efficient compression of already trained CNN models. We optimally transform (decorrelate) and quantize the weights post-training using a rate-distortion framework to improve compression at any given quantization bit-rate. Transform quantization unifies quantization and dimensionality reduction (decorrelation) techniques in a single framework to facilitate low bit-rate compression of CNNs and efficient inference in the transform domain. We first introduce a theory of rate and distortion for CNN quantization and pose optimum quantization as a rate-distortion optimization problem. We then show that this problem can be solved using optimal bit-depth allocation following decorrelation by the optimal End-to-end Learned Transform (ELT) we derive in this paper. Experiments demonstrate that transform quantization advances the state of the art in CNN compression in both retrained and non-retrained quantization scenarios. In particular, we find that transform quantization with retraining is able to compress CNN models such as AlexNet, ResNet and DenseNet to very low bit-rates (1-2 bits).
Details
- Language :
- English
- ISSN :
- 1939-3539
- Volume :
- 44
- Issue :
- 9
- Database :
- MEDLINE
- Journal :
- IEEE transactions on pattern analysis and machine intelligence
- Publication Type :
- Academic Journal
- Accession number :
- 34048338
- Full Text :
- https://doi.org/10.1109/TPAMI.2021.3084839