1. Global key knowledge distillation framework.
- Author
-
Wang, Junhuang, Zhang, Weiwei, Guo, Yufeng, Liang, Peng, Ji, Ming, Zhen, Chenghui, and Wang, Hanmeng
- Subjects
ARTIFICIAL neural networks ,KNOWLEDGE transfer ,CONVOLUTIONAL neural networks ,NETWORK performance - Abstract
The performance of deep neural networks is typically associated with their depth and width. As the performance of a model improves, it demands greater computational resources and memory. However, self-distillation offers a pathway to augment this performance by internally transferring knowledge. Traditional self-distillation methods lack guidance from teacher networks, thus lacking the ability to obtain correct and key information. In this paper, we propose a new self-distillation framework—Global key knowledge distillation framework (GK-KD). We implement error correction mechanisms to network logit outputs and assess the importance of feature maps through rank to obtain correct and important knowledge. Experiments demonstrate the effectiveness of this framework: on the CIFAR100 dataset, ResNet18 achieved a 4.56% accuracy improvement over the baseline, ResNet152 also achieved a 5.26% accuracy improvement, and for the VGG series, the average accuracy increased by 5.82%, outperforming previous research methods without adding additional computational costs. Further experiments show that this framework can effectively enhance the robustness and calibration of the network. • The GK-KD framework we propose enhances network performance by distilling accurate and important knowledge. • Experiments show our framework balances accuracy, computational resources, and efficiency. • Effectiveness on three datasets shown; further tests prove method's universality, reliability, and robustness. [ABSTRACT FROM AUTHOR]
- Published
- 2024
- Full Text
- View/download PDF