Search

Neural network pruning is a critical field aimed at reducing the infrastructure costs of neural networks by removing parameters. Traditional methods follow a fixed paradigm including pretraining, pruning, and fine-tuning. Despite the close relationship among these three stages, most pruning methods treat them as independent processes. In this paper, we propose a novel two-stage pruning method, which includes pretraining a network that is instructive for subsequent pruning, and a unified optimization model that integrates pruning and fine-tuning. Specifically, in the first stage, we design a group sparse regularized model for pretraining. This model not only safeguards the network from irreversible damage but also offers valuable insights for the pruning process. In the second stage, we introduce an element-wise sparse regularization into pruning model. This model enables us to pinpoint sparse weights more precisely than pretrained network. It automatically derives effective pruning criteria, and omits the step of fine-tuning. To implement the two-stage process in practice, we utilize stochastic gradient algorithm for the pretraining and design a threshold algorithm for pruning stage. Extensive experiments confirm the competitive performance of our proposed method in terms of both accuracy and memory cost when compared to various benchmarks. Furthermore, ablation experiments validate the effectiveness of the proposed pretraining model's guidance for the pruning process. • We propose SOGP to enhance the connection between pretraining and pruning. • We propose a novel non-convex group sparse regularization GT l 1 . • We construct a unified optimization model to integrate pruning and fine-tuning. • Extensive experiments validate the good pruning and accuracy performance of SOGP. [ABSTRACT FROM AUTHOR]

Model pruning is a commonly used technique for compressing DNNs and reducing computation requirements to accelerate inference. However, the required granularity of pruning varies across different application scenarios, making it difficult and cumbersome to customize different pruning methods for each hardware or computing platform. Therefore, a unified framework is necessary to accommodate various levels of granularity of pruning. Furthermore, some available methods require additional fine-tuning or model retraining to restore accuracy, which can result in significant time costs. With this motivation, this paper proposes a Multi-Granularity Pruning Framework, namely MGPF, to obtain sparse models of different granularity without fine-tuning the remaining connections. Specifically, a deformable soft mask is introduced as the pruning initiator to achieve different levels of pruning granularity, such as weight pruning, channel pruning, and filter pruning, etc. The model parameters and soft masks are jointly trained, and we just apply L 1 regularization on soft masks for sparsity to ensure that the model can be repaired during training without fine-tuning or retraining. After pruning, the soft masks are absorbed into the model parameters in the form of element product without changing the model structures. Experimental results on three image classification benchmarks CIFAR-10/100 and ImageNet-1K demonstrate the effectiveness of our method for various CNN architectures, datasets, and pruning rates. Particularly, for ResNet-50 on ImageNet-1K, we achieve a higher accuracy under the pruning rate of 98% for unstructured pruning which leads the advanced method by 12%. [ABSTRACT FROM AUTHOR]

Inference centers need more data to have a more comprehensive and beneficial learning model, and for this purpose, they need to collect data from data providers. On the other hand, data providers are cautious about delivering their datasets to inference centers in terms of privacy considerations. In this paper, by modifying the structure of the autoencoder, we present a method that manages the utility-privacy trade-off well. To be more precise, the data is first compressed using the encoder, then confidential and non-confidential features are separated and uncorrelated using the classifier. The confidential feature is appropriately combined with noise, and the non-confidential feature is enhanced, and at the end, data with the original data format is produced by the decoder. The suggested architecture additionally enables data providers to modify the degree of privacy needed for private features and the level of utility for non-private features. The proposed method has been examined for both image and categorical databases, and the results show a significant performance improvement compared to previous methods. [ABSTRACT FROM AUTHOR]