Search

Modern deep neural networks, particularly recent large language models, come with massive model sizes that require significant computational and storage resources. To enable the deployment of modern models on resource-constrained environments and to accelerate inference time, researchers have increasingly explored pruning techniques as a popular research direction in neural network compression. More than three thousand pruning papers have been published from 2020 to 2024. However, there is a dearth of up-to-date comprehensive review papers on pruning. To address this issue, in this survey, we provide a comprehensive review of existing research works on deep neural network pruning in a taxonomy of 1) universal/specific speedup, 2) when to prune, 3) how to prune, and 4) fusion of pruning and other compression techniques. We then provide a thorough comparative analysis of eight pairs of contrast settings for pruning (e.g., unstructured/structured, one-shot/iterative, data-free/data-driven, initialized/pre-trained weights, etc.) and explore several emerging topics, including pruning for large language models, vision transformers, diffusion models, and large multimodal models, post-training pruning, and different levels of supervision for pruning to shed light on the commonalities and differences of existing methods and lay the foundation for further method development. Finally, we provide some valuable recommendations on selecting pruning methods and prospect several promising research directions for neural network pruning. To facilitate future research on deep neural network pruning, we summarize broad pruning applications (e.g., adversarial robustness, natural language understanding, etc.) and build a curated collection of datasets, networks, and evaluations on different applications. We maintain a repository on https://github.com/hrcheng1066/awesome-pruning that serves as a comprehensive resource for neural network pruning papers and corresponding open-source codes. We will keep updating this repository to include the latest advancements in the field.

Objectives: To investigate the clinical and genetic features in a cohort of Chinese families with neurofibromatosis type 1 (NF1)., Methods: The clinical information of 21 patients with NF1 in 10 families was retrospectively analyzed. To broaden the genetic spectrum of NF1 , multiplex ligation-dependent probe amplification analysis was performed first, followed by the whole-exome sequencing, in order to identify pathogenic or potentially pathogenic variants of NF1 gene in 10 unrelated Chinese families., Results: Nine different NF1 variants were identified in all 10 families. Of these, 7 were known pathogenic variants and included the exon 1 deletion, exons 1-58 deletion, c.5401C>T (p.Q1801*), c.2291-2A>C, c.484C>T (p.Q162*), c.4922G>A (p.W1641*) and c.1019_1020del (p.S340Cfs*25). The 2 novel variants were c.5197T>C (p.S1733P) and c.783_797delinsC (p.K261Nfs*25). The p.S1733P variant was classified as a variant of uncertain significance, while p.K261Nfs*25 was classified as pathogenic. Hence, the positive detection rate of NF1 variants was 100% (10/10). While the truncating variants were responsible for 60.0% (6/10) of the cases, the splicing variant was responsible for 10% (1/10) of the cases., Conclusion: We identified 2 novel heterozygous variants (c.5197T>C and c.783_797delinsC) in the NF1 gene, which broadens the genetic spectrum of the NF1 gene., (Copyright: © Neurosciences.)

Channel pruning is attracting increasing attention in the deep model compression community due to its capability of significantly reducing computation and memory footprints without special support from specific software and hardware. A challenge of channel pruning is designing efficient and effective criteria to select channels to prune. A widely used criterion is minimal performance degeneration, e.g., loss changes before and after pruning being the smallest. To accurately evaluate the truth performance degeneration requires retraining the survived weights to convergence, which is prohibitively slow. Hence existing pruning methods settle to use previous weights (without retraining) to evaluate the performance degeneration. However, we observe that the loss changes differ significantly with and without retraining. It motivates us to develop a technique to evaluate true loss changes without retraining, using which to select channels to prune with more reliability and confidence. We first derive a closed-form estimator of the true loss change per mask change, using influence functions without retraining. Influence function is a classic technique from robust statistics that reveals the impacts of a training sample on the model's prediction and is repurposed by us to assess impacts on true loss changes. We then show how to assess the importance of all channels simultaneously and develop a novel global channel pruning algorithm accordingly. We conduct extensive experiments to verify the effectiveness of the proposed algorithm, which significantly outperforms the competing channel pruning methods on both image classification and object detection tasks. One of the attractive properties of our algorithm is that it automatically obtains the prune percentage without the cumbersome yet commonly used sensitivity analysis by local pruning. To the best of our knowledge, we are the first that shows evaluating true loss changes for pruning without retraining is possible. This finding will open up opportunities for a series of new paradigms to emerge that differ from existing pruning methods. The code is available at https://github.com/hrcheng1066/IFSO.