Search

Knowledge distillation (KD) has been one of the most potential methods to implement a lightweight detector, which plays a significant role in satellite in-orbit processing and unmanned aerial vehicle tracking. However, existing distillation paradigms exhibit limited accuracy in detecting arbitrary-oriented objects represented with rotated bounding boxes in remote sensing images. This issue is attributed to two aspects: (i) boundary discontinuity localization distillation, caused by angle periodicity of rotated bounding boxes, and (ii) spatial ossified feature distillation, induced by orientation-agnostic knowledge transitive regions, both of which contribute to ambiguous orientation estimation of objects. To address these issues, we propose an effective KD method called Orientation Distillation (OD) via anti-ambiguous spatial transformation, which consists of two modules. (i) Anti-ambiguous Location Prediction (ALP) module reformulates the regression transformation between teacher–student bounding boxes as Gaussian distributions fitting procedure. These distributions with distilled potential are optimized to accurately localize objects with the aid of boundary continuity cost. (ii) Orientation-guided Feature Calibration (OFC) module employs a learnable affine matrix to augment fixed CNN sampling grid into a spatially remapped one, which bridges between the multi-scale feature of teacher and student for effectively delivering the refined oriented awareness within adaptively distillation regions. Overall, OD customizes the spatial transformation of bounding box representation and sampling grid to transfer anti-ambiguous orientation knowledge, and significantly improves the performance of lightweight detectors upon non-axially arranged objects. Extensive experiments on multiple datasets demonstrate that our plug-and-play distillation framework achieves state-of-the-art performance. Codes are available at https://github.com/Molly6/OD. [ABSTRACT FROM AUTHOR]

The Transformer network architecture has gained attention due to its ability to learn global relations and its superior performance. To boost performance, it is natural to distill complementary knowledge from a Transformer network to a convolutional neural network (CNN). However, most existing knowledge distillation methods only consider homologous-architecture distillation, which may not be suitable for cross-architecture scenarios, such as from Transformer to CNN. To address this problem, we analyze the globality and transferability of models, which reflect the ability to capture global knowledge and transfer knowledge from teacher to student, respectively. Inspired by our observations, a novel cross-architecture knowledge distillation method is proposed, which supports bi-directional distillation including from Transformer to CNN and from CNN to Transformer. Specifically, rather than directly mimicking the output and intermediate features of the teacher, a partial cross-attention projector (PCA/iPCA) and a group-wise linear projector (GL/iGL) are introduced to align the student features with the teacher's in two projected feature spaces. To better match the teacher's knowledge with the student's knowledge, an adaptive distillation router (ADR) is presented to decide the knowledge from which layer the teacher should be distilled to guide which layer of the student. A multi-view robust training scheme is further presented, to improve the robustness of the framework for distillation. Extensive experiments show that the proposed method outperforms 17 state-of-the-art methods on both small-scale and large-scale datasets. [ABSTRACT FROM AUTHOR]

Color-changing melons are a kind of cucurbit plant that combines ornamental and food. With the aim of increasing the efficiency of harvesting Color-changing melon fruits while reducing the deployment cost of detection models on agricultural equipment, this study presents an improved YOLOv8s network approach that uses model pruning and knowledge distillation techniques. The method first merges Dilated Wise Residual (DWR) and Dilated Reparam Block (DRB) to reconstruct the C2f module in the Backbone for better feature fusion. Next, we designed a multilevel scale fusion feature pyramid network (HS-PAN) to enrich semantic information and strengthen localization information to enhance the detection of Color-changing melon fruits with different maturity levels. Finally, we used Layer-Adaptive Sparsity Pruning and Block-Correlation Knowledge Distillation to simplify the model and recover its accuracy. In the Color-changing melon images dataset, the mAP0.5 of the improved model reaches 96.1%, the detection speed is 9.1% faster than YOLOv8s, the number of Params is reduced from 6.47M to 1.14M, the number of computed FLOPs is reduced from 22.8GFLOPs to 7.5GFLOPs. The model's size has also decreased from 12.64MB to 2.47MB, and the performance of the improved YOLOv8 is significantly more outstanding than other lightweight networks. The experimental results verify the effectiveness of the proposed method in complex scenarios, which provides a reference basis and technical support for the subsequent automatic picking of Color-changing melons. [ABSTRACT FROM AUTHOR]

Anomaly detection poses a significant challenge in the industry and knowledge distillation constructed using a frozen teacher network and a trainable student network is the prevailing approach for detecting suspicious regions. Forward and reverse distillation are the main ways to achieve anomaly detection. To design an effective model and aggregate detection results, we propose a dual-student knowledge distillation (DSKD) based on forward and reverse distillation. Taking advantage of the priority of reverse distillation to obtain high-level representation, we combine a skip connection and an attention module to build a reverse distillation student network that simultaneously focuses on high-level representation and low-level features. DSKD uses a forward distillation network as an auxiliary to allow the student network to preferentially obtain the query image. For different anomaly score maps obtained by the dual-student network, we use synthetic noise enhancement in combination with image segmentation loss to adaptively learn the weight scores of individual maps. Empirical experiments conducted on the MVTec dataset show that the proposed DSKD method achieves good performance on texture images as well as competitive results on object images compared with other state-of-the-art methods. Meanwhile, ablation experiments and a visualization analysis validate the contributions of each of the model's components. [ABSTRACT FROM AUTHOR]

Road crack detection plays a crucial role in protecting road safety. However, early manual detection is not only time-consuming and laborious but also highly inefficient. Although existing road inspection vehicles have powerful functions and precise detection, their high cost makes them not universally applicable. Besides, existing object detection algorithms have high computational costs, but the computing power of edge devices is often limited, making it difficult to deploy detection algorithms on edge devices. To this end, we propose a lightweight and efficient road crack detection algorithm, YOLOv7 BiFPN-G, based on YOLOv7. In response to the shortcomings of PANet in feature fusion in the original YOLOv7, we introduce the BiFPN structure to construct a better backbone feature extraction network. It achieves a higher level of feature fusion through different weights. To reduce the number of parameters, we introduce a lightweight Ghost convolution instead of the standard convolution, which continuously compresses the number of parameters through depthwise separable convolution. Then, we reduce the channel width and depth training parameters to further reduce the number of parameters and use knowledge distillation to improve the performance of the reduced model. Finally, we developed a road crack detection system based on the obtained model, which is mobile-friendly, inexpensive, and universal. We conduct visual experiments to compare the detection performance of multiple algorithms in various complex road environments. Compared with YOLOv7, YOLOv7 BiFPN-G is more lightweight, with only 7.4M parameters and a model size of only 14.MB, while there is no significant degradation in model accuracy. Compared with lightweight models YOLOv5-s and YOLOv7-tiny, YOLOv7-BiFPN-G shows leading performance in average accuracy, recall, and accuracy. The experimental results demonstrate that YOLOv7-BiFPN-G has good performance in normal road environments. [ABSTRACT FROM AUTHOR]

Knowledge distillation techniques have been successfully used in computer vision in recent years. The two main distillation techniques now in use are logit-based distillation and middle layer feature-based distillation. Logit-based distillation does not handle the features of the middle layer of the teacher network well. The extraction of middle layer based features is mainly on local information. In this paper, we design a backbone network based on dual knowledge distillation. Dual knowledge distillation enables the student network to mimic the teacher predictions while learning the teacher middle layer features. A trained Resnet50 is used as a lightweight student network (feature extraction network) trained by the teacher network. Lightweight feature extraction networks can effectively reduce memory and improve tracking accuracy. Since different students have different comprehension abilities, in order to facilitate communication between student networks, we propose feature sharing between student networks in the template and search branches. Then, we propose a new tracking algorithm based on double knowledge extraction of backbone networks. Experimental results on five tracking datasets show that the tracking algorithm has good tracking performance and real-time tracking speed. [ABSTRACT FROM AUTHOR]

Existing research on multimodal machine translation (MMT) has typically enhanced bilingual translation by introducing additional alignment visual information. However, picture form requirements in multimodal datasets pose important constraints on the development of MMT because this requires a form of alignment between image, source text, and target text. This limitation is especially compounded by the fact that the inference phase, when aligning images, is not directly available in a conventional neural machine translation (NMT) setup. Therefore, we propose an innovative MMT framework called the DSKP-MMT model, which supports machine translation by enhancing knowledge distillation and feature refinement methods in the absence of images. Our model first generates multimodal features from the source text. Then, the purified features are obtained through the multimodal feature generator and knowledge distillation module. The features generated through image feature enhancement are subsequently further purified. Finally, the image–text fusion features are generated as input in the transformer-based machine translation reasoning task. In the Multi30K dataset test, the DSKP-MMT model has achieved a BLEU of 40.42 and a METEOR of 58.15, showing its ability to improve translation effectiveness and facilitating utterance communication. [ABSTRACT FROM AUTHOR]

Skill learning through reinforcement learning has significantly progressed in recent years. However, it often struggles to efficiently find optimal or near-optimal policies due to the inherent trial-and-error exploration in reinforcement learning. Although algorithms have been proposed to enhance skill learning efficacy, there is still much room for improvement in terms of skill learning performance and training stability. In this paper, we propose an algorithm called skill enhancement learning with knowledge distillation (SELKD), which integrates multiple actors and multiple critics for skill learning. SELKD employs knowledge distillation to establish a mutual learning mechanism among actors. To mitigate critic overestimation bias, we introduce a novel target value calculation method. We also perform theoretical analysis to ensure the convergence of SELKD. Finally, experiments are conducted on several continuous control tasks, illustrating the effectiveness of the proposed algorithm. [ABSTRACT FROM AUTHOR]

As the demand for efficient and lightweight models in image classification grows, knowledge distillation has emerged as a promising technique to transfer expertise from complex teacher models to simpler student models. However, the efficacy of knowledge distillation is intricately linked to the choice of datasets used during training. Datasets are pivotal in shaping a model's learning process, influencing its ability to generalize and discriminate between diverse patterns. While considerable research has independently explored knowledge distillation and image classification, a comprehensive understanding of how different datasets impact knowledge distillation remains a critical gap. This study systematically investigates the impact of diverse datasets on knowledge distillation in image classification. By varying dataset characteristics such as size, domain specificity, and inherent biases, we aim to unravel the nuanced relationship between datasets and the efficacy of knowledge transfer. Our experiments employ a range of datasets to comprehensively explore their impact on the performance gains achieved through knowledge distillation. This study contributes valuable guidance for researchers and practitioners seeking to optimize image classification models through kno-featured applications. By elucidating the intricate interplay between dataset characteristics and knowledge distillation outcomes, our findings empower the community to make informed decisions when selecting datasets, ultimately advancing the field toward more robust and efficient model development. [ABSTRACT FROM AUTHOR]

Intrusion detection based on federated learning allows the sharing of more high-quality attack samples to improve the intrusion detection performance of local models while preserving the privacy of local data. Most research on federated learning intrusion detection requires local models to be homogeneous. However, in practical scenarios, local models often include both homogeneous and heterogeneous models due to differences in hardware capabilities and business requirements among nodes. Additionally, there is still room for improvement in the accuracy of recognizing novel attacks in existing researches. To address the challenges mentioned above, we propose a Group-based Federated Knowledge Distillation Intrusion Detection approach. First, through a step-by-step grouping method, we achieve the grouping effect of intra-group homogeneity and inter-group heterogeneity, laying the foundation for reducing the aggregation difficulty in intra-group homogenous aggregation and inter-group heterogeneous aggregation. Second, in intra-group homogenous aggregation, a dual-objective optimization model is employed to quantify the learning quality of local models. Weight coefficients are assigned based on the learning quality to perform weighted aggregation. Lastly, in inter-group heterogeneous aggregation, the group leader model's learning quality is used to classify and aggregate local soft labels, generating global soft labels. Group leader models utilize global soft labels for knowledge distillation to acquire knowledge from heterogeneous models. Experimental results on NSL-KDD and UNSW-NB datasets demonstrate the superiority of our proposed method over other algorithms. [ABSTRACT FROM AUTHOR]

Aiming at the issue that the existing aerial work safety belt wearing detection model cannot meet the real-time operation on edge devices, this paper proposes a lightweight aerial work safety belt detection model with higher accuracy. First, the model is made lightweight by introducing Ghost convolution and model pruning. Second, for complex scenarios involving occlusion, color confusion, etc., the model’s performance is optimized by introducing the new up-sampling operator, the attention mechanism, and the feature fusion network. Lastly, the model is trained using knowledge distillation to compensate for accuracy loss resulting from the lightweight design, thereby maintain a higher accuracy. Experimental results based on the Guangdong Power Grid Intelligence Challenge safety belt wearable dataset show that, in the comparison experiments, the improved model, compared with the mainstream object detection algorithm YOU ONLY LOOK ONCE v5s (YOLOv5s), has only 8.7% of the parameters of the former with only 3.7% difference in the mean Average Precision (mAP.50) metrics and the speed is improved by 100.4%. Meanwhile, the ablation experiments show that the improved model’s parameter count is reduced by 66.9% compared with the original model, while mAP.50 decreases by only 1.9%. The overhead safety belt detection model proposed in this paper combines the model’s lightweight design, SimAM attention mechanism, Bidirectional Feature Pyramid Network feature fusion network, Carafe operator, and knowledge distillation training strategy, enabling the model to maintain lightweight and real-time performance while achieving high detection accuracy. [ABSTRACT FROM AUTHOR]

In recent years, the integration of artificial intelligence in education has become key to enhancing the quality of teaching. This study addresses the real-time detection of student behavior in classroom environments by proposing the Classroom Student Behavior YOLO (CSB-YOLO) model. We enhance the model’s multi-scale feature fusion capability using the Bidirectional Feature Pyramid Network (BiFPN). Additionally, we have designed a novel Efficient Re-parameterized Detection Head (ERD Head) to accelerate the model’s inference speed and introduced Self-Calibrated Convolutions (SCConv) to compensate for any potential accuracy loss resulting from lightweight design. To further optimize performance, model pruning and knowledge distillation are utilized to reduce the model size and computational demands while maintaining accuracy. This makes CSB-YOLO suitable for deployment on low-performance classroom devices while maintaining robust detection capabilities. Tested on the classroom student behavior dataset SCB-DATASET3, the distilled and pruned CSB-YOLO, with only 0.72M parameters and 4.3 Giga Floating-point Operations Per Second (GFLOPs), maintains high accuracy and exhibits excellent real-time performance, making it particularly suitable for educational environments. [ABSTRACT FROM AUTHOR]

In response to the increasing number of agents and changing task scenarios in multi-agent collaborative systems, existing collaborative strategies struggle to effectively adapt to new task scenarios. To address this challenge, this paper proposes a knowledge distillation method combined with a domain separation network (DSN-KD). This method leverages the well-performing policy network from a source task as the teacher model, utilizes a domain-separated neural network structure to correct the teacher model's outputs as supervision, and guides the learning of agents in new tasks. The proposed method does not require the pre-design or training of complex state-action mappings, thereby reducing the cost of transfer. Experimental results in scenarios such as UAV surveillance and UAV cooperative target occupation, robot cooperative box pushing, UAV cooperative target strike, and multi-agent cooperative resource recovery in a particle simulation environment demonstrate that the DSN-KD transfer method effectively enhances the learning speed of new task policies and improves the proximity of the policy model to the theoretically optimal policy in practical tasks. [ABSTRACT FROM AUTHOR]

Secondary structure prediction is a key step in understanding protein function and biological properties and is highly important in the fields of new drug development, disease treatment, bioengineering, etc. Accurately predicting the secondary structure of proteins helps to reveal how proteins are folded and how they function in cells. The application of deep learning models in protein structure prediction is particularly important because of their ability to process complex sequence information and extract meaningful patterns and features, thus significantly improving the accuracy and efficiency of prediction. In this study, a combined model integrating an improved temporal convolutional network (TCN), bidirectional long short-term memory (BiLSTM), and a multi-head attention (MHA) mechanism is proposed to enhance the accuracy of protein prediction in both eight-state and three-state structures. One-hot encoding features and word vector representations of physicochemical properties are incorporated. A significant emphasis is placed on knowledge distillation techniques utilizing the ProtT5 pretrained model, leading to performance improvements. The improved TCN, achieved through multiscale fusion and bidirectional operations, allows for better extraction of amino acid sequence features than traditional TCN models. The model demonstrated excellent prediction performance on multiple datasets. For the TS115, CB513 and PDB (2018–2020) datasets, the prediction accuracy of the eight-state structure of the six datasets in this paper reached 88.2%, 84.9%, and 95.3%, respectively, and the prediction accuracy of the three-state structure reached 91.3%, 90.3%, and 96.8%, respectively. This study not only improves the accuracy of protein secondary structure prediction but also provides an important tool for understanding protein structure and function, which is particularly applicable to resource-constrained contexts and provides a valuable tool for understanding protein structure and function. [ABSTRACT FROM AUTHOR]

To tackle the complexity and limited applicability of high-precision segmentation models for damaged road markings, this study proposes a Multi-level Adaptive Lightweight Network (MALNet) based on knowledge distillation. By incorporating multi-scale dilated convolution and adaptive spatial channel attention fusion modules, the MALNet model significantly enhances the precision, integrity, and robustness of its segmentation branch. Furthermore, it employs an intricate knowledge distillation strategy, channeling rich, layered insights from a teacher model to a student model, thus elevating the latter's segmentation ability. Concurrently, it streamlines the student model by markedly reducing its parameter count and computational demands, culminating in a segmentation network that is both high-performing and pragmatic. Rigorous testing on three distinct data sets for damaged road marking detection—CDM_P (Collective Damaged road Marking—Public), CDM_H (Collective Damaged road Marking—Highways), and CDM_C (Collective Damaged road Marking—Cityroad)—underscores the MALNet model's superior segmentation abilities across all damage types, outperforming competing models in accuracy and completeness. Notably, the MALNet model excels in parameter efficiency, computational economy, and throughput. After distillation, the student model's parameters and computational load decrease to only 31.78% and 27.40% of the teacher model's, respectively, while processing speeds increase to 1.9 times, demonstrating a significant improvement in lightweight design. [ABSTRACT FROM AUTHOR]

Natural language-processing tasks have been improved greatly by large language models (LLMs). However, numerous parameters make their execution computationally expensive and difficult on resource-constrained devices. For this problem, as well as maintaining accuracy, some techniques such as distillation and quantization have been proposed. Unfortunately, current methods fail to integrate model pruning with downstream tasks and overlook sentence-level semantic modeling, resulting in reduced efficiency of distillation. To alleviate these limitations, we propose a novel distilled lightweight model for BERT named MicroBERT. This method can transfer the knowledge contained in the "teacher" BERT model to a "student" BERT model. The sentence-level feature alignment loss (FAL) distillation mechanism, guided by Mixture-of-Experts (MoE), captures comprehensive contextual semantic knowledge from the "teacher" model to enhance the "student" model's performance while reducing its parameters. To make the outputs of "teacher" and "student" models comparable, we introduce the idea of a generative adversarial network (GAN) to train a discriminator. Our experimental results based on four datasets show that all steps of our distillation mechanism are effective, and the MicroBERT (101.14%) model outperforms TinyBERT (99%) by 2.24% in terms of average distillation reductions in various tasks on the GLUE dataset. [ABSTRACT FROM AUTHOR]

Rock image classification represents a challenging fine-grained image classification task characterized by subtle differences among closely related rock categories. Current contrastive learning methods prevalently utilized in fine-grained image classification restrict the model's capacity to discern critical features contrastively from image pairs, and are typically too large for deployment on mobile devices used for in situ rock identification. In this work, we introduce an innovative and compact model generation framework anchored by the design of a Feature Positioning Comparison Network (FPCN). The FPCN facilitates interaction between feature vectors from localized regions within image pairs, capturing both shared and distinctive features. Further, it accommodates the variable scales of objects depicted in images, which correspond to differing quantities of inherent object information, directing the network's attention to additional contextual details based on object size variability. Leveraging knowledge distillation, the architecture is streamlined, with a focus on nuanced information at activation boundaries to master the precise fine-grained decision boundaries, thereby enhancing the small model's accuracy. Empirical evidence demonstrates that our proposed method based on FPCN improves the classification accuracy mobile lightweight models by nearly 2% while maintaining the same time and space consumption. [ABSTRACT FROM AUTHOR]

In recent years, Siamese network-based trackers have achieved significant improvements in real-time tracking. Despite their success, performance bottlenecks caused by unavoidably complex scenarios in target-tracking tasks are becoming increasingly non-negligible. For example, occlusion and fast motion are factors that can easily cause tracking failures and are labeled in many high-quality tracking databases as challenging attributes. In addition, Siamese trackers tend to suffer from high memory costs, which restricts their applicability to mobile devices with tight memory budgets. To address these issues, we propose a Specialized teachers Distilled Siamese Tracker (SDST) framework to learn a student tracker, which is small, fast, and has enhanced performance in challenging attributes. SDST introduces two types of teachers for multi-teacher distillation: general teacher and specialized teachers. The former imparts basic knowledge to the students. The latter is used to transfer specialized knowledge to students, which helps improve their performance in challenging attributes. For students to efficiently capture critical knowledge from the two types of teachers, SDST is equipped with a carefully designed multi-teacher knowledge distillation model. Our model contains two processes: general teacher-student knowledge transfer and specialized teachers-student knowledge transfer. Extensive empirical evaluations of several popular Siamese trackers demonstrated the generality and effectiveness of our framework. Moreover, the results on Large-scale Single Object Tracking (LaSOT) show that the proposed method achieves a significant improvement of more than 2–4% in most challenging attributes. SDST also maintained high overall performance while achieving compression rates of up to 8x and framerates of 252 FPS and obtaining outstanding accuracy on all challenging attributes. [ABSTRACT FROM AUTHOR]

Environmental sound classification (ESC) is one of the important research topics within the nonspeech audio classification field. While deep neural networks (DNNs) have achieved significant advances in ESC recently, their high computational and memory demands render them highly unsuitable for direct deployment on resource-constrained Internet of Things (IoT) devices based on microcontroller units (MCUs). To address this challenge, we propose a novel DNN compression framework specifically designed for such devices. On the one hand, we leverage pruning techniques to significantly compress the large number of model parameters in DNNs. To reduce the accuracy loss that follows pruning, we propose a knowledge distillation scheme based on feature information from multiple intermediate layers. On the other hand, we design a two-stage quantization-aware knowledge distillation scheme to mitigate the accuracy degradation of mandatory quantization required by MCU hardware. We evaluate our framework on benchmark ESC datasets (UrbanSound8K, ESC-50) using the STM32F746ZG device. The experimental results demonstrate that our framework can achieve compression rates up to 97% while maintaining competitive inference performance compared to the uncompressed baseline. [ABSTRACT FROM AUTHOR]

Distantly supervised relation extraction (DSRE), first used to address the limitations of manually annotated data via automatically annotating the data with triplet facts, is prone to issues such as mislabeled annotations due to the interference of noisy annotations. To address the interference of noisy annotations, we leveraged a novel knowledge distillation (KD) method which was different from the conventional models on DSRE. More specifically, we proposed a model-agnostic KD method, Multi-Level Knowledge Distillation with Adaptive Temperature (MKDAT), which mainly involves two modules: Adaptive Temperature Regulation (ATR) and Multi-Level Knowledge Distilling (MKD). ATR allocates adaptive entropy-based distillation temperatures to different training instances for providing a moderate softening supervision to the student, in which label hardening is possible for instances with great entropy. MKD combines the bag-level and instance-level knowledge of the teacher as supervisions of the student, and trains the teacher and student at the bag and instance levels, respectively, which aims at mitigating the effects of noisy annotation and improving the sentence-level prediction performance. In addition, we implemented three MKDAT models based on the CNN, PCNN, and ATT-BiLSTM neural networks, respectively, and the experimental results show that our distillation models outperform the baseline models on bag-level and instance-level evaluations. [ABSTRACT FROM AUTHOR]

Humans are excellent at adapting to constantly changing circumstances, but deep neural networks have catastrophic forgetting. Recently, significant progress has been made with class-incremental methods based on dynamic network structures. However, these methods combine individual networks in a simplistic coupled manner, neglecting the fusion capabilities between modules, leading to a decline in the overall prediction performance. To address this, we propose a class-incremental learning paradigm with adaptive knowledge transfer. This paradigm leverages crucial self-learning factors to transfer the importance knowledge of old classes as much as possible, allowing each module to integrate the optimal information from the current class. Experiments demonstrate that our designed adaptive knowledge transfer module effectively reduces the sharpness of decision boundaries, thereby significantly improving the final accuracy. Additionally, we have devised a compression module with supplementary learning to mitigate errors arising from long-term session sequences during model fusion. Extensive experiments conducted on benchmarks, Our approach achieved an average accuracy exceeding 2.72%, 1.30% in the CIFAR-100 and ImageNet-100/1000 achieves SOTA performance in both ordinary and challenging class-incremental settings. • Adaptive learning knowledge transfers reduce the overall network decision boundary. • Using BDKD with complimentary learning helps further reduce accuracy loss. • Our method achieve superior performance in datasets CIFAR-100 and ImageNet100/1000. [ABSTRACT FROM AUTHOR]

Copyright of Journal of Frontiers of Computer Science & Technology is the property of Beijing Journal of Computer Engineering & Applications Journal Co Ltd. and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Electric power operation, as one of the key fields in the world, faces particularly prominent safety issues. Ensuring the safety of operators has become the most fundamental requirement in power operation. However, there are some safety hazards in power construction. These hazards are mainly due to weak safety awareness among staff and the failure to standardize the wearing of safety helmets. In order to effectively address this situation, technical means such as video surveillance technology and computer vision technology can be utilized to monitor whether staff are wearing helmets and provide timely feedback. Such measures will greatly enhance the safety level of power operation. This paper proposes an improved lightweight helmet detection algorithm named YOLO-M3C. The algorithm first replaces the YOLOv5s backbone network with MobileNetV3, successfully reducing the model size from 13.7 MB to 10.2 MB, thereby increasing the model's detection speed from 42.0 frames per second to 55.6 frames per second. Then, the CA attention mechanism is introduced into the backbone network to enhance the feature extraction capability of the model. Finally, in order to further improve the detection recall rate and accuracy of the model, a knowledge distillation of the model was carried out. The experimental results show that, compared with the original YOLOv5s algorithm, the average accuracy of the improved YOLO-M3C algorithm is improved by 0.123, and the recall rate is the same. These results verify that the algorithm YOLO-M3C has excellent performance in target detection and recognition, which can improve accuracy and confidence, while reducing false detection and missing detection, and effectively meet the needs of helmet-wearing detection. [ABSTRACT FROM AUTHOR]

The classification of remote sensing images (RSIs) poses a significant challenge due to the presence of clustered ground objects and noisy backgrounds. While many approaches rely on scaling models to enhance accuracy, the deployment of RSI classifiers often requires substantial computational resources, thus necessitating the use of lightweight algorithms. In this paper, we present an efficient and robust knowledge transfer network named ERKT-Net, which is designed to provide a lightweight yet accurate convolutional neural network (CNN) classifier. This method utilizes innovative yet straightforward concepts to better accommodate the inherent nature of RSIs, thereby significantly improving the efficiency and robustness of traditional knowledge distillation (KD) techniques developed on ImageNet-1K. We evaluate ERKT-Net on three benchmark RSI datasets. The results demonstrate that our model presents superior accuracy and a very compact size compared to 40 other advanced methods published between 2020 and 2023. On the most challenging NWPU45 dataset, ERKT-Net outperformed other KD-based methods with a maximum overall accuracy (OA) value of 22.4%. Using the same criterion, it also surpassed the first-ranked multi-model method with a minimum OA value of 0.6 but presented at least a 95% reduction in parameters. Furthermore, ablation experiments indicated that our training approach has significantly improved the efficiency and robustness of classic DA techniques. Notably, it can reduce the time expenditure in the distillation phase by at least 80%, with a slight sacrifice in accuracy. This study confirmed that a logit-based KD technique can be more efficient and effective in developing lightweight yet accurate classifiers, especially when the method is tailored to the inherent characteristics of RSIs. [ABSTRACT FROM AUTHOR]

In this paper, we design and evaluate the performance of the Multi-resolution Twinned Residual Auto-Encoders (MR-TRAE) model, a deep learning (DL)-based architecture specifically designed for achieving multi-resolution super-resolved images from low-resolution (LR) inputs at various scaling factors. For this purpose, we expand on the recently introduced Twinned Residual Auto-Encoders (TRAE) paradigm for single-image super-resolution (SISR) to extend it to the multi-resolution (MR) domain. The main contributions of this work include (i) the architecture of the MR-TRAE model, which utilizes cascaded trainable up-sampling modules for progressively increasing the spatial resolution of low-resolution (LR) input images at multiple scaling factors; (ii) a novel loss function designed for the joint and semi-blind training of all MR-TRAE model components; and (iii) a comprehensive analysis of the MR-TRAE trade-off between model complexity and performance. Furthermore, we thoroughly explore the connections between the MR-TRAE architecture and broader cognitive paradigms, including knowledge distillation, the teacher-student learning model, and hierarchical cognition. Performance evaluations of the MR-TRAE benchmarked against state-of-the-art models (such as U-Net, generative adversarial network (GAN)-based, and single-resolution baselines) were conducted using publicly available datasets. These datasets consist of LR computer tomography (CT) scans from patients with COVID-19. Our tests, which explored multi-resolutions at scaling factors × (2 , 4 , 8) , showed a significant finding: the MR-TRAE model can reduce training times by up to 60 % compared to those of the baselines, without a noticeable impact on achieved performance. [ABSTRACT FROM AUTHOR]

Knowledge distillation is often used for model compression and has achieved a great breakthrough in image classification, but there still remains scope for improvement in object detection, especially for knowledge extraction of small objects. The main problem is the features of small objects are often polluted by background noise and not prominent due to down-sampling of convolutional neural network (CNN), resulting in the insufficient refinement of small object features during distillation. In this paper, we propose Hierarchical Matching Knowledge Distillation Network (HMKD) that operates on the pyramid level P2 to pyramid level P4 of the feature pyramid network (FPN), aiming to intervene on small object features before affecting. We employ an encoder-decoder network to encapsulate low-resolution, highly semantic information, akin to eliciting insights from profound strata within a teacher network, and then match the encapsulated information with high-resolution feature values of small objects from shallow layers as the key. During this period, we use an attention mechanism to measure the relevance of the inquiry to the feature values. Also in the process of decoding, knowledge is distilled to the student. In addition, we introduce a supplementary distillation module to mitigate the effects of background noise. Experiments show that our method achieves excellent improvements for both one-stage and two-stage object detectors. Specifically, applying the proposed method on Faster R-CNN achieves 41.7% mAP on COCO2017 (ResNet50 as the backbone), which is 3.8% higher than that of the baseline. [ABSTRACT FROM AUTHOR]

To address the issues of varying defect sizes, inconsistent data quality, and real-time detection challenges in steel defect detection, we propose a real-time efficient steel defect detection network (RTSD). This model employs a multi-scale feature extraction module (MSC3) and a mid-sized object detector (MidObj) to comprehensively capture texture features of defects across different scales. We incorporate a coordinate attention module (CA) and replace the spatial pyramid pooling structure (SPPF) to enhance defect localization capabilities. Additionally, we introduce the Wise-IoU (WIoU) loss function to balance attention to various quality defects. To address the real-time detection issue, we use Taylor channel pruning to reduce model complexity and employ channel-wise knowledge distillation instead of fine-tuning to mitigate the negative impacts of pruning. Experimental results show that on the NEU-DET data set, the average precision of RTSD reaches 83.5%. The model parameters, calculation amount, and size are 5.9M, 7.9 GFLOPs, and 11.9M, respectively, with an inference speed of up to 247.6 FPS. This demonstrates that our method can enhance performance while significantly reducing model complexity and computational overhead, offering a highly practical solution for industrial applications. [ABSTRACT FROM AUTHOR]

Pedestrian detection in densely populated scenes, particularly in the presence of occlusions, remains a challenging issue in computer vision. Existing approaches often address detection leakage by enhancing model architectures or incorporating attention mechanisms; However, small-scale pedestrians have fewer features and are easily overfitted to the dataset and these approaches still face challenges in accurately detecting pedestrians with small target sizes. To tackle this issue, this research rethinks the occlusion region through small-scale pedestrian detection and proposes the You Only Look Once model for efficient pedestrian detection(YOLO-EPD). Firstly, we find that Standard Convolution and Dilated Convolution do not fit well with pedestrian targets with different scales due to a single receptive field, and we propose the Selective Content Aware Downsampling (SCAD) module, which is integrated into the backbone to attain enhanced feature extraction. In addition, to address the issue of missed detections resulting from insufficient feature extraction for small-scale pedestrian detection, we propose the Crowded Multi-Head Attention (CMHA) module, which makes full use of multi-layer information. Finally, for the challenge of optimizing the performance and effectiveness of small-object detection, we design Unified Channel-Task Distillation (UCTD) with channel attention and a Lightweight head (Lhead) using parameter sharing to keep it lightweight. Experimental results validate the superiority of YOLO-EPD, achieving a remarkable 91.1% Average Precision (AP) on the Widerperson dataset, while concurrently reducing parameters and computational overhead by 40%. The experimental findings demonstrate that YOLO-EPD greatly accelerates the convergence of model training and achieves better real-time performance in real-world dense scenarios. [ABSTRACT FROM AUTHOR]

Previous instance-relation knowledge distillation methods transfer structural relations between instances from the heavy teacher network to the lightweight student network, effectively enhancing the accuracy of the student. However, these methods have two limitations: (1) The modeling of relation knowledge only relies on the current mini-batch instances, causing the instance relations to be incomplete. (2) The information flow hidden in the evolution of instance relations throughout the network has been neglected. To address these problems, we propose a Global Instance Relation Distillation (GIRD) for convolutional neural network compression, which improves both the instance-level and relation-level globality. Firstly, we design a feature reutilization mechanism to store previously learned features to break through the shackles of the mini-batch. Secondly, we model the pairwise similarity-relation based on stored features to reveal more complete instance relations. Furthermore, we construct the pairwise relation-evolution across different layers to reflect the information flow. Extensive experiments on benchmark datasets demonstrate that our proposed method outperforms state-of-the-art approaches in various visual tasks. [ABSTRACT FROM AUTHOR]

The virtual try-on technology based on 2D images aims to seamlessly transfer provided garments onto target person images. Prior methods mainly concentrated on warping garments and generating images, overlooking the influence of feature alignment on the try-on results. In this study, we initially analyze the distortions present by existing methods and elucidate the critical role of feature alignment in the extraction stage. Building on this, we propose a novel feature alignment-based model (FA-VTON). Specifically, FA-VTON aligns the upsampled higher-level features from both person and garment images to acquire precise boundary information, which serves as guidance for subsequent garment warping. Concurrently, the Efficient Channel Attention mechanism (ECA) is introduced to generate the final result in the try-on generation module. This mechanism enables adaptive adjustment of channel feature weights to extract important features and reduce artifact generation. Furthermore, to make the student network focus on salient regions of each channel, we utilize channel-wise distillation (CWD) to minimize the Kullback–Leibler (KL) divergence between the channel probability maps of the two networks. The experiments show that our model achieves better results in both qualitative and quantitative analyses compared to current methods on the popular virtual try-on datasets. [ABSTRACT FROM AUTHOR]

Generally, the recognition performance of lightweight models is often lower than that of large models. Knowledge distillation, by teaching a student model using a teacher model, can further enhance the recognition accuracy of lightweight models. In this paper, we approach knowledge distillation from the perspective of intermediate feature-level knowledge distillation. We combine a cross-stage feature fusion symmetric framework, an attention mechanism to enhance the fused features, and a contrastive loss function for teacher and student models at the same stage to comprehensively implement a multistage feature fusion knowledge distillation method. This approach addresses the problem of significant differences in the intermediate feature distributions between teacher and student models, making it difficult to effectively learn implicit knowledge and thus improving the recognition accuracy of the student model. Compared to existing knowledge distillation methods, our method performs at a superior level. On the CIFAR100 dataset, it boosts the recognition accuracy of ResNet20 from 69.06% to 71.34%, and on the TinyImagenet dataset, it increases the recognition accuracy of ResNet18 from 66.54% to 68.03%, demonstrating the effectiveness and generalizability of our approach. Furthermore, there is room for further optimization of the overall distillation structure and feature extraction methods in this approach, which requires further research and exploration. [ABSTRACT FROM AUTHOR]

The classification of underwater fish species holds significant importance for fisheries management. Nevertheless, existing deep fish classification models require high computational resources, which hamper their deployment on underwater devices. Additionally, the complex underwater environment, the camouflaged appearance of fish, and the similarity among fish species pose challenges to the accuracy of lightweight fish classification models. To address the above issues, this paper proposes a novel two-tier knowledge distillation (T-KD) method to improve the accuracy and reduce the parameters of the underwater fish species classification models. Specifically, the T-KD involves the following key steps. Firstly, a new fish species dataset, Fish37, is constructed to augment the diversity of fish species present in existing datasets. Subsequently, we introduce a novel interlayer mapping similarity-preserving (IMSP), to facilitate the learning of richer discriminative features by capturing the mapping relationships between teacher and student network layers. Moreover, a new layer tail response (LTR) is proposed to mimic the predictions of the teacher network, efficiently improving classification performance and generalization capability. The proposed T-KD approach demonstrates remarkable performance in fish species classification, surpassing that of well-known lightweight models. The effectiveness of T-KD is extensively validated across various network depths, including ResNet and EfficientNet, and compared to other knowledge distillation methods like KD, PKT, RKD, and SP. Notably, T-KD outperforms MobileNetv3-large and obtains an impressive Top-1 accuracy of 97.20% on the Fish37 dataset only using about 1/15 model size of Vision Transformer. Furthermore, detailed generalization experiments are conducted to assess T-KD's performance on popular benchmark datasets, such as A_Large_Scale_Fish_Dataset, Fish4knowledge, WildFish and WildFish++. In conclusion, the results indicate the potential of the T-KD to facilitate underwater fish species classification with limited computational resources on underwater devices. This research also opens up promising avenues for the practical implementation of lightweight fish classification models. [ABSTRACT FROM AUTHOR]

Most logit-based knowledge distillation methods transfer soft labels from the teacher model to the student model via Kullback–Leibler divergence based on softmax, an exponential normalization function. However, this exponential nature of softmax tends to prioritize the largest class (target class) while neglecting smaller ones (non-target classes), leading to an oversight of the non-target classes's significance. To address this issue, we propose Non-Target-Class-Enhanced Knowledge Distillation (NTCE-KD) to amplify the role of non-target classes both in terms of magnitude and diversity. Specifically, we present a magnitude-enhanced Kullback–Leibler (MKL) divergence multi-shrinking the target class to enhance the impact of non-target classes in terms of magnitude. Additionally, to enrich the diversity of non-target classes, we introduce a diversity-based data augmentation strategy (DDA), further enhancing overall performance. Extensive experimental results on the CIFAR-100 and ImageNet-1k datasets demonstrate that non-target classes are of great significance and that our method achieves state-of-the-art performance across a wide range of teacher–student pairs. [ABSTRACT FROM AUTHOR]

The purpose of this paper is to propose a novel transfer learning regularization method based on knowledge distillation. Recently, transfer learning methods have been used in various fields. However, problems such as knowledge loss still occur during the process of transfer learning to a new target dataset. To solve these problems, there are various regularization methods based on knowledge distillation techniques. In this paper, we propose a transfer learning regularization method based on feature map alignment used in the field of knowledge distillation. The proposed method is composed of two attention-based submodules: self-pixel attention (SPA) and global channel attention (GCA). The self-pixel attention submodule utilizes both the feature maps of the source and target models, so that it provides an opportunity to jointly consider the features of the target and the knowledge of the source. The global channel attention submodule determines the importance of channels through all layers, unlike the existing methods that calculate these only within a single layer. Accordingly, transfer learning regularization is performed by considering both the interior of each single layer and the depth of the entire layer. Consequently, the proposed method using both of these submodules showed overall improved classification accuracy than the existing methods in classification experiments on commonly used datasets. [ABSTRACT FROM AUTHOR]

Malicious social bots pose a serious threat to social network security by spreading false information and guiding bad opinions in social networks. The singularity and scarcity of single organization data and the high cost of labeling social bots have given rise to the construction of federated models that combine federated learning with social bot detection. In this paper, we first combine the federated learning framework with the Relational Graph Convolutional Neural Network (RGCN) model to achieve federated social bot detection. A class-level cross entropy loss function is applied in the local model training to mitigate the effects of the class imbalance problem in local data. To address the data heterogeneity issue from multiple participants, we optimize the classical federated learning algorithm by applying knowledge distillation methods. Specifically, we adjust the client-side and server-side models separately: training a global generator to generate pseudo-samples based on the local data distribution knowledge to correct the optimization direction of client-side classification models, and integrating client-side classification models' knowledge on the server side to guide the training of the global classification model. We conduct extensive experiments on widely used datasets, and the results demonstrate the effectiveness of our approach in social bot detection in heterogeneous data scenarios. Compared to baseline methods, our approach achieves a nearly 3–10% improvement in detection accuracy when the data heterogeneity is larger. Additionally, our method achieves the specified accuracy with minimal communication rounds. [ABSTRACT FROM AUTHOR]

In marine biology research and fishing industries, ships using underwater robots to capture and identify marine organisms may face limited communication bandwidth and computing resources, making lightweight network models better suited for such conditions. To address this issue, a modified YOLOv5n model for marine organism detection was proposed, incorporating improvements in the neck network, model pruning, and knowledge distillation techniques. The model size was effectively reduced by leveraging the Gsconv lightweight convolution module to replace standard convolutions in the YOLOv5n Neck section. A novel α- giou loss function was adopted to enhance bounding box regression accuracy. Based on weight coefficients, L1-norm regularization pruning was applied to eliminate unnecessary channels and associated convolutional kernels. Finally, retraining and L2 knowledge distillation were employed to fine-tune the model accuracy close to pre-pruning levels. Experimental results demonstrated a 53% reduction in computational load and a 51% decrease in parameters compared to the original YOLOv5n baseline network. The proposed algorithm ensures the effectiveness [ABSTRACT FROM AUTHOR]

With the increasing complexity of products, the utilization of automated requirements classification in the process of requirement engineering is of positive significance to improve the efficiency of product development. There have been many experiments on the requirements classification based on artificial intelligence. However, many studies only use a single feature in semantic features or statistical features to classify requirements and put forward the problem of small-scale requirement datasets caused by the confidentiality of requirement data, which limits the application research of requirement classification in a large number of fields. Therefore, we propose a requirement classification model that integrates semantic and statistical features, and design a federated learning framework based on knowledge distillation without centralized servers in a ring architecture. In this paper, we extract a large amount of requirement data from a large number of requirement documents, and conduct experiments in the centralized learning environment and the federated learning environment respectively. The final experimental results show that our method can maintain the confidentiality of the requirement data and improve the communication efficiency in the process of model training with a slight decrease in the effect of requirement classification compared with centralized learning. [ABSTRACT FROM AUTHOR]

Object-oriented micro-video background music recommendation is a complicated task where the matching degree between videos and background music is a major issue. However, music selections in user-generated content (UGC) are prone to selection bias caused by historical preferences of uploaders. Since historical preferences are not fully reliable and may reflect obsolete behaviors, over-reliance on them should be avoided as knowledge and interests dynamically evolve. In this article, we propose a Deconfounded Cross-Modal matching model to mitigate such bias. Specifically, uploaders' personal preferences of music genres are identified as confounders that spuriously correlate music embeddings and background music selections, causing the learned system to over-recommend music from majority groups. To resolve such confounders, backdoor adjustment is utilized to deconfound the spurious correlation between music embeddings and prediction scores. We further utilize Monte Carlo estimator with batch-level average as the approximations to avoid integrating the entire confounder space calculated by the adjustment. Furthermore, we design a teacher–student network to utilize the matching of music videos, which is professionally generated content (PGC) with specialized matching, to better recommend content-matching background music. The PGC data are modeled by a teacher network to guide the matching of uploader-selected UGC data of student network by Kullback–Leibler–based knowledge transfer. Extensive experiments on the TT-150k-genre dataset demonstrate the effectiveness of the proposed method. The code is publicly available on https://github.com/jing-1/DecCM [ABSTRACT FROM AUTHOR]

Composition detection and semantic line detection are important research topics in computer vision and play an important auxiliary role in the analysis of image esthetics. However, at present, few researchers have considered the internal relationship between these two related tasks for comprehensive research. In order to solve this problem, we propose a synchronous detection network of composition class and semantic lines based on cross-attention, which can realize the mutual supervision and guidance between composition class detection and semantic line detection, to improve the accuracy of each other’s detection. First, the pre-trained composition detection model and the pre-trained semantic line detection model as two teacher models to provide data labels of composition and semantic line information for the student model. Then, we train a student model with the help of the teacher model. The student model adopts the multi-task learning architecture by combining soft and hard parameter sharing, as we propose. At the same time, we develop a cross-attention module to ensure that both tasks get the help and supervision they need from each other. Experimental results show that our method can draw semantic lines while detecting composition classes, which increases the interpretability of composition class detection. Our composition detection accuracy reaches 92.57%, and for benchmark semantic lines, the accuracy of our AUC_A metric can reach 92.00%. [ABSTRACT FROM AUTHOR]

Convolutional neural networks (CNNs) serve as the backbone for extracting image features in the majority of computer vision tasks. In an attempt to make them deployable on small devices, many academics have released small neural networks that they developed by hand or employed compression on large models via model pruning. Model pruning is a simple and efficient way to speed up neural networks. However, the performance of the pruned model (sparse network) falls short of the original model (dense network), and it is not easy to train towards convergence. Recent popular work has focused on improving the effectiveness and convergence of sub-networks. In this paper, we present our solution from the perspective of how to narrow the performance gap between sparse and dense networks, rather than how to obtain a better sub-network. For bridging the gap in their performance, we propose a novel training strategy by way of mutual learning. Furthermore, we provide a new pruning criterion called matching distance (MD) that aims to enable the sparse networks to inherit the majority of the knowledge learned from the dense networks. The experimental results demonstrate that our approach enables knowledge from dense networks to be transferred to sparse networks more efficiently. [ABSTRACT FROM AUTHOR]

Knowledge distillation algorithms have a great effect on the streamlining of deep neural networks. The current feature-based knowledge distillation algorithms either focus on a single part for improvement and ignore other beneficial parts, or provides effective guidance for the part that a small model should focus on, which makes the distillation effect insufficient. In order to make full use of the beneficial information of the large model and process it to improve the knowledge conversion rate of the small model, this paper proposed a new distillation algorithm. Firstly, it used the conditional probability distribution to fit the feature spatial distribution of the large model's middle layer, and then extracted the spatial attention maps that tended to be similar after fitting together with other beneficial information. Finally, it used the small convolutional layer, narrowed the gap between models, transmitted the transformed information to the small model to achieve distillation. Experimental results show that the algorithm has the applicability of multiple teacher-student combinations and the generality of multiple data sets, and compared with the current more advanced distillation algorithms, the performance is improved by about 1.19% and the time is shortened by 0.16 h. It has important engineering significance and wide application prospects for large networks' optimization and the application of deep learning on low-resource devices. [ABSTRACT FROM AUTHOR]

We present a new method of self-supervised learning and knowledge distillation based on multi-views and multi-representations (MV–MR). MV–MR is based on the maximization of dependence between learnable embeddings from augmented and non-augmented views, jointly with the maximization of dependence between learnable embeddings from the augmented view and multiple non-learnable representations from the non-augmented view. We show that the proposed method can be used for efficient self-supervised classification and model-agnostic knowledge distillation. Unlike other self-supervised techniques, our approach does not use any contrastive learning, clustering, or stop gradients. MV–MR is a generic framework allowing the incorporation of constraints on the learnable embeddings via the usage of image multi-representations as regularizers. The proposed method is used for knowledge distillation. MV–MR provides state-of-the-art self-supervised performance on the STL10 and CIFAR20 datasets in a linear evaluation setup. We show that a low-complexity ResNet50 model pretrained using proposed knowledge distillation based on the CLIP ViT model achieves state-of-the-art performance on STL10 and CIFAR100 datasets. [ABSTRACT FROM AUTHOR]

Knowledge distillation can transfer the knowledge from the pre-trained teacher model to the student model, thus effectively accomplishing model compression. Previous studies have carefully crafted knowledge representation, targeting loss function design, and distillation location selection, but there have been few studies on the role of classifiers in distillation. Previous experiences have shown that the final classifier of the model has an essential role in making inferences, so this paper attempts to narrow the gap in performance between models by having the student model directly use the classifier of the teacher model for the final inference, which requires an additional projector to help match features of the student encoder with the teacher's classifier. However, a single projector cannot fully align the features, and integrating multiple projectors may result in better performance. Considering the balance between projector size and performance, through experiments, we obtain the size of projectors for different network combinations and propose a simple method for projector integration. In this way, the student model undergoes feature projection and then uses the classifiers of the teacher model for inference, obtaining a similar performance to the teacher model. Through extensive experiments on the CIFAR-100 and Tiny-ImageNet datasets, we show that our approach applies to various teacher–student frameworks simply and effectively. [ABSTRACT FROM AUTHOR]

Accurate underwater target detection is crucial for the operation of autonomous underwater vehicles (AUVs), enhancing their environmental awareness and target search and rescue capabilities. Current deep learning-based detection models are typically large, requiring substantial storage and computational resources. However, the limited space on AUVs poses significant challenges for deploying these models on the embedded processors. Therefore, research on model compression is of great practical importance, aiming to reduce model parameters and computational load without significantly sacrificing accuracy. To address the challenge of deploying large detection models, this paper introduces an automated pruning method based on dependency graphs and successfully implements efficient pruning on the YOLOv7 model. To mitigate the accuracy degradation caused by extensive pruning, we design a hybrid distillation method that combines output-based and feature-based distillation techniques, thereby improving the detection accuracy of the pruned model. Finally, we deploy the compressed model on an embedded processor within an AUV to evaluate its performance. Multiple experiments confirm the effectiveness of our proposed method in practical applications. [ABSTRACT FROM AUTHOR]

In this study, a tyre defect detection model is improved and optimized under the YOLOv5 framework, aiming at radial tyre defects with characteristics such as an elongated shape and various target sizes and defect types. The DySneakConv module is introduced to replace the first BotteneckCSP in the Backbone network. The deformation offset of the DySneakConv module is used to make the convolutional energy freely adapt to the structure to improve the recognition rate of tyre defects with elongated features; the AIFI module is introduced to replace the fourth BotteneckCSP, and the self-attention mechanism and the processing of large-scale features are used to improve the recognition rate of tyre defects with elongated features using the AIFI module. This latter module has a self-attention mechanism and the ability to handle large-scale features to solve the problems of diverse tyre defects and different sizes. Secondly, the CARAFE up-sampling operator is introduced to replace the up-sampling operator in the Neck network. The up-sampling kernel prediction module in the CARAFE operator is used to increase the receptive field and allow the feature reorganization module to capture more semantic information to overcome the information loss problem of the up-sampling operator. Finally, based on the improved YOLOv5 detection algorithm, the Channel-wise Knowledge Distillation algorithm lightens the model, reducing its computational requirements and size while ensuring detection accuracy. Experimental studies were conducted on a dataset containing four types of tyre defects. Experimental results for the training set show that the improved algorithm improves the mAP0.5 by 4.6 pp, reduces the model size by 25.6 MB, reduces the computational complexity of the model by 31.3 GFLOPs, and reduces the number of parameters by 12.7 × 106 compared to the original YOLOv5m algorithm. Experimental results for the test set show that the improved algorithm improves the mAP0.5 by 2.6 pp compared to the original YOLOv5m algorithm. This suggests that the improved algorithm is more suitable for tyre defect detection than the original YOLOv5. [ABSTRACT FROM AUTHOR]

As a specific form of knowledge distillation (KD), self-knowledge distillation enables a student network to progressively distill its own knowledge without relying on a pretrained, complex teacher network; however, recent studies of self-KD have discovered that additional dark knowledge captured by auxiliary architecture or data augmentation could create better soft targets for enhancing the network but at the cost of significantly more computations and/or parameters. Moreover, most existing self-KD methods extract the soft label as a supervisory signal from individual input samples, which overlooks the knowledge of relationships among categories. Inspired by human associative learning, we propose a simple yet effective self-KD method named associative learning for self-distillation (ALSD), which progressively distills richer knowledge regarding the relationships between categories across independent samples. Specifically, in the process of distillation, the propagation of knowledge is weighted based on the intersample relationship between associated samples generated in different minibatches, which are progressively estimated with the current network. In this way, our ALSD framework achieves knowledge ensembling progressively across multiple samples using a single network, resulting in minimal computational and memory overhead compared to existing ensembling methods. Extensive experiments demonstrate that our ALSD method consistently boosts the classification performance of various architectures on multiple datasets. Notably, ALSD pushes forward the self-KD performance to 80.10% on CIFAR-100, which exceeds the standard backpropagation by 4.81%. Furthermore, we observe that the proposed method shows comparable performance with the state-of-the-art knowledge distillation methods without the pretrained teacher network. [ABSTRACT FROM AUTHOR]

Graph neural networks (GNNs) are crucial tools for processing non-Euclidean data. However, due to scalability issues caused by the dependency and topology of graph data, deploying GNNs in practical applications is challenging. Some methods aim to address this issue by transferring GNN knowledge to MLPs through knowledge distillation. However, distilled MLPs cannot directly capture graph structure information and rely only on node features, resulting in poor performance and sensitivity to noise. To solve this problem, we propose a lightweight optimization method for GNNs that combines graph contrastive learning and variable-temperature knowledge distillation. First, we use graph contrastive learning to capture graph structural representations, enriching the input information for the MLP. Then, we transfer GNN knowledge to the MLP using variable temperature knowledge distillation. Additionally, we enhance both node content and structural features before inputting them into the MLP, thus improving its performance and stability. Extensive experiments on seven datasets show that the proposed KDGCL model outperforms baseline models in both transductive and inductive settings; in particular, the KDGCL model achieves an average improvement of 1.63% in transductive settings and 0.8% in inductive settings when compared to baseline models. Furthermore, KDGCL maintains parameter efficiency and inference speed, making it competitive in terms of performance. [ABSTRACT FROM AUTHOR]

Deep neural networks perform better than shallow neural networks, but the former tends to be deeper or wider, introducing large numbers of parameters and computations. We know that networks that are too wide have a high risk of overfitting and networks that are too deep require a large amount of computation. This paper proposed a narrow-deep ResNet, increasing the depth of the network while avoiding other issues caused by making the network too wide, and used the strategy of knowledge distillation, where we set up a trained teacher model to train an unmodified, wide, and narrow-deep ResNet that allows students to learn the teacher’s output. To validate the effectiveness of this method, it is tested on Cifar-100 and Pascal VOC datasets. The method proposed in this paper allows a small model to have about the same accuracy rate as a large model, while dramatically shrinking the response time and computational effort. [ABSTRACT FROM AUTHOR]

Knowledge distillation is a model compression technique that transfers knowledge learned by teacher networks to student networks. Existing knowledge distillation methods greatly expand the forms of knowledge, but also make the distillation models complex and symmetric. However, few studies have explored the commonalities among these methods. In this study, we propose a concise distillation framework to unify these methods and a method to construct asymmetric knowledge distillation under the framework. Asymmetric distillation aims to enable differentiated knowledge transfers for different distillation objects. We designed a multi-stage shallow-wide branch bifurcation method to distill different knowledge representations and a grouping ensemble strategy to supervise the network to teach and learn selectively. Consequently, we conducted experiments using image classification benchmarks to verify the proposed method. Experimental results show that our implementation can achieve considerable improvements over existing methods, demonstrating the effectiveness of the method and the potential of the framework. [ABSTRACT FROM AUTHOR]