Search

Cyberattacks have been increasing in volume and intensity, necessitating proactive measures. Cybersecurity risk management frameworks are deployed to provide actionable intelligence to mitigate potential threats by analyzing the available cybersecurity data. Existing frameworks, such as MITRE ATT&CK, provide timely mitigation strategies against attacker capabilities yet do not account for hacker data when developing cyber threat intelligence. Therefore, we developed a novel information technology artifact, ATT&CK-Link, which incorporates a novel transformer and multi-teacher knowledge distillation design, to link hacker threats to this broadly used framework. Here, we illustrated how hospital systems can use this framework to proactively protect their cyberinfrastructure against hacker threats. Our ATT&CK-Link framework has practical implications for cybersecurity professionals, who can implement our framework to generate strategic, operational, and tactical cyber threat intelligence. ATT&CK-Link also contributes to the information systems knowledge base by providing design principles to pursue targeted cybersecurity analytics, risk management, and broader text analytics research through simultaneous multi-modal (e.g., text and code) distillation and classification. [ABSTRACT FROM AUTHOR]

Abstract: Knowledge distillation is a form of model compression that allows artificial neural networks of different sizes to learn from one another. Its main application is the compactification of large deep neural networks to free up computational resources, in particular on edge devices. In this article, we consider proton-proton collisions at the High-Luminosity Large Hadron Collider (HL-LHC) and demonstrate a successful knowledge transfer from an event-level graph neural network (GNN) to a particle-level small deep neural network (DNN). Our algorithm, DistillNet, is a DNN that is trained to learn about the provenance of particles, as provided by the soft labels that are the GNN outputs, to predict whether or not a particle originates from the primary interaction vertex. The results indicate that for this problem, which is one of the main challenges at the HL-LHC, there is minimal loss during the transfer of knowledge to the small student network, while improving significantly the computational resource needs compared to the teacher. This is demonstrated for the distilled student network on a CPU, as well as for a quantized and pruned student network deployed on an field programmable gate array. Our study proves that knowledge transfer between networks of different complexity can be used for fast artificial intelligence (AI) in high-energy physics that improves the expressiveness of observables over non-AI-based reconstruction algorithms. Such an approach can become essential at the HL-LHC experiments, e.g. to comply with the resource budget of their trigger stages.

Recently, additive manufacturing (AM) has received increased attention due to its high energy consumption. By extracting hidden information or highly representative features from energy-relevant data, knowledge distillation (KD) reduces predictive model complexity and computational load. By using almost predetermined and fixed models, the distillation process restricts students and teachers from learning from one model to another. To reduce computational costs while maintaining acceptable performance, a teacher assistant (TA) was added to the teacher-student architecture. Firstly, a teacher ensemble was combined with three baseline models to enhance accuracy. In the second step, a teacher ensemble (TA) was formed to bridge the capacity gap between the ensemble and the simplified model. As a result, the complexity of the student model was reduced. Using geometry-based features derived from layer-wise image data, a KD-based predictive model was developed to evaluate the feasibility and effectiveness of two independently trained student models. In comparison with independently trained student models, the performance of the proposed method has the lowest RMSE, MAE, and training time. [ABSTRACT FROM AUTHOR]

Biometric face and lip-reading systems are susceptible to face replay attacks. Where the intruder presents a recorded video of a legitimate user or presents printed photos to gain system access without authorization. Consequently, liveness detection becomes essential to confirm whether the person in camera view is real rather than fake replays. This study aims to detect such printed photo attacks and video or photo replay attacks on high-resolution screens. The main objective of this research is to develop a lightweight 3D-DNN model that considers both spatial and temporal features to distinguish between real and attack videos. To achieve this objective, a lightweight 3D-StudentNet face replay attack defense system is proposed leveraging 3D-ArrowNet deep neural network and knowledge distillation. The system captures dynamic spatio-temporal features from five video frames captured by an RGB camera. Considerable experimentation is conducted using the Replay-Attack and Replay-Mobile benchmark datasets. Experimental results demonstrate that the proposed 3D-ArrowNet achieves state-of-the-art performance and transfers its knowledge successfully to a lightweight 3D-StudentNet with fewer network parameters. Thus, 3D-StudentNet can supplant existing 2D-DNN architectures utilized for replay attack defense systems, which capture only spatial features. [ABSTRACT FROM AUTHOR]

The reproductive performance of sows can play a critical role in animal breeding, particularly in the efficiency and effectiveness of selection. However, manual recordings of piglet births and their survival rates cannot fully meet the large-scale production in recent years. The high precision is often required to capture more nuanced data, such as the intervals between births. The advanced technologies can be expected to enhance both the accuracy and efficiency of animal breeding programs. In this study, a lightweight network was developed to rapidly and accurately monitor the sow birthing in real time. Specifically, essential birthing metrics were engineered to analyze, such as the number of piglets born and the precise intervals between each birth. The lightweight network was tailored for the real-time monitoring of sow birthing activities. The critical birthing parameters were obtained to significantly enhance the efficiency and accuracy of breeding programs. Initially, the efficacy of different monitoring views—specifically, single versus double-column views—were evaluated on the accuracy of the improved model. A single-column view was significantly improved to accurately monitor the birthing events. The real-time decisionmaking and direct implications were obtained from the breeding outcomes. Advanced video processing techniques were incorporated, such as horizontal and vertical flipping. Some challenges were remained on the dynamic changes in the sow posture and varying camera perspectives during monitoring. Moreover, different lighting conditions were adapted to capture the inherent motion blur of active piglets during birth. Color jittering and Gaussian blur were then employed to significantly enhance the robustness of the model. The reliable performance was obtained under diverse operational conditions. Further advancements were achieved through a comparative analysis of classification networks. The results revealed that ResNet50 was greatly contributed to the recognition accuracy. MobileNetV3-S was performed the best with the compact model size and superior processing speed of 505.14 frames per second, indicating the optimal operational efficiency. Furthermore, MobileNetV3-S was refined to apply with the masked generative distillation—a sophisticated technique that was effectively enhanced the network's ability to capture and interpret essential birthing features. ResNet50 was utilized as the teacher model in the practical application, while MobileNetV3-S as the student model. The training was conducted using masked generative distillation followed by dependency graph pruning. The tests were carried out on a DELL OptiPlex microcomputer. An impressive detection speed of 83.10 frames per second was achieved with a test accuracy in a single-column field of view of 91.48%. Although there was a slight decrease in the accuracy of 0.98 percentage points, the detection speed was improved by 67.13 frames per second. This improved model was then deployed at the edge for testing. The better performance was achieved in the managed farrowing intervals with a detection error of just 0.31 seconds and the duration of piglet birth events with a mere 0.02-second error. Highly efficient and exceptionally precise real-time monitoring was obtained to promote the management practices of breeding activities in complex farm environments. In conclusion, the advanced computational techniques were integrated for the transformative potential to the monitoring of sow birthing. Real-time data was acquired to combine the image processing and machine learning. Some standards can be offered for the accuracy and efficiency in livestock management. The reproductive dynamics can greatly contribute to the sustainable and scientifically-informed animal husbandry. [ABSTRACT FROM AUTHOR]

Unsupervised anomaly detection holds significant importance in large-scale industrial manufacturing. Recent methods have capitalized on the benefits of employing a classifier pretrained on natural images to extract representative features from specific layers, which are subsequently processed using various techniques. Notably, memory bank-based methods, which have demonstrated exceptional accuracy, often incur a trade-off in terms of latency, posing a challenge in real-time industrial applications where prompt anomaly detection and response are crucial. Indeed, alternative approaches such as knowledge distillation and normalized flow have demonstrated promising performance in unsupervised anomaly detection while maintaining low latency. In this paper, we aim to revisit the concept of knowledge distillation in the context of unsupervised anomaly detection, emphasizing the significance of feature selection. By employing distinctive features and leveraging different models, we intend to highlight the importance of carefully selecting and utilizing relevant features specifically tailored for the task of anomaly detection. This article presents a novel approach for anomaly detection, which employs dual model knowledge distillation and incorporates various types of semantic information by leveraging high and low-level semantic information. [ABSTRACT FROM AUTHOR]

Medical image segmentation models are commonly known for their complex structures, which often render them impractical for use on edge computing devices and compromising efficiency in the segmentation process. In light of this, the industry has proposed the adoption of knowledge distillation techniques. Nevertheless, the vast majority of existing knowledge distillation methods are focused on the classification tasks of skin diseases. Specifically, for the segmentation tasks of dermoscopy lesion images, these knowledge distillation methods fail to fully recognize the importance of features in the boundary regions of lesions within medical images, lacking boundary awareness for skin lesions. This paper introduces pioneering medical image knowledge distillation architecture. The aim of this method is to facilitate the efficient transfer of knowledge from existing complex medical image segmentation networks to a more simplified student network. Initially, a masked boundary feature (MBF) distillation module is designed. By applying random masking to the periphery of skin lesions, the MBF distillation module obliges the student network to reproduce the comprehensive features of the teacher network. This process, in turn, augments the representational capabilities of the student network. Building on the MBF distillation module, this paper employs a cascaded combination approach to integrate the MBF distillation module into a multi-head boundary feature (M2BF) distillation module, further strengthening the student network's feature learning ability and enhancing the overall image segmentation performance of the distillation model. This method has been experimentally validated on the public datasets ISIC-2016 and PH2, with results showing significant performance improvements in the student network. Our findings highlight the practical utility of the lightweight network distilled using our approach, particularly in scenarios demanding high operational speed and minimal storage usage. This research offers promising prospects for practical applications in the realm of medical image segmentation. [ABSTRACT FROM AUTHOR]

Learning a fast and discriminative patch descriptor is a challenging topic in computer vision. Recently, many existing works focus on training various descriptor learning networks by minimizing a triplet loss (or its variants), which is expected to decrease the distance between each positive pair and increase the distance between each negative pair. However, such an expectation has to be lowered due to the non-perfect convergence of network optimizer to a local solution. Addressing this problem and the open computational speed problem, we propose a Descriptor Distillation framework for local descriptor learning, called DesDis, where a student model gains knowledge from a pre-trained teacher model, and it is further enhanced via a designed teacher-student regularizer. This teacher-student regularizer is to constrain the difference between the positive (also negative) pair similarity from the teacher model and that from the student model, and we theoretically prove that a more effective student model could be trained by minimizing a weighted combination of the triplet loss and this regularizer, than its teacher which is trained by minimizing the triplet loss singly. Under the proposed DesDis, many existing descriptor networks could be embedded as the teacher model, and accordingly, both equal-weight and light-weight student models could be derived, which outperform their teacher in either accuracy or speed. Experimental results on 3 public datasets demonstrate that the equal-weight student models, derived from the proposed DesDis framework by utilizing three typical descriptor learning networks as teacher models, could achieve significantly better performances than their teachers and several other comparative methods. In addition, the derived light-weight models could achieve 8 times or even faster speeds than the comparative methods under similar patch verification performances. [ABSTRACT FROM AUTHOR]

Current knowledge distillation (KD) methods primarily focus on transferring various structured knowledge and designing corresponding optimization goals to encourage the student network to imitate the output of the teacher network. However, introducing too many additional optimization objectives may lead to unstable training, such as gradient conflicts. Moreover, these methods ignored the guidelines of relative learning difficulty between the teacher and student networks. Inspired by human cognitive science, in this paper, we redefine knowledge from a new perspective — the student and teacher networks’ relative difficulty of samples, and propose a pixel-level KD paradigm for semantic segmentation named relative difficulty distillation (RDD). We propose a two-stage RDD framework: teacher-full evaluated RDD (TFE-RDD) and teacher-student evaluated RDD (TSE-RDD). RDD allows the teacher network to provide effective guidance on learning focus without additional optimization goals, thus avoiding adjusting learning weights for multiple losses. Extensive experimental evaluations using a general distillation loss function on popular datasets such as Cityscapes, CamVid, Pascal VOC, and ADE20k demonstrate the effectiveness of RDD against state-of-the-art KD methods. Additionally, our research showcases that RDD can integrate with existing KD methods to improve their upper performance bound. Codes are available at . [ABSTRACT FROM AUTHOR]

Research on knowledge distillation has become active in deep neural networks. Knowledge distillation involves training a low-capacity model from a high-capacity model. However, when the capacities of the teacher and student models differ, it can result to poor learning and low generalization performance. We propose here a novel teacher assistant model called Knowledge in Attention Assistant. This model learns learning a discriminative representation of important regions and statistical information along with spatial and channel knowledge. Moreover, by using a triplet attention mechanism, the student model can learn both the inner and outer distribution of different categories, and also memorize the knowledge distribution of the teacher model. This alignment improves the effectiveness and generalization of knowledge distillation and reduces the capacity gap between the teacher and student models. The present model addresses feature inconsistency by adjusting the attention weight distribution based on the resemblance between the features of the teacher and student. The evaluation of the proposed teacher assistant method shows remarkable results. The student model outperforms the teacher model in terms of generalization performance, achieving improvements of 93.37% and 94.09% on CIFAR-10 and CIFAR-100 datasets, respectively. Furthermore, the proposed model enhances the F1-scores 91.98% on CIFAR-10 and 79.69% on CIFAR-100. [ABSTRACT FROM AUTHOR]

In the context of distribution networks, constructing 3D point cloud maps is crucial, particularly for UAV navigation and path planning tasks. Methods that utilize reflections from surfaces, such as laser and structured light, to obtain depth point clouds for surface modeling and environmental depth estimation are already quite mature in some professional scenarios. However, acquiring dense and accurate depth information typically requires very high costs. In contrast, monocular image-based depth estimation methods do not require relatively expensive equipment and specialized personnel, making them available for a wider range of applications. To achieve high precision and efficiency in UAV distribution networks, inspired by knowledge distillation, we employ a teacher–student architecture to enable efficient inference. This approach maintains high-quality depth estimation while optimizing the point cloud to obtain more precise results. In this paper, we propose KD-MonoRec, which integrates knowledge distillation into the semi-supervised MonoRec framework for UAV distribution networks. Our method demonstrates excellent performance on the KITTI dataset and performs well in collected distribution network environments. [ABSTRACT FROM AUTHOR]

At present, the early diagnosis of diabetic retinopathy (DR), a possible complication of diabetes due to elevated glucose concentrations in the blood, is usually performed by specialists using a manual inspection of high-resolution fundus images based on lesion screening, leading to problems such as high work-intensity and accessibility only in specialized health centers. To support the diagnosis of DR, we propose a deep learning-based (DL) DR lesion classification method through a knowledge distillation (KD) strategy. First, we use the pre-trained DL architecture, Inception-v3, as a teacher model to distill the dataset. Then, a student model, also using the Inception-v3 model, is trained on the distilled dataset to match the performance of the teacher model. In addition, a new combination of Kullback–Leibler (KL) divergence and categorical cross-entropy (CCE) loss is used to measure the difference between the teacher and student models. This combined metric encourages the student model to mimic the predictions of the teacher model. Finally, the trained student model is evaluated on a validation dataset to assess its performance and compare it with both the teacher model and another competitive DL model. Experiments are conducted on the two datasets, corresponding to an imbalanced and a balanced dataset. Two baseline models (Inception-v3 and YOLOv8) are evaluated for reference, obtaining a maximum training accuracy of 66.75% and 90.90%, respectively, and a maximum validation accuracy of 35.94% and 81.52%, both for the imbalanced dataset. On the other hand, the proposed DR classification model achieves an average training accuracy of 99.01% and an average validation accuracy of 97.30%, overcoming the baseline models and other state-of-the-art works. Experimental results show that the proposed model achieves competitive results in DR lesion detection and classification tasks, assisting in the early diagnosis of diabetic retinopathy. [ABSTRACT FROM AUTHOR]

With the wide application of deep learning in anomaly detection (AD), industrial vision AD has achieved remarkable success. However, current AD usually focuses on anomaly localization and rarely investigates anomaly classification. Furthermore, anomaly classification is currently requested for quality management and anomaly reason analysis. Therefore, it is essential to classify anomalies while improving the accuracy of AD. This paper designs a novel multi-classification AD (MCAD) framework to achieve high-accuracy AD with an anomaly classification function. In detail, the proposal model based on relational knowledge distillation consists of two components. The first one employs a teacher–student AD model, utilizing a relational knowledge distillation approach to transfer the interrelationships of images. The teacher–student critical layer feature activation values are used in the knowledge transfer process to achieve anomaly detection. The second component realizes anomaly multi-classification using the lightweight convolutional neural network. Our proposal has achieved 98.95, 96.04, and 92.94% AUROC AD results on MNIST, FashionMNIST, and CIFAR10 datasets. Meanwhile, we earn 97.58 and 98.10% AUROC for AD and localization in the MVTecAD dataset. The average classification accuracy of anomaly classification has reached 76.37% in fifteen categories of the MVTec-AD dataset. In particular, the classification accuracy of the leather category has gained 95.24%. The results on the MVTec-AD dataset show that MCAD achieves excellent detection, localization, and classification results. [ABSTRACT FROM AUTHOR]

Multi-teacher knowledge distillation is a powerful technique that leverages diverse information sources from multiple pre-trained teachers to enhance student model performance. However, existing methods often overlook the challenge of effectively transferring knowledge to weaker student models. To address this limitation, we propose BOMD (Bi-level Optimization for Multi-teacher Distillation), a novel approach that combines bi-level optimization with multiple orthogonal projections. Our method employs orthogonal projections to align teacher feature representations with the student's feature space while preserving structural properties. This alignment is further reinforced through a dedicated feature alignment loss. Additionally, we utilize bi-level optimization to learn optimal weighting factors for combining knowledge from heterogeneous teachers, treating the weights as upper-level variables and the student's parameters as lower-level variables. Extensive experiments on multiple benchmark datasets demonstrate the effectiveness and flexibility of BOMD. Our method achieves state-of-the-art performance on the CIFAR-100 benchmark for multi-teacher knowledge distillation across diverse scenarios, consistently outperforming existing approaches. BOMD shows significant improvements for both homogeneous and heterogeneous teacher ensembles, even when distilling to compact student models. [ABSTRACT FROM AUTHOR]

In our research, we aimed to address the shortcomings of traditional fruit image classification models, which struggle with inconsistent lighting, complex backgrounds, and high computational demands. To overcome these challenges, we developed a novel multi-label classification method incorporating advanced image preprocessing techniques, such as Contrast Limited Adaptive Histogram Equalization and the Gray World algorithm, which enhance image quality and color balance. Utilizing lightweight encoder–decoder architectures, specifically MobileNet, DenseNet, and EfficientNet, optimized with an Asymmetric Binary Cross-Entropy Loss function, we improved model performance in handling diverse sample difficulties. Furthermore, Multi-Label Knowledge Distillation (MLKD) was implemented to transfer knowledge from large, complex teacher models to smaller, efficient student models, thereby reducing computational complexity without compromising accuracy. Experimental results on the DeepFruit dataset, which includes 21,122 images of 20 fruit categories, demonstrated that our method achieved a peak mean Average Precision (mAP) of 90.2% using EfficientNet-B3, with a computational cost of 7.9 GFLOPs. Ablation studies confirmed that the integration of image preprocessing, optimized loss functions, and knowledge distillation significantly enhances performance compared to the baseline models. This innovative method offers a practical solution for real-time fruit classification on resource-constrained devices, thereby supporting advancements in smart agriculture and the food industry. [ABSTRACT FROM AUTHOR]

This paper introduces a novel method to extract the sentiment embedded in the Management's Discussion and Analysis (MD &A) section of 10-K filings. The proposed method outperforms traditional approaches in terms of sentiment classification accuracy. Utilizing this method, the MD &A sentiment is found to be a strong negative predictor of future stock returns, demonstrating consistency in both in-sample and out-of-sample settings. By contrast, if traditional sentiment extraction methods are used, the MD &A sentiment exhibits no predictive ability for stock markets. Additionally, the MD &A sentiment is associated with dividend-related macroeconomic channels regarding future stock return prediction. [ABSTRACT FROM AUTHOR]

Color-changing melons are a kind of cucurbit plant that combines ornamental and food. With the aim of increasing the efficiency of harvesting Color-changing melon fruits while reducing the deployment cost of detection models on agricultural equipment, this study presents an improved YOLOv8s network approach that uses model pruning and knowledge distillation techniques. The method first merges Dilated Wise Residual (DWR) and Dilated Reparam Block (DRB) to reconstruct the C2f module in the Backbone for better feature fusion. Next, we designed a multilevel scale fusion feature pyramid network (HS-PAN) to enrich semantic information and strengthen localization information to enhance the detection of Color-changing melon fruits with different maturity levels. Finally, we used Layer-Adaptive Sparsity Pruning and Block-Correlation Knowledge Distillation to simplify the model and recover its accuracy. In the Color-changing melon images dataset, the mAP0.5 of the improved model reaches 96.1%, the detection speed is 9.1% faster than YOLOv8s, the number of Params is reduced from 6.47M to 1.14M, the number of computed FLOPs is reduced from 22.8GFLOPs to 7.5GFLOPs. The model's size has also decreased from 12.64MB to 2.47MB, and the performance of the improved YOLOv8 is significantly more outstanding than other lightweight networks. The experimental results verify the effectiveness of the proposed method in complex scenarios, which provides a reference basis and technical support for the subsequent automatic picking of Color-changing melons. [ABSTRACT FROM AUTHOR]

In order to shorten detection times and improve average precision in embedded devices, a lightweight and high-accuracy model is proposed to detect passion fruit in complex environments (e.g., with backlighting, occlusion, overlap, sun, cloud, or rain). First, replacing the backbone network of YOLOv5 with a lightweight GhostNet model reduces the number of parameters and computational complexity while improving the detection speed. Second, a new feature branch is added to the backbone network and the feature fusion layer in the neck network is reconstructed to effectively combine the lower- and higher-level features, which improves the accuracy of the model while maintaining its lightweight nature. Finally, a knowledge distillation method is used to transfer knowledge from the more capable teacher model to the less capable student model, significantly improving the detection accuracy. The improved model is denoted as G-YOLO-NK. The average accuracy of the G-YOLO-NK network is 96.00%, which is 1.00% higher than that of the original YOLOv5s model. Furthermore, the model size is 7.14 MB, half that of the original model, and its real-time detection frame rate is 11.25 FPS when implemented on the Jetson Nano. The proposed model is found to outperform state-of-the-art models in terms of average precision and detection performance. The present work provides an effective model for real-time detection of passion fruit in complex orchard scenes, offering valuable technical support for the development of orchard picking robots and greatly improving the intelligence level of orchards. [ABSTRACT FROM AUTHOR]

Wheat scab can reduce wheat yield and quality. Currently, unmanned aerial vehicles (UAVs) are widely used for monitoring field crops. However, UAV is constrained by limited computational resources on-board the platforms. In addition, compared to ground images, UAV images have complex backgrounds and smaller targets. Given the aforementioned challenges, this paper proposes a lightweight wheat scab detection network based on UAV. In addition, overlapping cropping and image contrast enhancement methods are designed to preprocess UAV remote-sensing images. Additionally, this work constructed a lightweight wheat scab detection network called LWSDNet using mixed deep convolution (MixConv) to monitor wheat scab in field environments. MixConv can significantly reduce the parameters of the LWSDNet network through depthwise convolution and pointwise convolution, and different sizes of kernels can extract rich scab features. In order to enable LWSDNet to extract more scab features, a scab feature enhancement module, which utilizes spatial attention and dilated convolution, is designed to improve the ability of the network to extract scab features. The MixConv adaptive feature fusion module is designed to accurately detect lesions of different sizes, fully utilizing the semantic and detailed information in the network to enable more accurate detection by LWSDNet. During the training process, a knowledge distillation strategy that integrates scab features and responses is employed to further improve the average precision of LWSDNet detection. Experimental results demonstrate that the average precision of LWSDNet in detecting wheat scab is 79.8%, which is higher than common object detection models and lightweight object detection models. The parameters of LWSDNet are only 3.2 million (M), generally lower than existing lightweight object detection networks. [ABSTRACT FROM AUTHOR]

A precision agriculture approach that uses drones for crop protection and variable rate application has become the main method of rice weed control, but it suffers from excessive spraying issues, which can pollute soil and water environments and harm ecosystems. This study proposes a method to generate variable spray prescription maps based on the actual distribution of weeds in rice fields and utilize DJI plant protection UAVs to perform automatic variable spraying operations according to the prescription maps, achieving precise pesticide application. We first construct the YOLOv8n DT model by transferring the "knowledge features" learned by the larger YOLOv8l model with strong feature extraction capabilities to the smaller YOLOv8n model through knowledge distillation. We use this model to identify weeds in the field and generate an actual distribution map of rice field weeds based on the recognition results. The number of weeds in each experimental plot is counted, and the specific amount of pesticide for each plot is determined based on the amount of weeds and the spraying strategy proposed in this study. Variable spray prescription maps are then generated accordingly. DJI plant protection UAVs are used to perform automatic variable spraying operations based on prescription maps. Water-sensitive papers are used to collect droplets during the automatic variable operation process of UAVs, and the variable spraying effect is evaluated through droplet analysis. YOLOv8n-DT improved the accuracy of the model by 3.1% while keeping the model parameters constant, and the accuracy of identifying weeds in rice fields reached 0.82, which is close to the accuracy of the teacher network. Compared to the traditional extensive spraying method, the approach in this study saves approximately 15.28% of herbicides. This study demonstrates a complete workflow from UAV image acquisition to the evaluation of the variable spraying effect of plant protection UAVs. The method proposed in this research may provide an effective solution to balance the use of chemical herbicides and protect ecological safety. [ABSTRACT FROM AUTHOR]

Accurate and efficient motion estimation is a crucial component of real-time ultrasound elastography (USE). However, obtaining radiofrequency ultrasound (RF) data in clinical practice can be challenging. In contrast, although B-mode (BM) data is readily available, elastographic data derived from BM data results in sub-optimal elastographic images. Furthermore, existing conventional ultrasound devices (e.g., portable devices) cannot provide elastography modes, which has become a significant obstacle to the widespread use of traditional ultrasound devices. To address the challenges above, we developed a teacher-student guided knowledge distillation for an unsupervised convolutional neural network (TSGUPWC-Net) to improve the accuracy of BM motion estimation by employing a well-established convolutional neural network (CNN) named modified pyramid warping and cost volume network (MPWC-Net). A pre-trained teacher model based on RF is utilized to guide the training of a student model using BM data. Innovations outlined below include employing spatial attention transfer at intermediate layers to enhance the guidance effect of the model. The loss function consists of smoothness of the displacement field, knowledge distillation loss, and intermediate layer loss. We evaluated our method on simulated data, phantoms, and in vivo ultrasound data. The results indicate that our method has higher signal-to-noise ratio (SNR) and contrast-to-noise ratio (CNR) values in axial strain estimation than the model trained on BM. The model is unsupervised and requires no ground truth labels during training, making it highly promising for motion estimation applications. [ABSTRACT FROM AUTHOR]

Time-frequency analysis is a successfully used tool for analyzing the local features of seismic data. However, it suffers from several inevitable limitations, such as the restricted time-frequency resolution, the difficulty in selecting parameters, and the low computational efficiency. Inspired by deep learning, we suggest a deep learning-based workflow for seismic time-frequency analysis. The sparse S transform network (SSTNet) is first built to map the relationship between synthetic traces and sparse S transform spectra, which can be easily pre-trained by using synthetic traces and training labels. Next, we introduce knowledge distillation (KD) based transfer learning to re-train SSTNet by using a field data set without training labels, which is named the sparse S transform network with knowledge distillation (KD-SSTNet). In this way, we can effectively calculate the sparse time-frequency spectra of field data and avoid the use of field training labels. To test the availability of the suggested KD-SSTNet, we apply it to field data to estimate seismic attenuation for reservoir characterization and make detailed comparisons with the traditional time-frequency analysis methods. [ABSTRACT FROM AUTHOR]

Detecting wafer map anomalies is crucial for preventing yield loss in semiconductor fabrication, although intricate patterns and resource-intensive labeled data prerequisites hinder precise deep-learning segmentation. This paper presents an innovative, unsupervised method for detecting pixel-level anomalies in wafer maps. It utilizes an efficient dual attention module with a knowledge distillation network to learn defect distributions without anomalies. Knowledge transfer is achieved by distilling information from a pre-trained teacher into a student network with similar architecture, except an efficient dual attention module is incorporated atop the teacher network’s feature pyramid hierarchies, which enhances feature representation and segmentation across pyramid hierarchies that selectively emphasize relevant and discard irrelevant features by capturing contextual associations in positional and channel dimensions. Furthermore, it enables student networks to acquire an improved knowledge of hierarchical features to identify anomalies across different scales accurately. The dissimilarity in feature pyramids acts as a discriminatory function, predicting the likelihood of an abnormality, resulting in highly accurate pixel-level anomaly detection. Consequently, our proposed method excelled on the WM-811K and MixedWM38 datasets, achieving AUROC, AUPR, AUPRO, and F1-Scores of (99.65%, 99.35%), (97.31%, 92.13%), (90.76%, 84.66%) respectively, alongside an inference speed of 3.204 FPS, showcasing its high precision and efficiency. [ABSTRACT FROM AUTHOR]

Cloud coverage poses a significant challenge to optical image interpretation, degrading ground information on Earth's surface. Synthetic aperture radar (SAR), with its ability to penetrate clouds, provides supplementary information to optical data. However, existing optical-SAR fusion methods predominantly focus on cloud-free scenarios, neglecting the practical challenge of semantic segmentation under cloudy conditions. To tackle this issue, we propose CloudSeg, a novel framework tailored for land cover mapping in the presence of clouds. It addresses the challenges posed by cloud cover from two aspects: reducing semantic ambiguity in areas of the cloudy image that are obscured by clouds and enhancing effective information in the unobstructed portions. Specifically, CloudSeg employs a multi-task learning strategy to simultaneously handle low-level visual task and high-level semantic understanding task, mitigating the semantic ambiguity caused by cloud cover by acquiring discriminative features through an auxiliary cloud removal task. Additionally, CloudSeg incorporates a knowledge distillation strategy, which utilizes the knowledge learned by the teacher network under cloud-free conditions to guide the student network to overcome the interference of cloud-covered areas, enhancing the valuable information from the unobstructed parts of cloud-covered images. Extensive experiments conducted on two datasets, M3M-CR and WHU-OPT-SAR , demonstrate the effectiveness and superiority of the proposed CloudSeg method for land cover mapping under cloudy conditions. Specifically, CloudSeg outperforms the state-of-the-art competitors by 3.16% in terms of mIoU on M3M-CR and by 5.56% on WHU-OPT-SAR , highlighting its substantial advantages for analyzing regions frequently obscured by clouds. Codes are available at https://github.com/xufangchn/CloudSeg. [ABSTRACT FROM AUTHOR]

Knowledge distillation (KD) has been one of the most potential methods to implement a lightweight detector, which plays a significant role in satellite in-orbit processing and unmanned aerial vehicle tracking. However, existing distillation paradigms exhibit limited accuracy in detecting arbitrary-oriented objects represented with rotated bounding boxes in remote sensing images. This issue is attributed to two aspects: (i) boundary discontinuity localization distillation, caused by angle periodicity of rotated bounding boxes, and (ii) spatial ossified feature distillation, induced by orientation-agnostic knowledge transitive regions, both of which contribute to ambiguous orientation estimation of objects. To address these issues, we propose an effective KD method called Orientation Distillation (OD) via anti-ambiguous spatial transformation, which consists of two modules. (i) Anti-ambiguous Location Prediction (ALP) module reformulates the regression transformation between teacher–student bounding boxes as Gaussian distributions fitting procedure. These distributions with distilled potential are optimized to accurately localize objects with the aid of boundary continuity cost. (ii) Orientation-guided Feature Calibration (OFC) module employs a learnable affine matrix to augment fixed CNN sampling grid into a spatially remapped one, which bridges between the multi-scale feature of teacher and student for effectively delivering the refined oriented awareness within adaptively distillation regions. Overall, OD customizes the spatial transformation of bounding box representation and sampling grid to transfer anti-ambiguous orientation knowledge, and significantly improves the performance of lightweight detectors upon non-axially arranged objects. Extensive experiments on multiple datasets demonstrate that our plug-and-play distillation framework achieves state-of-the-art performance. Codes are available at https://github.com/Molly6/OD. [ABSTRACT FROM AUTHOR]

The Transformer network architecture has gained attention due to its ability to learn global relations and its superior performance. To boost performance, it is natural to distill complementary knowledge from a Transformer network to a convolutional neural network (CNN). However, most existing knowledge distillation methods only consider homologous-architecture distillation, which may not be suitable for cross-architecture scenarios, such as from Transformer to CNN. To address this problem, we analyze the globality and transferability of models, which reflect the ability to capture global knowledge and transfer knowledge from teacher to student, respectively. Inspired by our observations, a novel cross-architecture knowledge distillation method is proposed, which supports bi-directional distillation including from Transformer to CNN and from CNN to Transformer. Specifically, rather than directly mimicking the output and intermediate features of the teacher, a partial cross-attention projector (PCA/iPCA) and a group-wise linear projector (GL/iGL) are introduced to align the student features with the teacher's in two projected feature spaces. To better match the teacher's knowledge with the student's knowledge, an adaptive distillation router (ADR) is presented to decide the knowledge from which layer the teacher should be distilled to guide which layer of the student. A multi-view robust training scheme is further presented, to improve the robustness of the framework for distillation. Extensive experiments show that the proposed method outperforms 17 state-of-the-art methods on both small-scale and large-scale datasets. [ABSTRACT FROM AUTHOR]

This paper introduces Knowledge Distillation of Vision Transformer (KDViT), a novel approach for medical image classification. The Vision Transformer architecture incorporates a self-attention mechanism to autonomously learn image structure. The input medical image is segmented into patches and transformed into low-dimensional linear embeddings. Position information is integrated into each patch, and a learnable classification token is appended for classification, thereby preserving spatial relationships within the image. The output vectors are then fed into a Transformer encoder to extract both local and global features, leveraging the inherent attention mechanism for robust feature extraction across diverse medical imaging scenarios. Furthermore, knowledge distillation is employed to enhance performance by transferring insights from a large teacher model to a small student model. This approach reduces the computational requirements of the larger model and improves overall effectiveness. Integrating knowledge distillation with two Vision Transformer models not only showcases the novelty of the proposed solution for medical image classification but also enhances model interpretability, reduces computational complexity, and improves generalization capabilities. The proposed KDViT model achieved high accuracy rates of 98.39%, 88.57%, and 99.15% on the SARS-CoV-2-CT, COVID-CT, and iCTCF datasets respectively, surpassing the performance of other state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Skill learning through reinforcement learning has significantly progressed in recent years. However, it often struggles to efficiently find optimal or near-optimal policies due to the inherent trial-and-error exploration in reinforcement learning. Although algorithms have been proposed to enhance skill learning efficacy, there is still much room for improvement in terms of skill learning performance and training stability. In this paper, we propose an algorithm called skill enhancement learning with knowledge distillation (SELKD), which integrates multiple actors and multiple critics for skill learning. SELKD employs knowledge distillation to establish a mutual learning mechanism among actors. To mitigate critic overestimation bias, we introduce a novel target value calculation method. We also perform theoretical analysis to ensure the convergence of SELKD. Finally, experiments are conducted on several continuous control tasks, illustrating the effectiveness of the proposed algorithm. [ABSTRACT FROM AUTHOR]

Anomaly detection poses a significant challenge in the industry and knowledge distillation constructed using a frozen teacher network and a trainable student network is the prevailing approach for detecting suspicious regions. Forward and reverse distillation are the main ways to achieve anomaly detection. To design an effective model and aggregate detection results, we propose a dual-student knowledge distillation (DSKD) based on forward and reverse distillation. Taking advantage of the priority of reverse distillation to obtain high-level representation, we combine a skip connection and an attention module to build a reverse distillation student network that simultaneously focuses on high-level representation and low-level features. DSKD uses a forward distillation network as an auxiliary to allow the student network to preferentially obtain the query image. For different anomaly score maps obtained by the dual-student network, we use synthetic noise enhancement in combination with image segmentation loss to adaptively learn the weight scores of individual maps. Empirical experiments conducted on the MVTec dataset show that the proposed DSKD method achieves good performance on texture images as well as competitive results on object images compared with other state-of-the-art methods. Meanwhile, ablation experiments and a visualization analysis validate the contributions of each of the model's components. [ABSTRACT FROM AUTHOR]

As the demand for efficient and lightweight models in image classification grows, knowledge distillation has emerged as a promising technique to transfer expertise from complex teacher models to simpler student models. However, the efficacy of knowledge distillation is intricately linked to the choice of datasets used during training. Datasets are pivotal in shaping a model's learning process, influencing its ability to generalize and discriminate between diverse patterns. While considerable research has independently explored knowledge distillation and image classification, a comprehensive understanding of how different datasets impact knowledge distillation remains a critical gap. This study systematically investigates the impact of diverse datasets on knowledge distillation in image classification. By varying dataset characteristics such as size, domain specificity, and inherent biases, we aim to unravel the nuanced relationship between datasets and the efficacy of knowledge transfer. Our experiments employ a range of datasets to comprehensively explore their impact on the performance gains achieved through knowledge distillation. This study contributes valuable guidance for researchers and practitioners seeking to optimize image classification models through kno-featured applications. By elucidating the intricate interplay between dataset characteristics and knowledge distillation outcomes, our findings empower the community to make informed decisions when selecting datasets, ultimately advancing the field toward more robust and efficient model development. [ABSTRACT FROM AUTHOR]

Aiming at the issue that the existing aerial work safety belt wearing detection model cannot meet the real-time operation on edge devices, this paper proposes a lightweight aerial work safety belt detection model with higher accuracy. First, the model is made lightweight by introducing Ghost convolution and model pruning. Second, for complex scenarios involving occlusion, color confusion, etc., the model’s performance is optimized by introducing the new up-sampling operator, the attention mechanism, and the feature fusion network. Lastly, the model is trained using knowledge distillation to compensate for accuracy loss resulting from the lightweight design, thereby maintain a higher accuracy. Experimental results based on the Guangdong Power Grid Intelligence Challenge safety belt wearable dataset show that, in the comparison experiments, the improved model, compared with the mainstream object detection algorithm YOU ONLY LOOK ONCE v5s (YOLOv5s), has only 8.7% of the parameters of the former with only 3.7% difference in the mean Average Precision (mAP.50) metrics and the speed is improved by 100.4%. Meanwhile, the ablation experiments show that the improved model’s parameter count is reduced by 66.9% compared with the original model, while mAP.50 decreases by only 1.9%. The overhead safety belt detection model proposed in this paper combines the model’s lightweight design, SimAM attention mechanism, Bidirectional Feature Pyramid Network feature fusion network, Carafe operator, and knowledge distillation training strategy, enabling the model to maintain lightweight and real-time performance while achieving high detection accuracy. [ABSTRACT FROM AUTHOR]

In recent years, the integration of artificial intelligence in education has become key to enhancing the quality of teaching. This study addresses the real-time detection of student behavior in classroom environments by proposing the Classroom Student Behavior YOLO (CSB-YOLO) model. We enhance the model’s multi-scale feature fusion capability using the Bidirectional Feature Pyramid Network (BiFPN). Additionally, we have designed a novel Efficient Re-parameterized Detection Head (ERD Head) to accelerate the model’s inference speed and introduced Self-Calibrated Convolutions (SCConv) to compensate for any potential accuracy loss resulting from lightweight design. To further optimize performance, model pruning and knowledge distillation are utilized to reduce the model size and computational demands while maintaining accuracy. This makes CSB-YOLO suitable for deployment on low-performance classroom devices while maintaining robust detection capabilities. Tested on the classroom student behavior dataset SCB-DATASET3, the distilled and pruned CSB-YOLO, with only 0.72M parameters and 4.3 Giga Floating-point Operations Per Second (GFLOPs), maintains high accuracy and exhibits excellent real-time performance, making it particularly suitable for educational environments. [ABSTRACT FROM AUTHOR]

Intrusion detection based on federated learning allows the sharing of more high-quality attack samples to improve the intrusion detection performance of local models while preserving the privacy of local data. Most research on federated learning intrusion detection requires local models to be homogeneous. However, in practical scenarios, local models often include both homogeneous and heterogeneous models due to differences in hardware capabilities and business requirements among nodes. Additionally, there is still room for improvement in the accuracy of recognizing novel attacks in existing researches. To address the challenges mentioned above, we propose a Group-based Federated Knowledge Distillation Intrusion Detection approach. First, through a step-by-step grouping method, we achieve the grouping effect of intra-group homogeneity and inter-group heterogeneity, laying the foundation for reducing the aggregation difficulty in intra-group homogenous aggregation and inter-group heterogeneous aggregation. Second, in intra-group homogenous aggregation, a dual-objective optimization model is employed to quantify the learning quality of local models. Weight coefficients are assigned based on the learning quality to perform weighted aggregation. Lastly, in inter-group heterogeneous aggregation, the group leader model's learning quality is used to classify and aggregate local soft labels, generating global soft labels. Group leader models utilize global soft labels for knowledge distillation to acquire knowledge from heterogeneous models. Experimental results on NSL-KDD and UNSW-NB datasets demonstrate the superiority of our proposed method over other algorithms. [ABSTRACT FROM AUTHOR]

Road crack detection plays a crucial role in protecting road safety. However, early manual detection is not only time-consuming and laborious but also highly inefficient. Although existing road inspection vehicles have powerful functions and precise detection, their high cost makes them not universally applicable. Besides, existing object detection algorithms have high computational costs, but the computing power of edge devices is often limited, making it difficult to deploy detection algorithms on edge devices. To this end, we propose a lightweight and efficient road crack detection algorithm, YOLOv7 BiFPN-G, based on YOLOv7. In response to the shortcomings of PANet in feature fusion in the original YOLOv7, we introduce the BiFPN structure to construct a better backbone feature extraction network. It achieves a higher level of feature fusion through different weights. To reduce the number of parameters, we introduce a lightweight Ghost convolution instead of the standard convolution, which continuously compresses the number of parameters through depthwise separable convolution. Then, we reduce the channel width and depth training parameters to further reduce the number of parameters and use knowledge distillation to improve the performance of the reduced model. Finally, we developed a road crack detection system based on the obtained model, which is mobile-friendly, inexpensive, and universal. We conduct visual experiments to compare the detection performance of multiple algorithms in various complex road environments. Compared with YOLOv7, YOLOv7 BiFPN-G is more lightweight, with only 7.4M parameters and a model size of only 14.MB, while there is no significant degradation in model accuracy. Compared with lightweight models YOLOv5-s and YOLOv7-tiny, YOLOv7-BiFPN-G shows leading performance in average accuracy, recall, and accuracy. The experimental results demonstrate that YOLOv7-BiFPN-G has good performance in normal road environments. [ABSTRACT FROM AUTHOR]

Knowledge distillation techniques have been successfully used in computer vision in recent years. The two main distillation techniques now in use are logit-based distillation and middle layer feature-based distillation. Logit-based distillation does not handle the features of the middle layer of the teacher network well. The extraction of middle layer based features is mainly on local information. In this paper, we design a backbone network based on dual knowledge distillation. Dual knowledge distillation enables the student network to mimic the teacher predictions while learning the teacher middle layer features. A trained Resnet50 is used as a lightweight student network (feature extraction network) trained by the teacher network. Lightweight feature extraction networks can effectively reduce memory and improve tracking accuracy. Since different students have different comprehension abilities, in order to facilitate communication between student networks, we propose feature sharing between student networks in the template and search branches. Then, we propose a new tracking algorithm based on double knowledge extraction of backbone networks. Experimental results on five tracking datasets show that the tracking algorithm has good tracking performance and real-time tracking speed. [ABSTRACT FROM AUTHOR]

Existing research on multimodal machine translation (MMT) has typically enhanced bilingual translation by introducing additional alignment visual information. However, picture form requirements in multimodal datasets pose important constraints on the development of MMT because this requires a form of alignment between image, source text, and target text. This limitation is especially compounded by the fact that the inference phase, when aligning images, is not directly available in a conventional neural machine translation (NMT) setup. Therefore, we propose an innovative MMT framework called the DSKP-MMT model, which supports machine translation by enhancing knowledge distillation and feature refinement methods in the absence of images. Our model first generates multimodal features from the source text. Then, the purified features are obtained through the multimodal feature generator and knowledge distillation module. The features generated through image feature enhancement are subsequently further purified. Finally, the image–text fusion features are generated as input in the transformer-based machine translation reasoning task. In the Multi30K dataset test, the DSKP-MMT model has achieved a BLEU of 40.42 and a METEOR of 58.15, showing its ability to improve translation effectiveness and facilitating utterance communication. [ABSTRACT FROM AUTHOR]

Secondary structure prediction is a key step in understanding protein function and biological properties and is highly important in the fields of new drug development, disease treatment, bioengineering, etc. Accurately predicting the secondary structure of proteins helps to reveal how proteins are folded and how they function in cells. The application of deep learning models in protein structure prediction is particularly important because of their ability to process complex sequence information and extract meaningful patterns and features, thus significantly improving the accuracy and efficiency of prediction. In this study, a combined model integrating an improved temporal convolutional network (TCN), bidirectional long short-term memory (BiLSTM), and a multi-head attention (MHA) mechanism is proposed to enhance the accuracy of protein prediction in both eight-state and three-state structures. One-hot encoding features and word vector representations of physicochemical properties are incorporated. A significant emphasis is placed on knowledge distillation techniques utilizing the ProtT5 pretrained model, leading to performance improvements. The improved TCN, achieved through multiscale fusion and bidirectional operations, allows for better extraction of amino acid sequence features than traditional TCN models. The model demonstrated excellent prediction performance on multiple datasets. For the TS115, CB513 and PDB (2018–2020) datasets, the prediction accuracy of the eight-state structure of the six datasets in this paper reached 88.2%, 84.9%, and 95.3%, respectively, and the prediction accuracy of the three-state structure reached 91.3%, 90.3%, and 96.8%, respectively. This study not only improves the accuracy of protein secondary structure prediction but also provides an important tool for understanding protein structure and function, which is particularly applicable to resource-constrained contexts and provides a valuable tool for understanding protein structure and function. [ABSTRACT FROM AUTHOR]

In response to the increasing number of agents and changing task scenarios in multi-agent collaborative systems, existing collaborative strategies struggle to effectively adapt to new task scenarios. To address this challenge, this paper proposes a knowledge distillation method combined with a domain separation network (DSN-KD). This method leverages the well-performing policy network from a source task as the teacher model, utilizes a domain-separated neural network structure to correct the teacher model's outputs as supervision, and guides the learning of agents in new tasks. The proposed method does not require the pre-design or training of complex state-action mappings, thereby reducing the cost of transfer. Experimental results in scenarios such as UAV surveillance and UAV cooperative target occupation, robot cooperative box pushing, UAV cooperative target strike, and multi-agent cooperative resource recovery in a particle simulation environment demonstrate that the DSN-KD transfer method effectively enhances the learning speed of new task policies and improves the proximity of the policy model to the theoretically optimal policy in practical tasks. [ABSTRACT FROM AUTHOR]

To tackle the complexity and limited applicability of high-precision segmentation models for damaged road markings, this study proposes a Multi-level Adaptive Lightweight Network (MALNet) based on knowledge distillation. By incorporating multi-scale dilated convolution and adaptive spatial channel attention fusion modules, the MALNet model significantly enhances the precision, integrity, and robustness of its segmentation branch. Furthermore, it employs an intricate knowledge distillation strategy, channeling rich, layered insights from a teacher model to a student model, thus elevating the latter's segmentation ability. Concurrently, it streamlines the student model by markedly reducing its parameter count and computational demands, culminating in a segmentation network that is both high-performing and pragmatic. Rigorous testing on three distinct data sets for damaged road marking detection—CDM_P (Collective Damaged road Marking—Public), CDM_H (Collective Damaged road Marking—Highways), and CDM_C (Collective Damaged road Marking—Cityroad)—underscores the MALNet model's superior segmentation abilities across all damage types, outperforming competing models in accuracy and completeness. Notably, the MALNet model excels in parameter efficiency, computational economy, and throughput. After distillation, the student model's parameters and computational load decrease to only 31.78% and 27.40% of the teacher model's, respectively, while processing speeds increase to 1.9 times, demonstrating a significant improvement in lightweight design. [ABSTRACT FROM AUTHOR]

Natural language-processing tasks have been improved greatly by large language models (LLMs). However, numerous parameters make their execution computationally expensive and difficult on resource-constrained devices. For this problem, as well as maintaining accuracy, some techniques such as distillation and quantization have been proposed. Unfortunately, current methods fail to integrate model pruning with downstream tasks and overlook sentence-level semantic modeling, resulting in reduced efficiency of distillation. To alleviate these limitations, we propose a novel distilled lightweight model for BERT named MicroBERT. This method can transfer the knowledge contained in the "teacher" BERT model to a "student" BERT model. The sentence-level feature alignment loss (FAL) distillation mechanism, guided by Mixture-of-Experts (MoE), captures comprehensive contextual semantic knowledge from the "teacher" model to enhance the "student" model's performance while reducing its parameters. To make the outputs of "teacher" and "student" models comparable, we introduce the idea of a generative adversarial network (GAN) to train a discriminator. Our experimental results based on four datasets show that all steps of our distillation mechanism are effective, and the MicroBERT (101.14%) model outperforms TinyBERT (99%) by 2.24% in terms of average distillation reductions in various tasks on the GLUE dataset. [ABSTRACT FROM AUTHOR]

Rock image classification represents a challenging fine-grained image classification task characterized by subtle differences among closely related rock categories. Current contrastive learning methods prevalently utilized in fine-grained image classification restrict the model's capacity to discern critical features contrastively from image pairs, and are typically too large for deployment on mobile devices used for in situ rock identification. In this work, we introduce an innovative and compact model generation framework anchored by the design of a Feature Positioning Comparison Network (FPCN). The FPCN facilitates interaction between feature vectors from localized regions within image pairs, capturing both shared and distinctive features. Further, it accommodates the variable scales of objects depicted in images, which correspond to differing quantities of inherent object information, directing the network's attention to additional contextual details based on object size variability. Leveraging knowledge distillation, the architecture is streamlined, with a focus on nuanced information at activation boundaries to master the precise fine-grained decision boundaries, thereby enhancing the small model's accuracy. Empirical evidence demonstrates that our proposed method based on FPCN improves the classification accuracy mobile lightweight models by nearly 2% while maintaining the same time and space consumption. [ABSTRACT FROM AUTHOR]

In recent years, Siamese network-based trackers have achieved significant improvements in real-time tracking. Despite their success, performance bottlenecks caused by unavoidably complex scenarios in target-tracking tasks are becoming increasingly non-negligible. For example, occlusion and fast motion are factors that can easily cause tracking failures and are labeled in many high-quality tracking databases as challenging attributes. In addition, Siamese trackers tend to suffer from high memory costs, which restricts their applicability to mobile devices with tight memory budgets. To address these issues, we propose a Specialized teachers Distilled Siamese Tracker (SDST) framework to learn a student tracker, which is small, fast, and has enhanced performance in challenging attributes. SDST introduces two types of teachers for multi-teacher distillation: general teacher and specialized teachers. The former imparts basic knowledge to the students. The latter is used to transfer specialized knowledge to students, which helps improve their performance in challenging attributes. For students to efficiently capture critical knowledge from the two types of teachers, SDST is equipped with a carefully designed multi-teacher knowledge distillation model. Our model contains two processes: general teacher-student knowledge transfer and specialized teachers-student knowledge transfer. Extensive empirical evaluations of several popular Siamese trackers demonstrated the generality and effectiveness of our framework. Moreover, the results on Large-scale Single Object Tracking (LaSOT) show that the proposed method achieves a significant improvement of more than 2–4% in most challenging attributes. SDST also maintained high overall performance while achieving compression rates of up to 8x and framerates of 252 FPS and obtaining outstanding accuracy on all challenging attributes. [ABSTRACT FROM AUTHOR]

Environmental sound classification (ESC) is one of the important research topics within the nonspeech audio classification field. While deep neural networks (DNNs) have achieved significant advances in ESC recently, their high computational and memory demands render them highly unsuitable for direct deployment on resource-constrained Internet of Things (IoT) devices based on microcontroller units (MCUs). To address this challenge, we propose a novel DNN compression framework specifically designed for such devices. On the one hand, we leverage pruning techniques to significantly compress the large number of model parameters in DNNs. To reduce the accuracy loss that follows pruning, we propose a knowledge distillation scheme based on feature information from multiple intermediate layers. On the other hand, we design a two-stage quantization-aware knowledge distillation scheme to mitigate the accuracy degradation of mandatory quantization required by MCU hardware. We evaluate our framework on benchmark ESC datasets (UrbanSound8K, ESC-50) using the STM32F746ZG device. The experimental results demonstrate that our framework can achieve compression rates up to 97% while maintaining competitive inference performance compared to the uncompressed baseline. [ABSTRACT FROM AUTHOR]

Distantly supervised relation extraction (DSRE), first used to address the limitations of manually annotated data via automatically annotating the data with triplet facts, is prone to issues such as mislabeled annotations due to the interference of noisy annotations. To address the interference of noisy annotations, we leveraged a novel knowledge distillation (KD) method which was different from the conventional models on DSRE. More specifically, we proposed a model-agnostic KD method, Multi-Level Knowledge Distillation with Adaptive Temperature (MKDAT), which mainly involves two modules: Adaptive Temperature Regulation (ATR) and Multi-Level Knowledge Distilling (MKD). ATR allocates adaptive entropy-based distillation temperatures to different training instances for providing a moderate softening supervision to the student, in which label hardening is possible for instances with great entropy. MKD combines the bag-level and instance-level knowledge of the teacher as supervisions of the student, and trains the teacher and student at the bag and instance levels, respectively, which aims at mitigating the effects of noisy annotation and improving the sentence-level prediction performance. In addition, we implemented three MKDAT models based on the CNN, PCNN, and ATT-BiLSTM neural networks, respectively, and the experimental results show that our distillation models outperform the baseline models on bag-level and instance-level evaluations. [ABSTRACT FROM AUTHOR]

Humans are excellent at adapting to constantly changing circumstances, but deep neural networks have catastrophic forgetting. Recently, significant progress has been made with class-incremental methods based on dynamic network structures. However, these methods combine individual networks in a simplistic coupled manner, neglecting the fusion capabilities between modules, leading to a decline in the overall prediction performance. To address this, we propose a class-incremental learning paradigm with adaptive knowledge transfer. This paradigm leverages crucial self-learning factors to transfer the importance knowledge of old classes as much as possible, allowing each module to integrate the optimal information from the current class. Experiments demonstrate that our designed adaptive knowledge transfer module effectively reduces the sharpness of decision boundaries, thereby significantly improving the final accuracy. Additionally, we have devised a compression module with supplementary learning to mitigate errors arising from long-term session sequences during model fusion. Extensive experiments conducted on benchmarks, Our approach achieved an average accuracy exceeding 2.72%, 1.30% in the CIFAR-100 and ImageNet-100/1000 achieves SOTA performance in both ordinary and challenging class-incremental settings. • Adaptive learning knowledge transfers reduce the overall network decision boundary. • Using BDKD with complimentary learning helps further reduce accuracy loss. • Our method achieve superior performance in datasets CIFAR-100 and ImageNet100/1000. [ABSTRACT FROM AUTHOR]

In this paper, we design and evaluate the performance of the Multi-resolution Twinned Residual Auto-Encoders (MR-TRAE) model, a deep learning (DL)-based architecture specifically designed for achieving multi-resolution super-resolved images from low-resolution (LR) inputs at various scaling factors. For this purpose, we expand on the recently introduced Twinned Residual Auto-Encoders (TRAE) paradigm for single-image super-resolution (SISR) to extend it to the multi-resolution (MR) domain. The main contributions of this work include (i) the architecture of the MR-TRAE model, which utilizes cascaded trainable up-sampling modules for progressively increasing the spatial resolution of low-resolution (LR) input images at multiple scaling factors; (ii) a novel loss function designed for the joint and semi-blind training of all MR-TRAE model components; and (iii) a comprehensive analysis of the MR-TRAE trade-off between model complexity and performance. Furthermore, we thoroughly explore the connections between the MR-TRAE architecture and broader cognitive paradigms, including knowledge distillation, the teacher-student learning model, and hierarchical cognition. Performance evaluations of the MR-TRAE benchmarked against state-of-the-art models (such as U-Net, generative adversarial network (GAN)-based, and single-resolution baselines) were conducted using publicly available datasets. These datasets consist of LR computer tomography (CT) scans from patients with COVID-19. Our tests, which explored multi-resolutions at scaling factors × (2 , 4 , 8) , showed a significant finding: the MR-TRAE model can reduce training times by up to 60 % compared to those of the baselines, without a noticeable impact on achieved performance. [ABSTRACT FROM AUTHOR]

To address the issues of varying defect sizes, inconsistent data quality, and real-time detection challenges in steel defect detection, we propose a real-time efficient steel defect detection network (RTSD). This model employs a multi-scale feature extraction module (MSC3) and a mid-sized object detector (MidObj) to comprehensively capture texture features of defects across different scales. We incorporate a coordinate attention module (CA) and replace the spatial pyramid pooling structure (SPPF) to enhance defect localization capabilities. Additionally, we introduce the Wise-IoU (WIoU) loss function to balance attention to various quality defects. To address the real-time detection issue, we use Taylor channel pruning to reduce model complexity and employ channel-wise knowledge distillation instead of fine-tuning to mitigate the negative impacts of pruning. Experimental results show that on the NEU-DET data set, the average precision of RTSD reaches 83.5%. The model parameters, calculation amount, and size are 5.9M, 7.9 GFLOPs, and 11.9M, respectively, with an inference speed of up to 247.6 FPS. This demonstrates that our method can enhance performance while significantly reducing model complexity and computational overhead, offering a highly practical solution for industrial applications. [ABSTRACT FROM AUTHOR]

Pedestrian detection in densely populated scenes, particularly in the presence of occlusions, remains a challenging issue in computer vision. Existing approaches often address detection leakage by enhancing model architectures or incorporating attention mechanisms; However, small-scale pedestrians have fewer features and are easily overfitted to the dataset and these approaches still face challenges in accurately detecting pedestrians with small target sizes. To tackle this issue, this research rethinks the occlusion region through small-scale pedestrian detection and proposes the You Only Look Once model for efficient pedestrian detection(YOLO-EPD). Firstly, we find that Standard Convolution and Dilated Convolution do not fit well with pedestrian targets with different scales due to a single receptive field, and we propose the Selective Content Aware Downsampling (SCAD) module, which is integrated into the backbone to attain enhanced feature extraction. In addition, to address the issue of missed detections resulting from insufficient feature extraction for small-scale pedestrian detection, we propose the Crowded Multi-Head Attention (CMHA) module, which makes full use of multi-layer information. Finally, for the challenge of optimizing the performance and effectiveness of small-object detection, we design Unified Channel-Task Distillation (UCTD) with channel attention and a Lightweight head (Lhead) using parameter sharing to keep it lightweight. Experimental results validate the superiority of YOLO-EPD, achieving a remarkable 91.1% Average Precision (AP) on the Widerperson dataset, while concurrently reducing parameters and computational overhead by 40%. The experimental findings demonstrate that YOLO-EPD greatly accelerates the convergence of model training and achieves better real-time performance in real-world dense scenarios. [ABSTRACT FROM AUTHOR]

The classification of remote sensing images (RSIs) poses a significant challenge due to the presence of clustered ground objects and noisy backgrounds. While many approaches rely on scaling models to enhance accuracy, the deployment of RSI classifiers often requires substantial computational resources, thus necessitating the use of lightweight algorithms. In this paper, we present an efficient and robust knowledge transfer network named ERKT-Net, which is designed to provide a lightweight yet accurate convolutional neural network (CNN) classifier. This method utilizes innovative yet straightforward concepts to better accommodate the inherent nature of RSIs, thereby significantly improving the efficiency and robustness of traditional knowledge distillation (KD) techniques developed on ImageNet-1K. We evaluate ERKT-Net on three benchmark RSI datasets. The results demonstrate that our model presents superior accuracy and a very compact size compared to 40 other advanced methods published between 2020 and 2023. On the most challenging NWPU45 dataset, ERKT-Net outperformed other KD-based methods with a maximum overall accuracy (OA) value of 22.4%. Using the same criterion, it also surpassed the first-ranked multi-model method with a minimum OA value of 0.6 but presented at least a 95% reduction in parameters. Furthermore, ablation experiments indicated that our training approach has significantly improved the efficiency and robustness of classic DA techniques. Notably, it can reduce the time expenditure in the distillation phase by at least 80%, with a slight sacrifice in accuracy. This study confirmed that a logit-based KD technique can be more efficient and effective in developing lightweight yet accurate classifiers, especially when the method is tailored to the inherent characteristics of RSIs. [ABSTRACT FROM AUTHOR]