Search

Federated Continual Learning (FCL) approaches exist two major problems of the probability bias and the imbalance in parameter variations. These two problems lead to catastrophic forgetting of the network in the FCL process. Therefore, this paper proposes a novel FCL framework, Federated Probability Memory Recall (FedPMR), to mitigate the probability bias problem and the imbalance in parameter variations. Firstly, for the probability bias problem, this paper designs the Probability Distribution Alignment (PDA) module, which consolidates the memory of old probability experience. Specifically, PDA maintains a replay buffer and uses the probability memory stored in the buffer to correct the offset probabilities of the previous tasks during the two-stage training. Secondly, to alleviate the imbalance in parameter variations, this paper designs the Parameter Consistency Constraint (PCC) module, which constrains the magnitude of neural weight changes for previous tasks. Concretely, PCC applies a set of adaptive weights to subsets of the regularization term that constrains parameter changes, forcing the current model to be sufficiently close to the past model in parameter space distance. Experiments with various levels of task similitude across clients demonstrate that our technique establishes the new state-of-the-art performance when compared to previous FCL approaches. [ABSTRACT FROM AUTHOR]

• The proposed FMLA hybridizes attention mechanism, deformable convolutional networks. • FMLA offers promising classification performance with linear complexity. • FMLA adopts a mask mechanism to reduce the noise interference in time series. The potential of attention mechanisms in time series classification (TSC) is limited owing to general drawbacks, like weak local perception and quadratic complexity. To promote the performance of attention mechanisms, we present a flexible multi-head linear attention (FMLA) architecture, which enhances locality awareness through layer-wise interactions with deformable convolutional blocks and online knowledge distillation. We develop a simple but effective mask mechanism that helps reduce noise influence in time series and reduces the redundancy of FMLA by probabilistically selecting and masking the positions of each given series. We use incremental ablation studies on 85 UCR2018 datasets to evaluate the performance of the main techniques developed. Experimental results demonstrate that FMLA outperformed 11 state-of-the-art TSC algorithms, obtaining a mean accuracy of 89.37%. FMLA achieved the best performance on 29 short-medium and 7 long time-series datasets regarding accuracy. FMLA has a complexity of O N , where N is the sample length. As N increases from 100 to 1000, the floating-point operations per second grow linearly from 0.13G to 1.34G. [ABSTRACT FROM AUTHOR]

Recently, human head pose estimation and masked face classification are two essential problems in facial analysis. It is necessary to design a compact model to resolve both tasks in order to reduce the computational cost when deploying face recognition-based applications such as camera surveillance systems, AI cameras while maintaining accuracy. In this work, we proposed a lightweight multi-task model called MHPNet that simultaneously addresses both head pose estimation and masked face classification problems. Because of the lack of datasets with available labels for both tasks, we first train teacher models independently on two labelled datasets 300 W-LPA and MAFA to extract the head pose and masked soft label. After that, we design architecture with ResNet18 backbone and two branches for two tasks and train our proposed model with the predictions of teacher models on joint datasets via the knowledge distillation process. To evaluate the effectiveness of our model, we use AFLW2000 and BIWI datasets for head pose estimation problems and MAFA datasets for masked face classification problems. Experiment results show that our proposed model significantly improves the accuracy compared to the state-of-the-art head pose estimation methods and achieves remarkable performance on masked face dataset. Furthermore, our model has the real-time speed of ∼ 400 FPS when inferring on a Tesla V100 GPU device. Source codes and datasets are available at https://github.com/chientv99/maskpose. [ABSTRACT FROM AUTHOR]

Heterogeneous Knowledge Amalgamation (HKA) algorithms attempt to learn a versatile and lightweight student neural network from multiple pre-trained heterogeneous teachers. They encourage the student not only to produce the same prediction as the teachers but also to imitate each teacher's features separately in a learned Common Feature Space (CFS) by using Maximum Mean Discrepancy (MMD). However, there is no theoretical guarantee of the Out-of-Distribution robustness of teacher models in CFS, which can cause an overlap of feature representations when mapping unknown category samples. Furthermore, global alignment MMD can easily result in a negative transfer without considering class-level alignment and the relationships among all teachers. To overcome these issues, we propose a Dual Discriminative Feature Alignment (DDFA) framework, consisting of a Discriminative Centroid Clustering Strategy (DCCS) and a Joint Group Feature Alignment method (JGFA). DCCS promotes the class-separability of the teachers' features to alleviate the overlap issue. Meanwhile, JGFA decouples the complex discrepancy among teachers and the student at both category and group levels, extending MMD to align the features discriminatively. We test our model on a list of benchmarks and demonstrate that the learned student is robust and even outperforms its teachers in most cases. [ABSTRACT FROM AUTHOR]

Time-series classification approaches based on deep neural networks easily overfit UCR datasets, which is caused by the few-shot problem of those datasets. Therefore, to alleviate the overfitting phenomenon to further improve accuracy, we first propose label smoothing for InceptionTime (LSTime), which adopts the soft label information compared to only hard labels. Next, instead of manually adjusting soft labels by LSTime, knowledge distillation for InceptionTime (KDTime) is proposed to automatically generate soft labels by the teacher model while compressing the inference model. Finally, to rectify the incorrectly predicted soft labels from the teacher model, knowledge distillation with calibration for InceptionTime (KDCTime) is proposed, which contains two optional calibrating strategies, i.e., KDC by translating (KDCT) and KDC by reordering (KDCR). The experimental results show that the KDCTime accuracy is promising, while its inference time is orders of magnitude faster than state-of-the-art approaches. [ABSTRACT FROM AUTHOR]

Contrastive representation learning is crucial in time series analysis as it alleviates the issue of data noise and incompleteness as well as sparsity of supervision signal. However, existing contrastive learning frameworks usually focus on intra-temporal features, which fail to fully exploit the intricate nature of time series data. To address this issue, we propose DE-TSMCL, an innovative distillation enhanced framework for long sequence time series forecasting. Specifically, we design a learnable data augmentation mechanism which adaptively learns whether to mask a timestamp to obtain optimized sub-sequences. Then, we propose a contrastive learning task with momentum update to explore inter-sample and intra-temporal correlations of time series to learn the underlying structure feature on the unlabeled time series. Meanwhile, we design a supervised task to learn more robust representations and facilitate the contrastive learning process. Finally, we jointly optimize the above two tasks. By developing model loss from multiple tasks, we can learn effective representations for downstream forecasting task. Extensive experiments, in comparison with state-of-the-arts, well demonstrate the effectiveness of DE-TSMCL, where the maximum improvement can reach to 27.3%. Source code for the algorithm is available at https://github.com/gaohaozhi/DE-TSMCL. • Proposing a distillation enhanced framework for time series forecasting. • Solving distribution shift problem and noise false positive focusing of contrastive learning method. • Adopting teacher-student paradigm for contrastive learning to enhance model ability. • Experiments on real-world datasets show the superiority of the proposed model. [ABSTRACT FROM AUTHOR]

To accurately predict the complex dynamical processes of supercontinuum generation in optical fibers, an integrated deep learning model was constructed in this study, fully incorporating the strengths of bidirectional Long Short-Term Memory, Gated Recurrent Units, and Fully Connected Networks. Superior prediction precision was achieved by the integrated model in both time and frequency domains. However, the large number of parameters in the integrated model makes it unfavorable for practical deployment. To address this issue, knowledge distillation was employed, where the pre-trained integrated model guides the learning of a lightweight model. The results demonstrate that compared to standalone Long Short-Term Memory and the undistilled lightweight model, the distilled lightweight model maintains high prediction accuracy and generalization capability while significantly reducing model complexity, making it better suited for deployment on hardware systems. This research marks the first time that integrated learning and knowledge distillation are applied to predict supercontinuum generation in optical fibers, successfully striking a balance between precision and efficiency. This research provides important guidance for deep learning applications in the field of nonlinear dynamics. [Display omitted] • Integrated modeling: Complementary neural network components improve the prediction of complex spectral structures. • Efficient prediction: Knowledge distillation reduces model parameters while maintaining high accuracy in fiber dynamics forecasting. • Novel approach: First application of knowledge distillation and integrated learning in supercontinuum spectral prediction. • Deep learning advances fiber optic dynamics: Precision and efficiency synergized. [ABSTRACT FROM AUTHOR]

In recent years, lightweight object detection networks have been increasingly applied to remote sensing platforms due to their fast inference speed and flexible deployment advantages. Knowledge distillation methods have been widely used to reduce the performance gap between large and small models, and many studies have combined knowledge distillation with object detection. However, existing knowledge distillation methods often overlook the transfer of localization knowledge. Therefore, this paper proposes a method called Discretized Position Knowledge Distillation (DPKD) to improve the use of knowledge distillation in object detection. Specifically, the DPKD method incorporates a Discretization Algorithm Module (DAM), which leverages both general probability distribution and cross-Gaussian distribution to transfer high-quality bounding box position and pose information. Additionally, the Position Knowledge Distillation (PKD) method splits the target and non-target bounding boxes to form the distillation loss function, addressing the issue of missing background knowledge transfer during the distillation process. To further enhance the learning of high-quality bounding boxes, a Region Weighting Module (RWM) based on EIoU is introduced in DPKD, assigning weights to the various bounding boxes in the teacher's output. The effectiveness of DPKD in the field of remote sensing image object detection in multi-modal scenarios was verified through multi-modal training on the publicly available DOTA and HRSID datasets. [ABSTRACT FROM AUTHOR]

Due to privacy, security, and costly labeling of images, unsupervised semantic segmentation with very few samples has become a promising direction, but still remains unexplored. This inspires us to introduce the few-sample unsupervised semantic segmentation task, which is very challenging because generalizing the segmentation model from only a few unlabeled images is far from sufficient. We address this problem in the knowledge distillation perspective, by proposing a medium-sized auxiliary network as the bridge, which narrows down the semantic knowledge gap between teacher network (large) and student network (small). To this end, we develop the Knowledge Distillation Bridge (KDB) framework for few-sample unsupervised semantic segmentation. In particular, it consists of the teacher-auxiliary-student architecture, which adopts the block-wise distillation that encourages the auxiliary to imitate the teacher and the student to imitate the auxiliary. In this way, the knowledge gap between the source feature distribution and the target one is reduced, allowing the student with the smaller network to be readily deployed in highly-demanding environment. Meanwhile, each channel characterizes different semantics in feature map, which motivates us to distill the features of decoder in a channel-wise manner. Extensive experiments on two benchmarks including Pascal VOC2012 and Cityscapes demonstrate the promising performance of the proposed method, which strikes a good balance between precision and speed, e.g., it achieves the inference speed of 230 fps for a 512 × 512 image. [ABSTRACT FROM AUTHOR]

The batch-based hash learning paradigm for uni-modal or cross-modal retrieval has made great progress in recent decades. However, methods that are based on this paradigm cannot adapt to the scenario of incoming new data streams continuously; in addition, they are inefficient in terms of training time and memory cost in big data searches because they have to accumulate all the database data and new incoming data before training. Although a few online hash retrieval methods have been proposed to address these issues in recent years, they are based on shallow models, and none of them can perform uni-modal and cross-modal retrieval in one framework. To this end, we propose a novel method, namely, Online Deep Hashing for both Uni-modal and Cross-modal retrieval (ODHUC). For online deep hashing, ODHUC first trains image and text neural networks with image and text fundamental databases, and their hash codes are learned. When new data arrive, ODHUC samples images and texts from the newly arriving data and fundamental databases to update the image and text networks and learn the hash codes of the new data, the aim of which is to keep the similarity of the hash codes of the incremental and old data consistent with the supervised semantic similarity. ODHUC is also trained with sampled old data by knowledge distillation to learn new knowledge and avoid catastrophic forgetting of old knowledge. In this way, ODHUC achieves online deep hash learning continually without performance decline. For two categories of retrieval tasks, ODHUC learns image and text deep hash functions in two stages to align image and text features; it generates image hash codes first and then utilizes them to supervise the text hash code learning. Thus, ODHUC can perform both uni-modal and cross-modal retrieval tasks in one framework. Extensive experiments on three real-world datasets demonstrate that ODHUC outperforms several state-of-the-art methods. [ABSTRACT FROM AUTHOR]

• Introducing a method for arrhythmia classification of 12-lead electrocardiogram(ECG). • Classification is reduced to 1-lead ECG by teacher-student knowledge distillation(KD). • A teacher model is developed through 12-lead ECG signals. • A student model is developed through 1-lead ECG signals. • Teacher-student KD is based on the decompose feature maps within the middle layers. • The results showed competitive performance of the student as opposed to the teacher. Deep learning models developed through multi-lead electrocardiogram (ECG) signals are considered the leading methods for the automated detection of arrhythmia on computer systems. However, due to the amplitudes of input signals, these models generate too many parameters for practical use. Therefore, they are rarely used on devices with limited computational resources in the newly-emerged technology of the Internet of medical things (IoMT). Knowledge distillation was utilized in this paper to propose a method for bridging the gap between the arrhythmia classification model with multi-lead ECG signals and the arrhythmia classification model with single-lead ECG signals by minimizing the performance decline. The proposed method consists of a teacher model with advanced architecture and a student model with simple architecture. The teacher model was already developed through multi-lead ECG signals, whereas the student model was developed through single-lead signals under the supervision of the teacher. Despite its simplicity, the student model receives the dark knowledge of multi-lead ECG signals from the teacher by imitating the teacher's behavior in the development process. According to the results, the student model was nearly 262.18 times more compressed than its teacher. Moreover, the student experienced approximately 0.81% of accuracy decline in Chapman ECG with 10646 patients. [ABSTRACT FROM AUTHOR]

Estimating the predictive uncertainty of a Bayesian learning model is critical in various decision-making problems, e.g., reinforcement learning, detecting the adversarial attack, self-driving car. As the model posterior is almost always intractable, most efforts were made on finding an accurate approximation to the true posterior. Even though a decent estimation of the model posterior is obtained, another approximation is required to compute the predictive distribution over the desired output. A common accurate solution is to use Monte Carlo (MC) integration. However, it needs to maintain a large number of samples, and evaluate the model repeatedly, and average multiple model outputs. In many real-world cases, this is computationally prohibitive. In this work, assuming that the exact posterior or a decent approximation is obtained, we propose a generic framework to approximate the output probability distribution induced by the model posterior with a parameterized model and in an amortized fashion. The aim is to approximate the predictive uncertainty of a specific Bayesian model, meanwhile alleviating the heavy workload of MC integration at testing time. The proposed method is universally applicable to Bayesian classification models that allow for posterior sampling. Theoretically, we show that the idea of amortization incurs no additional costs on approximation performance. Empirical results validate the strong practical performance of our approach. [ABSTRACT FROM AUTHOR]

Underwater acoustic target recognition (UATR) systems are crucial to both military and civilian activities. However, the complex ship working conditions will largely affect the performance of recognition systems, especially in the case of working conditions mismatching (WCMM). For WCMM problems, an intelligent UATR system for working condition mismatching (UAWC) is proposed. UAWC uses auditory features as input to the system and uses knowledge distillation to learn the intrinsic connections of target features under different working conditions. In the proposed approach, the teacher network obtains initial knowledge by utilizing a large amount of existing working condition data. Next, the student network uses a small amount of target working data for training, and extracts incremental knowledge in teacher network through knowledge distillation technology to enhance the accuracy of its classification of target working data, so as to effectively deal with the WCMM problem.The tests make use of datasets for ship-radiated noise under various working conditions.The results showed that UAWC performs better than other methods on a wide range of WCMM problems. [ABSTRACT FROM AUTHOR]

Deep neural networks (DNNs) have recently been found to be vulnerable to adversarial examples, which raises concerns about their reliability and poses potential security threats. Adversarial training has been extensively studied to counter adversarial attacks. However, the limited attack types incorporated during the training phase will restrict the defense performance of models against unknown attacks and impact their standard accuracies. Furthermore, we discover that adversarial training models tend to overfit redundant noisy features, which hinders their generalization. To alleviate these issues, this paper proposes the attention information bottleneck-guided knowledge distillation (AIB-KD) method to enhance models' adversarial robustness. We integrate adversarial training with attention information bottleneck as the defense framework to achieve an optimal trade-off between information compression and classification performance. Simultaneously, we specifically employ knowledge distillation to guide the adversarial training models in learning both the standard attention information and valuable deep feature distributions to enhance their defense generalization capability. Experimental results demonstrate that AIB-KD can effectively classify adversarial examples in multiple attack settings. The average white-box and black-box classification accuracies for the WideResNet-28-10 model on the CIFAR-10 dataset are 56.59% and 85.49%, respectively, and the average accuracies on the SVHN dataset are 61.71% and 88.96%. When applied to unknown attack scenarios, AIB-KD is more effective and interpretable than state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Knowledge distillation is a highly effective method for transferring knowledge from a cumbersome teacher network to a lightweight student network. However, teacher networks are not always available. An alternative method called online knowledge distillation, which applies a group of peer networks to learn collaboratively with each other, has been proposed previously. In this study, we revisit online knowledge distillation and find that the existing training strategy limits the diversity among peer networks. Thus, online knowledge distillation cannot achieve its full potential. To address this, a novel online knowledge distillation with elastic peer (KDEP) strategy is introduced here. The entire training process is divided into two phases by KDEP. In each phase, a specific training strategy is applied to adjust the diversity to an appropriate degree. Extensive experiments have been conducted on individual benchmarks, including CIFAR-100, CINIC-10, Tiny ImageNet, and Caltech-UCSD Birds. The results demonstrate the superiority of KDEP. For example, when the peer networks are ShuffleNetV2-1.0 and ShuffleNetV2-0.5, the target peer network ShuffleNetV2-0.5 achieves 57.00 % top-1 accuracy on Tiny ImageNet via KDEP. [ABSTRACT FROM AUTHOR]

Recent recommender system (RS) have adopted large and sophisticated model architecture to better understand the complex user-item relationships, and accordingly, the size of the recommender is continuously increasing. To reduce the high inference costs of the large recommender, knowledge distillation (KD), which is a model compression technique from a large pre-trained model (teacher) to a small model (student), has been actively studied for RS. The state-of-the-art method is based on the ranking distillation approach, which makes the student preserve the ranking orders among items predicted by the teacher. In this work, we propose a new regularization method designed to maximize the effect of the ranking distillation in RS. We first point out an important limitation and a room for improvement of the state-of-the-art ranking distillation method based on our in-depth analysis.Then, we introduce the item-side ranking regularization, which can effectively prevent the student with limited capacity from being overfitted and enables the student to more accurately learn the teacher's prediction results. We validate the superiority of the proposed method by extensive experiments on real-world datasets. [ABSTRACT FROM AUTHOR]

• We raise the issue that existing KD methods overlook the quality of soft targets in KD performance. • We propose the professor model to provide high-quality and informative soft target for KD. • Our professor model is model-agnostic, thus improving various KD methods. • Our professor model is more effective than having the stronger teacher in KD. Knowledge distillation (KD) is a successful technique for transferring knowledge from one machine learning model to another model. Specifically, the idea of KD has been widely used for various tasks such as model compression and knowledge transfer between different models. However, existing studies in KD have overlooked the possibility that dark knowledge (i.e., soft targets) obtained from a complex and large model (a.k.a., a teacher model) may be either incorrect or insufficient. Such knowledge can hinder the effective learning of another small model (a.k.a., a student model). In this paper, we propose the professor model , which refines the soft target from the teacher model to improve KD. The professor model aims to achieve two goals; 1) improving the prediction accuracy and 2) capturing the inter-class correlation of the soft target from the teacher model. We first design the professor model by reformulating a conditional adversarial autoencoder (CAAE). Then, we devise two KD strategies using both teacher and professor models. Our empirical study demonstrates that the professor model effectively improves KD in three benchmark datasets: CIFAR100, TinyImagenet, and ILSVRC2015. Moreover, our comprehensive analysis shows that the professor model is much more effective than employing the stronger teacher model, in which parameters are greater than the sum of the teacher's and professor's parameters. Since the proposed model is model-agnostic, our model can be combined with any KD algorithm and consistently improves various KD techniques. [ABSTRACT FROM AUTHOR]

Most existing recommenders focus on providing users with a list of recommended products. In practical, users may only pay attention to the recommendations at the top positions. Our analysis, however, shows that the correctly recommended products from existing methods are not top posited; this would result in sacrificing users' patience and engagements. To address this issue, this paper proposes a top-aware recommender distillation (TRD) framework, viz., the rank of recommendation lists from given state-of-the-art recommendation approaches are further reinforced and refined using reinforcement learning. Different from the traditional goal of knowledge distillation to mimic the behavior of its teacher, our recommender contrastingly goes step further with the goal of surpassing its teacher recommender. More importantly, our proposed TRD can be plugged on any existed recommender; thus being generic for real-world deployment. Theoretical analysis demonstrates that TRD is guaranteed to perform at least as well as the basic recommender. Extensive experiments on five state-of-the-art recommendation algorithms across three real-world datasets further show that TRD consistently improves the recommendations at top positions. [ABSTRACT FROM AUTHOR]

Fundamental information extraction tasks, such as relation extraction and event detection, suffer from a data imbalance problem. To alleviate this problem, existing methods rely mostly on well-designed loss functions to reduce the negative influence of imbalanced data. However, this approach requires additional hyper-parameters and limits scalability. Furthermore, these methods can only benefit specific tasks and do not provide a unified framework across relation extraction and event detection. In this paper, a Classifier-Adaptation Knowledge Distillation (CAKD) framework is proposed to address these issues, thus improving relation extraction and event detection performance. The first step is to exploit sentence-level identification information across relation extraction and event detection, which can reduce identification errors caused by the data imbalance problem without relying on additional hyper-parameters. Moreover, this sentence-level identification information is used by a teacher network to guide the baseline model's training by sharing its classifier. Like an instructor, the classifier improves the baseline model's ability to extract this sentence-level identification information from raw texts, thus benefiting overall performance. Experiments were conducted on both relation extraction and event detection using the Text Analysis Conference Relation Extraction Dataset (TACRED) and Automatic Content Extraction (ACE) 2005 English datasets, respectively. The results demonstrate the effectiveness of the proposed framework. [ABSTRACT FROM AUTHOR]

In this article, we propose a knowledge distillation approach with two teachers for facial age estimation. Due to the nonstationary patterns of the facial-aging process, the relative order of age labels provides more reliable information than exact age values for facial age estimation. Thus, the first teacher is a novel ranking method capturing the ordinal relation among age labels. Especially, it formulates the ordinal relation learning as a task of recovering the original ordered sequences from shuffled ones. The second teacher adopts the same model as the student that treats facial age estimation as a multiclass classification task. The proposed method leverages the intermediate representations learned by the first teacher and the softened outputs of the second teacher as supervisory signals to improve the training procedure and final performance of the compact student for facial age estimation. Hence, the proposed knowledge distillation approach is capable of distilling the ordinal knowledge from the ranking model and the dark knowledge from the multiclass classification model into a compact student, which facilitates the implementation of facial age estimation on platforms with limited memory and computation resources, such as mobile and embedded devices. Extensive experiments involving several famous data sets for age estimation have demonstrated the superior performance of our proposed method over several existing state-of-the-art methods. [ABSTRACT FROM AUTHOR]

• We propose a novel federated learning scheme using knowledge distillation technology for heterogeneous model architecture called MHAT. • We train an auxiliary model on the server to realize information aggregation, which significantly improves the speed of model convergence while maintaining acceptable model convergence accuracy. • We conduct various experiments to validate our scheme achieves a good performance. Federated Learning allows multiple participants to jointly train a global model while guaranteeing the confidentiality and integrity of private datasets. However, current server aggregation algorithms for federated learning only focus on model parameters, resulting in heavy communication costs and low convergence speed. Most importantly, they are unable to handle the scenario wherein different clients hold different local models with various network architectures. In this paper, we view these challenges from an alternative perspective: we draw attention to what should be aggregated and how to improve convergence efficiency. Specifically, we propose MHAT, a novel model-heterogenous aggregation training federated learning scheme which exploits a technique of Knowledge Distillation (KD) to extract the update information of the heterogenous model of all clients and trains an auxiliary model on the server to realize information aggregation. MHAT relaxes clients from fixing on an unified model architecture and significantly reduces the required computing resources while maintaining acceptable model convergence accuracy. Various experiments verify the effectiveness and applicability of our proposed scheme. [ABSTRACT FROM AUTHOR]

• A knowledge sharing framework for collaborative learning in distributed system. • A P2P federated distillation mechanism for decentralized federated learning. • A reinforcement learning scheme for blockchain enhancement. With the development of Internet of Things (IoT) and Artificial Intelligence (AI) technologies, smart services have penetrated into every aspect of our daily lives, including the medical treatment and healthcare fields. However, due to security and privacy issues, medical data cannot be easily shared, which may lead to the situation of the so-called data silos. Challenges in existing approaches when building medical data sharing models can be summarized as: i) It is very challenging to ensure that the privacy of the medical data is protected and to identify the ownership of medical data; ii) Their models always result in poor performance or have the problem of excessive communication overhead due to the large amount of model parameters; iii) The current scenario of combining federated learning and blockchain generally ignores the load on nodes, which may easily lead to a lack of efficiency and fairness during the consensus process. In this study, we propose a Federated Distillation and Blockchain empowered Secure Knowledge Sharing (FDBC-SKS) model, which transforms the data sharing problem into a collaborative model knowledge sharing problem, aiming to provide a lightweight distributed deep learning framework in Internet of Medical Things (IoMT) environments. A peer-to-peer federated distillation mechanism is designed to enable a decentralized federated learning with better model flexibility and less communication consumption, based on the better knowledge utilization from each local model. A reinforcement learning enhanced consensus mechanism for blockchain is devised to improve the model convergence performance, alleviate the problem of low computational efficiency, while enhancing the fairness in terms of node load balancing during the node selection and block generation process. Experiment and evaluation results based on two real-world datasets demonstrate the usefulness of our proposed model toward secure and effective data sharing in IoMT oriented smart application development compared with other similar methods. [ABSTRACT FROM AUTHOR]

Knowledge distillation (KD) aims to build a lightweight deep neural network model under the guidance of a large-scale teacher model for model simplicity. Despite improved model efficiency through the KD technique, the performance gap between a teacher model and the trained student model remains significant. This is because the knowledge of the teacher model is not effectively transferred to the student model since the mapping landscape of the large-scale teacher model is not fully explored. To tackle this research gap, we propose a novel H ybrid M ix-up C ontrastive K nowledge D istillation (HMCKD) approach, which facilitates a thorough and reliable mapping solution space exploration in order to significantly improve the performance of the student model. Specifically, we design a hybrid mixing strategy, including image-level mixing and feature-level mixing, to form a smoother mapping landscape as a means to provide a stronger guidance in order to embed its richer dark knowledge from a teacher model to its student model. Additionally, we apply two other strategies, including contrastive learning and top-k guided selection, in order to ensure more effective knowledge transferability from the teacher model. Extensive experiments have proved that our proposed HMCKD approach outperforms state-of-the-art knowledge distillation methods when tested on 6 publicly available datasets, such as CIFAR-100, CIFAR-100-C, STL-10, SVHN, TinyImageNet, and ImageNet. Particularly on CIFAR-100 dataset, the average accuracy of students using HMCKD increased by 1.47%. Further, both the visualization results and similarity quantifications have confirmed the narrowed knowledge gap between the teacher and student models. Our source code is available at https://github.com/lambett/HMCKD. [ABSTRACT FROM AUTHOR]

In recent years, the study of real-world graphs has revealed their inherent heterogeneity, prompting growing research interest in heterogeneous graphs. Characterized by diverse node and relation types, heterogeneous graphs have led to the development of heterogeneous graph neural networks, which possess the remarkable ability of modeling such heterogeneity. Consequently, researchers have embraced these networks, applying them in various domains. A prevalent approach is using meta-path based methods in heterogeneous graph neural networks. However, a significant limitation arises from the fact that such methods tend to overlook vital attribute information within intermediate nodes and disregard relevant semantics across various meta-paths. To address the above limitations, we propose a new model named HGNN-MRCS. Specifically, HGNN-MRCS incorporates three key components, i.e., a relation aware module to encapsulate the attribute information of the intermediate nodes; a meta-path aware technique to facilitate learning of semantic information of each meta-path and enable higher-order representation learning; and a knowledge distillation strategy to learn relevant semantics across meta-paths and fuse them. Experimental results on four real-world datasets demonstrate the superior performance of this work over the SOAT methods. The source codes of this work are available at https://github.com/ZZY-GraphMiningLab/HGNN-MRCS. [ABSTRACT FROM AUTHOR]

Self-Knowledge Distillation (Self-KD), a technique that enables neural networks to learn from themselves, often relies on auxiliary modules or networks to generate supervisory signals for training. However, this approach incurs significant additional resource costs. Moreover, incorporating auxiliary classifiers within the network architecture creates a capacity mismatch when distilling knowledge from deep to shallow classifiers. This paper proposes a concise and efficient Self-KD method called Neighbor Self-Knowledge Distillation (NSKD), which introduces teacher assistants into the Self-KD by adding auxiliary classifiers to the shallow part of the network to construct distillations of multiple neighboring student-teacher assistant combinations to reduce the mismatch between the students' and teachers' abilities. During distillation, NSKD utilizes only the soft labels generated by each classifier and corresponding ground truth labels as supervisory signals, minimizing resource consumption. NSKD enables neighboring modules to learn from each other through neighboring distillation, enhancing overall network performance. Experimental results on five network models and seven popular datasets demonstrate the superiority of NSKD over other state-of-the-art Self-KD methods. Notably, NSKD achieves average accuracy improvements of 2.26%, 2.32%, and 2.4% on the CIFAR100, TinyImageNet, and fine-grained visual classification datasets. [ABSTRACT FROM AUTHOR]

Sequential models based on deep learning are widely used in sequential recommendation task, but the increase of model parameters results in a higher latency in the inference stage, which limits the real-time performance of the model. In order to make the model strike a balance between efficiency and effectiveness, the knowledge distillation technology is adopted to transfer the pre-trained knowledge from the large teacher model to the small student model. We propose a multi-stage knowledge distillation method based on interest knowledge, including interest representation knowledge and interest drift knowledge. In the process of knowledge transfer, expert distillation is designed to transform the knowledge dimension of student model to alleviate the loss of original knowledge information. Specially, curriculum learning is introduced for multi-stage knowledge learning, which further makes the teacher model effectively transfer the knowledge to the student model with limited ability. The proposed method on three real-world datasets including MovieLen-1M, Amazon Game and Steam datasets. The experimental results demonstrate that our method is superior to the other compared distillation method significantly and multi-stage learning makes the student model achieve the knowledge step by step for improvement. [ABSTRACT FROM AUTHOR]

Recommendation services play a pivotal role in financial decision-making and multimedia content services, as they suggest investment operations and personalized items to users, typically characterized by multi-modal features such as visual, textual, and acoustic attributes. Graph Neural Networks (GNNs), demonstrating the immense potential for graph representation learning and recommendation systems, are capable of learning user/item embeddings by taking into account the graph topological structure and the multi-modal node features. Yet, a substantial number of multi-modal recommendation studies have seemingly ignored the inherent bias among different modalities during feature fusion, consequently leading to sub-optimal embeddings for items with multi-modal features. To mitigate this issue, we propose a novel multi-modal recommendation framework that integrates GNNs with deep mutual learning techniques, termed GNNMR. GNNMR uses the mutual knowledge distillation technique to collaboratively train multiple uni-modal bipartite user-item graphs. Each GNN is trained specifically on the uni-modal user-item bipartite graph, which is separated from the original multi-modal user-item bipartite graph, to generate uni-modal embeddings. These uni-modal embeddings then act as mutual supervision signals, allowing the model to uncover and synchronize the latent semantic relationships among different modalities. Subsequently, the model can conduct inference in an ensemble manner, leveraging uni-modal embeddings from diverse modalities. Experimental results on two real-world datasets demonstrate that the proposed GNNMR outperforms other multi-modal recommendation methods in the Top-K recommendation task. [ABSTRACT FROM AUTHOR]

Deep neural networks perform well but require high-performance hardware for their use in real-world environments. Knowledge distillation is a simple method for improving the performance of a small network by using the knowledge of a large complex network. Small and large networks are referred to as student and teacher models, respectively. Previous knowledge distillation approaches perform well in a relatively small teacher network (20–30 layers) but poorly in large teacher networks (50 layers). Here, we propose an approach called block change learning that performs local and global knowledge distillation by changing blocks comprised of layers. The method focuses on the knowledge transfer without losing information in a large teacher model, as the approach considers intra-relationships between layers using local knowledge distillation and inter-relationships between corresponding blocks. The results are demonstrated this approach as superior to state-of-the-art methods using feature extraction datasets (Market1501 and DukeMTMC-relD) and object classification datasets (CIFAR-100 and Caltech256). Furthermore, we showed that the performance of the proposed approach was superior to that of a fine-tuning approach using pretrained models. [ABSTRACT FROM AUTHOR]

• A novel ECG Diagnosis model for mobile devices is proposed based on dynamic neural network. • The model has high inference efficiency and device versatility opposed to static models. • Two multi-classifier collaborative learning methods are proposed. • A sigmoid-based early exit criterion is designed for multi-label ECG diagnosis. Mobile health devices with automatic electrocardiogram diagnosis models facilitate long-term cardiac monitoring and enhance the sensitivity of detecting paroxysmal cardiovascular disease. However, these devices often encounter limitations in terms of computational power and battery life. Moreover, iterations of mobile health device versions may result in changes to their computational limitations, leading to additional costs for model design and training. Therefore, it is crucial to develop lightweight models that enable efficient inference under resource constraints and are compatible with various devices. To address these challenges, we propose a resource-efficient and device-adaptive electrocardiogram diagnosis model based on a multi-classifier dynamic neural network. This model achieves efficient computation by exiting simple samples early and can adapt to the computational limits of various devices by adjusting predefined thresholds. Furthermore, we propose two novel approaches, namely rethinking structure and feature-based knowledge distillation, which enhance the collaboration among classifiers and improve the overall performance. The proposed model is evaluated using three electrocardiogram datasets: PTB-XL (AUC = 0.9706), ICBEB2018 (AUC = 0.964), and MIT-BIH arrhythmia (ACC = 0.994, F1 = 0.961). The results consistently demonstrate the superiority of the proposed model over the comparative models across all three datasets, with average improvements of 2.4%, 0.5%, and 1.3% respectively. [ABSTRACT FROM AUTHOR]

As a long-time wish, researchers always want to find a way to fuse human knowledge directly into machine models that can fulfill intelligent tasks. Existing researches attempted to approach this goal by manually labeling data or writing simple rules without considering the human's inherent ability to estimate uncertainty. These approaches cannot take full advantage of human abilities and hence make the overall system inefficient. In this paper, we propose a novel approach INFER that can distill knowledge from humans in the form of rules with uncertainty. Firstly, we propose a new paradigm of providing human intelligence by expressing human knowledge as uncertain rules. After obtaining the set of rules, we design a rule aggregation model and a classification model, these two models are jointly trained in a knowledge distillation framework with the uncertainty knowledge. We further improve the distillation process with a curriculum learning-based training method. Through these, we can directly extract knowledge from inaccurate human knowledge. Empirical results on four different tasks demonstrate our proposed INFER approach can significantly improve model performance. Furthermore, the proposed uncertainty can provide more information and be effective in refining rules. • We propose a new paradigm INFER that can effectively use human intelligence. • We model the uncertain rules that contain more informative knowledge. • We design a new knowledge distillation algorithm to extract knowledge from the obtained rule set. • Experimental results demonstrate INFER outperforms alternative benchmarks. [ABSTRACT FROM AUTHOR]

Data-free knowledge distillation (DFKD) is a widely-used strategy for Knowledge Distillation (KD) whose training data is not available. It trains a lightweight student model with the aid of a large pretrained teacher model without any access to training data. However, existing DFKD methods suffer from inadequate and unstable training process, as they do not adjust the generation target dynamically based on the status of the student model during learning. To address this limitation, we propose a novel DFKD method called CuDFKD. It teaches students by a dynamic strategy that gradually generates easy-to-hard pseudo samples, mirroring how humans learn. Besides, CuDFKD adapts the generation target dynamically according to the status of student model. Moreover, we provide a theoretical analysis of the majorization minimization (MM) algorithm and explain the convergence of CuDFKD. To measure the robustness and fidelity of DFKD methods, we propose two more metrics, and experiments shows CuDFKD has comparable performance to state-of-the-art (SOTA) DFKD methods on all datasets. Experiments also present that our CuDFKD has the fastest convergence and best robustness over other SOTA DFKD methods. [ABSTRACT FROM AUTHOR]

In recent years, numerous high accuracy methods have been developed for classifying activities using multi inertial sensors. Despite their reliability and precision, they suffer from high computational cost and which make them improper for deploying in edge devices that are limited resources. This paper addresses this drawback by employing a knowledge distillation (KD) paradigm which maps tri-axial multi signals into single axis signals, thus; it can recognize activities with fewer number of signals and consequently less computation. In this method, a big teacher model is trained in advanced with three IMU sensors each of which have tri-axial signals. Then, a small student model is trained with just one of the axes of these sensors under monitoring of teacher which reduces the number of signals. Tucker decomposition is also exploited in order to improve KD performance by separating a core tensor from feature maps that has more informative knowledge. Evaluation of our method on REALDISP dataset demonstrates that the student model could achieve accuracy of 92.90% with much less complexity making it suitable for embedded devices. Moreover, it outperforms in comparison to other state-of-the-art KD approaches. • Employing the knowledge distillation paradigm in human activity recognition. • Using combination of response-based and feature-based in knowledge distillation. • Applying Tucker decomposition on tensors of feature maps to improve performance. • Training the teacher and the student via three axes and single axis IMU, respectively. • Competitive results of model on REALDISP make it capable to deploy in edge devices. [ABSTRACT FROM AUTHOR]

Knowledge distillation is a method to train a lightweight network by transferring class probability knowledge from a cumbersome teacher network. However, transferring only the class probability knowledge would limit the distillation performance. Therefore, several approaches have been proposed to transfer the teacher's knowledge at the feature map level. In this paper, we revisit the feature distillation method and have found that the larger the teacher's architecture/capacity becomes, the more difficult it is for the student to imitate. Thus, the feature distillation method is unable to achieve its full potential. To address this, a novel end-to-end distillation framework, termed Customizing a Teacher for Feature Distillation (CTFD), is proposed to train a teacher to be more compatible with its student. In addition, we apply the customized teacher to three feature distillation methods. Moreover, data augmentation is used as a trick to train the student to improve its generalization performance. Extensive empirical experiments and analyses are conducted on three computer vision tasks, including image classification, transfer learning, and object detection, to substantiate the effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]

Cross-technology interference sources poses a great challenge for improving throughputs of wireless local area networks (WLAN) and wireless interference signal recognition (WISR) can provide a precondition for mitigating this problem. The quadruple generative adversarial network (QGAN) has shown its prevailing performance for specific emitter identification (SEI). In this paper, an enhanced collaborative learning mechanism is proposed to improve QGAN's performance for WISR. ACGAN is involved in the Improved-QGAN architecture to substitute original GAN, and loss functions are further optimized for generative, representation and classification sub-networks. Besides, a lightweight model based on knowledge distillation (KD) is presented to reduce memory consumption and computational complexity at inference phase. Numerical results indicate that the proposed Improved-QGAN outperforms the other baseline algorithms both on the experimental dataset and benchmark dataset. [ABSTRACT FROM AUTHOR]