Search

Large deep learning models are impressive, but they struggle when real-time data is not available. Few-shot class-incremental learning (FSCIL) poses a significant challenge for deep neural networks to learn new tasks from just a few labeled samples without forgetting the previously learned ones. This setup can easily leads to catastrophic forgetting and overfitting problems, severely affecting model performance. Studying FSCIL helps overcome deep learning model limitations on data volume and acquisition time, while improving practicality and adaptability of machine learning models. This paper provides a comprehensive survey on FSCIL. Unlike previous surveys, we aim to synthesize few-shot learning and incremental learning, focusing on introducing FSCIL from two perspectives, while reviewing over 30 theoretical research studies and more than 20 applied research studies. From the theoretical perspective, we provide a novel categorization approach that divides the field into five subcategories, including traditional machine learning methods, meta learning-based methods, feature and feature space-based methods, replay-based methods, and dynamic network structure-based methods. We also evaluate the performance of recent theoretical research on benchmark datasets of FSCIL. From the application perspective, FSCIL has achieved impressive achievements in various fields of computer vision such as image classification, object detection, and image segmentation, as well as in natural language processing and graph. We summarize the important applications. Finally, we point out potential future research directions, including applications, problem setups, and theory development. Overall, this paper offers a comprehensive analysis of the latest advances in FSCIL from a methodological, performance, and application perspective. • In-depth survey on FSCIL methods, applications, performance. • Categorizing FSCIL into five subcategories for clear analysis. • Evaluating FSCIL research on benchmarks for strengths, weaknesses. • FSCIL applications in computer vision, NLP, and graph. [ABSTRACT FROM AUTHOR]

Skin cancer is one of the most serious threats to human health among skin lesions. Computer-aided diagnosis methods can assist patients in identifying and detecting skin lesion types early, thereby enabling corresponding treatments. In this paper, we propose a dual-branch neural network model Conformer with Residual Cosine Similarity Attention and Bidirectional Convolutional fusion strategy, named RCSABC-Conformer. The core of this network structure comprises three parts: a Convolutional Neural Network (CNN) branch with Residual Cosine Similarity Attention (RCSA), a Transformer branch, and a Feature Couple Unit with Bidirectional Convolutional strategy (BC-FCU). The RCSA module calculates the cosine similarity value between the feature map generated by the convolutional operation and the feature map of the residual edge to assess whether their semantic information is similar. The semantic information of similar parts is weighted by exponential normalization to enhance the network's memory of similar features of the same type of skin lesion. The BC-FCU module interactively fuses local features and global representations of skin lesion images with different resolutions in the two branches. Specifically, when the global representations is integrated into local features, we introduce a new bidirectional convolution strategy to extract the feature map from both forward and backward directions, and then select the element with the smaller feature value from the two directions to fuse into local features. In this way, we can minimize the interference of the artifact features extracted by the Transformer branch on the CNN branch. In addition, taking advantage of the Transformer branch's capacity to construct global representations, our model can learn contextual semantic information of normal skin and lesion areas to enhance model robustness. We conducted experiments on three datasets, consisting of clinical and dermoscopic skin lesion images, as well as a hybrid of both. The experimental results show that RCSABC-Conformer outperforms both advanced and classical classification methods in terms of classification accuracy across all three datasets, without requiring an increase in the number of parameters and computational complexity. Compared with the baseline model, the classification accuracy of our proposed method improves by 2.40%, 5.39%, and 4.44% on the three datasets, respectively. To the best of our knowledge, this is the first study to apply an interactive fusion dual-branch network for multi-disease classification on different modalities of skin lesion databases. Code will be available at https://github.com/AlenLi817/RCSABC-Conformer. [ABSTRACT FROM AUTHOR]

Skin cancer is one of the most serious threats to human health among skin lesions. Computer-aided diagnosis methods can assist patients in identifying and detecting skin lesion types early, thereby enabling corresponding treatments. In this paper, we propose a dual-branch neural network model Conformer with Residual Cosine Similarity Attention and Bidirectional Convolutional fusion strategy, named RCSABC-Conformer. The core of this network structure comprises three parts: a Convolutional Neural Network (CNN) branch with Residual Cosine Similarity Attention (RCSA), a Transformer branch, and a Feature Couple Unit with Bidirectional Convolutional strategy (BC-FCU). The RCSA module calculates the cosine similarity value between the feature map generated by the convolutional operation and the feature map of the residual edge to assess whether their semantic information is similar. The semantic information of similar parts is weighted by exponential normalization to enhance the network's memory of similar features of the same type of skin lesion. The BC-FCU module interactively fuses local features and global representations of skin lesion images with different resolutions in the two branches. Specifically, when the global representations is integrated into local features, we introduce a new bidirectional convolution strategy to extract the feature map from both forward and backward directions, and then select the element with the smaller feature value from the two directions to fuse into local features. In this way, we can minimize the interference of the artifact features extracted by the Transformer branch on the CNN branch. In addition, taking advantage of the Transformer branch's capacity to construct global representations, our model can learn contextual semantic information of normal skin and lesion areas to enhance model robustness. We conducted experiments on three datasets, consisting of clinical and dermoscopic skin lesion images, as well as a hybrid of both. The experimental results show that RCSABC-Conformer outperforms both advanced and classical classification methods in terms of classification accuracy across all three datasets, without requiring an increase in the number of parameters and computational complexity. Compared with the baseline model, the classification accuracy of our proposed method improves by 2.40%, 5.39%, and 4.44% on the three datasets, respectively. To the best of our knowledge, this is the first study to apply an interactive fusion dual-branch network for multi-disease classification on different modalities of skin lesion databases. Code will be available at https://github.com/AlenLi817/RCSABC-Conformer. [ABSTRACT FROM AUTHOR]

Active learning alleviates labeling costs by selecting and labeling the most informative examples from an unlabeled pool. However, most existing active learning approaches estimate informativeness with uncalibrated confidence, resulting in unreliable informativeness estimation. These approaches generally ignored two significant issues caused by uncalibrated confidence methods. Firstly, the average uncalibrated confidence generated by modern neural networks is usually higher than the accuracy. Secondly, examples located near the decision boundaries are unstable during prediction when the target model updates parameters in the last several epochs, even throughout the training process. This phenomenon, caused by the forgetting characteristic of neural networks, has a significant impact on some specific models that estimate the informativeness by predicted probability vectors or pseudo labels. To address these issues, in this paper, we propose a novel active learning approach to reliably estimate informativeness with calibrated confidence. Specifically, we integrate the intermediate predictions for each unlabeled example, generated by the target model during the training process, to generate calibrated confidence. The calibrated confidence can capture a tendentious label from an indecisive subset of the class space. We show that the calibrated confidence with tendentiousness can maintain the ability of correct predictions. The empirical results demonstrate that our approach outperforms the state-of-the-art active learning methods on image classification tasks. • The active learning algorithm selects unstable examples by calibrated confidence.. • The confidence calibration method only requires training a single model. • Extensive experiments show the competitive performance of the proposed approach. [ABSTRACT FROM AUTHOR]

Adversarial training has become the mainstream method to boost adversarial robustness of deep models. However, it often suffers from the trade-off dilemma, where the use of adversarial examples hurts the standard generalization of models on natural data. To study this phenomenon, we investigate it from the perspective of spatial attention. In brief, standard training typically encourages a model to conduct a comprehensive check to input space. But adversarial training often causes a model to overly concentrate on sparse spatial regions. This reduced tendency is beneficial to avoid adversarial accumulation but easily makes the model ignore abundant discriminative information, thereby resulting in weak generalization. To address this issue, this paper introduces an Attention-Enhanced Learning Framework (AELF) for robustness training. The main idea is to enable the model to inherit the attention pattern of standard pre-trained model through an embedding-level regularization. To be specific, given a teacher model built on natural examples, the embedding distribution of teacher model is used as a static constraint to regulate the embedding outputs of the objective model. This design is mainly supported with that the embedding feature of standard model is usually recognized as a rich semantic integration of input. For implementation, we present a simplified AELFs that can achieve the regularization with single cross entropy loss via the parameter initialization and parameter update strategy. This avoids the extra consistency comparison operation between embedding vectors. Experimental observations verify the rationality of our argument, and experimental results demonstrate that it can achieve remarkable improvements in generalization under the high-level robustness. [ABSTRACT FROM AUTHOR]

Deep neural networks tend to suffer from the overfitting issue when the training data are not enough. In this paper, we introduce two metrics from the intra-class distribution of correct-predicted and incorrect-predicted samples to provide a new perspective on the overfitting issue. Based on it, we propose a knowledge distillation approach without pretraining a teacher model in advance named Tolerant Self-Distillation (TSD) for alleviating the overfitting issue. It introduces an online updating memory and selectively stores the class predictions of the samples from the past iterations, making it possible to distill knowledge across the iterations. Specifically, the class predictions stored in the memory bank serve as the soft labels for supervising the samples from the same class for the current iteration in a reverse way, i.e. the correct-predicted samples are supervised with the incorrect predictions while the incorrect-predicted samples are supervised with the correct predictions. Consequently, the premature convergence issue caused by the over-confident samples would be mitigated, which helps the model to converge to a better local optimum. Extensive experimental results on several image classification benchmarks, including small-scale, large-scale, and fine-grained datasets, demonstrate the superiority of the proposed TSD. [ABSTRACT FROM AUTHOR]

An increasing need of running Convolutional Neural Network (CNN) models on mobile devices encourages the studies on efficient and lightweight neural network model. In this paper, an Inverse Residual Multi-Branch Network named IremulbNet is proposed to solve the problem of insufficient classification accuracy in existing lightweight network models. The core module of this model is to reconstruct an inverse residual structure, in which a special feature fusion method, multi-branch feature extraction, and depthwise separable convolution techniques are used to improve the classification accuracy. The performance of model is tested using image databases. Experimental results show that for the fine-grained image dataset Imagenet-woof, IremulbNet achieved 10.9%, 12.2%, and 15.3% higher accuracy than that of MobileNet V3, ShuffleNet V2, and PeleeNet, respectively. Moreover, it can reduce inference time (GPU) about 42.09% and 75.56% compared to classic ResNet50 and DenseNet121. • A lightweight IremulbNet is proposed to achieve high accuracy image recognition with a small number of parameters and operations. • A special feature fusion method, multi-branch feature extraction, and depthwise separable convolution are used to improve the performance of the network. • A significant improvement in recognition accuracy can be obtained in image classification tasks with low computation load. [ABSTRACT FROM AUTHOR]

• As the steep switching device has double-gate, the threshold voltage can be adaptively controlled. • The adjustment of the threshold can be employed to mitigate the accuracy deterioration. • Off current of proposed integrate and fire neuron circuit decreases significantly. • Adjustable threshold of the neuron circuit can achieve a high inference accuracy. Deep learning has shown impressive capabilities in tasks like speech recognition and image classification. However, modern deep neural networks often demand a significant number of weights and extensive computational resources, creating efficiency challenges for applications on edge devices. To address these issues, researchers have introduced deep spiking neural networks (DSNNs) that leverage specialized hardware for synapses and neurons. DSNNs offer a potential solution by improving efficiency in edge-device applications. In this paper, the hardware based DSNN with integrate and fire neuron using steep switching device was investigated. We propose integrate and fire neuron using steep switching device to implement rate coding as input encoding method. Because the steep switching device has double-gate, the threshold voltage of the neuron circuits can be adaptively controlled, which changes the rates of input pulse. Hence, the adjustment of the threshold of neuron can be employed to mitigate the accuracy deterioration resulting from the transformation from deep neural networks (DNNs) to DSNNs. In addition, the off-current of proposed integrate and fire neuron circuit decreases significantly as the steep switching device has steep subthreshold swing. A system simulation of a hardware based DSNN shows that the adjustable threshold of the neuron circuit can achieve a high inference accuracy of 98.36 % which is comparable to that obtained with software based DNN. [ABSTRACT FROM AUTHOR]

In this paper, we present a novel dimensionality reduction term (i.e. TripletPCA and ContrastivePCA) designed to enhance the robustness of pair-based loss functions in metric learning against image corruptions. Notably, our approach achieves this without the need for any data preprocessing steps. Our method can be seamlessly integrated into existing models, such as a block with Plug & Play framework. To the best of our knowledge, our TripletPCA and ContrastivePCA represent the first attempts to incorporate dimensionality reduction directly into pair-based metric learning losses for end-to-end metric learning. By projecting image features into low-dimensional vectors, our proposed loss functions effectively retain the essential components of images while mitigating the impact of corrupted features. Consequently, our metric learning loss function accurately computes feature distances through the projection. Experimental results demonstrate that our dimensionality reduction term can be easily incorporated into various types of existing deep neural networks. This integration leads to a substantial improvement in performance on standard benchmark datasets for corrupted image classification tasks. On several corruption datasets, we achieve an average performance improvement of 10.55% compared to existing baseline methods. Our code is available at https://github.com/Juryun/TDRL. [ABSTRACT FROM AUTHOR]

Spiking neural networks (SNNs) have garnered significant attention due to their ultra-high-speed and ultra-low-power operation, rendering them suitable for a range of energy-efficient applications. However, their performance in image generation is relatively mediocre, resulting in a lack of an effective generative replay mechanism to handle sequential probabilistic representations of multiple datasets. Consequently, this limitation often leads to catastrophic forgetting. This paper introduces a novel SNN lifelong learning framework namely Dynamic Lifelong learning with Spiking Generative Networks (DL-SGN), aimed at providing better generation ability while mitigating the problem of catastrophic forgetting in a continual learning scenario. DL-SGN comprises three key components: dynamic experts, a student, and an assistant. The dynamic expert is implemented as a dynamically expanding mixture model, with a proposed network expansion mechanism Dynamic Knowledge Adversarial Fusion (DKAF) facilitating the automatic handling of an increasing number of tasks. the student module adopts an SNN-based Variational Autoencoder (VAE) and classifier, leveraging accumulated knowledge from each expert. To enhance the student's ability to generalize to images, we introduce a discriminator as an assistant module trained using adversarial training. We validate the effectiveness of the proposed by conducting experiments on image generation and classification in a lifelong learning environment, and the results substantiate the effectiveness of DL-SGN. • The DL-SGN framework is proposed to addresses catastrophic forgetting in SNNs. • Our model is based on multiple dynamic experts and adversarial training. • Develop a dynamic knowledge adversarial fusion to expand expert module's capacity. • Employ an assistant module to enhance the student module's generalization ability. [ABSTRACT FROM AUTHOR]

Model pruning is a commonly used technique for compressing DNNs and reducing computation requirements to accelerate inference. However, the required granularity of pruning varies across different application scenarios, making it difficult and cumbersome to customize different pruning methods for each hardware or computing platform. Therefore, a unified framework is necessary to accommodate various levels of granularity of pruning. Furthermore, some available methods require additional fine-tuning or model retraining to restore accuracy, which can result in significant time costs. With this motivation, this paper proposes a Multi-Granularity Pruning Framework, namely MGPF, to obtain sparse models of different granularity without fine-tuning the remaining connections. Specifically, a deformable soft mask is introduced as the pruning initiator to achieve different levels of pruning granularity, such as weight pruning, channel pruning, and filter pruning, etc. The model parameters and soft masks are jointly trained, and we just apply L 1 regularization on soft masks for sparsity to ensure that the model can be repaired during training without fine-tuning or retraining. After pruning, the soft masks are absorbed into the model parameters in the form of element product without changing the model structures. Experimental results on three image classification benchmarks CIFAR-10/100 and ImageNet-1K demonstrate the effectiveness of our method for various CNN architectures, datasets, and pruning rates. Particularly, for ResNet-50 on ImageNet-1K, we achieve a higher accuracy under the pruning rate of 98% for unstructured pruning which leads the advanced method by 12%. [ABSTRACT FROM AUTHOR]

Deep Neural Networks have received considerable attention in recent years. As the complexity of network architecture increases in relation to the task complexity, it becomes harder to manually craft an optimal neural network architecture and train it to convergence. As such, Neural Architecture Search (NAS) is becoming far more prevalent within computer vision research, especially when the construction of efficient, smaller network architectures is becoming an increasingly important area of research, for which NAS is well suited. However, despite their promise, contemporary and end-to-end NAS pipeline require vast computational training resources. In this paper, we present a comprehensive overview of contemporary NAS approaches with respect to image classification, object detection, and image segmentation. We adopt consistent terminology to overcome contradictions common within existing NAS literature. Furthermore, we identify and compare current performance limitations in addition to highlighting directions for future NAS research. [ABSTRACT FROM AUTHOR]

Recently, convolutional neural networks (CNNs), which obtain powerful deep features in an end-to-end manner, have achieved powerful performance in remote sensing scene classification. However, the average or maximum pooling operations defined in the spatial domain and coarser-resolution features with high levels cannot extract reliable features and clear boundaries for small-scale targets in remote sensing scene imagery. This paper attempts to address these problems and proposes a multi-domain sematic high-order network for scene classification, named MSHNet. First, wavelet-spatial and detachable pooling blocks defined in the wavelet and spatial domains are inserted at the end of the convolutional block to learn the features in a more structural fusion manner. Second, multi-scale and multi-resolution semantic embedding modules are proposed to take full advantage of the complementary information and effectively maintain the spatial structures of learned deep features. Third, we employ a factorized bilinear coding approach to obtain compact and discriminative second-order features. MSHNet is thoroughly evaluated on two publicly available benchmarks, i.e., AID (Aerial Image Dataset) and NWPU-RESISC45 (Northwestern Polytechnical University-Remote Sensing Image Scene Classification 45). The extensive results illustrate that our MSHNet is competitive with other related multi-scale deep neural networks. [Display omitted] • Wavelet-spatial and detachable pooling maintain structures of learned features. • Factorized bilinear coding method with semantic features captures rich statistics. • MSHNet obtains competitive performance on NWPU-RESISC45 and AID datasets. [ABSTRACT FROM AUTHOR]

Traditional knowledge distillation approaches are typically designed for specific tasks, as they primarily distilling deep features from intermediate layers of a neural network, generally with ingeniously designed knowledge representations, which increase the difficulty of model development and interpretation. In contrast, we empirically show that a soft logits knowledge distillation is enough to significantly narrow the teacher–student model performance gap. In this paper, the S oft H ybrid K nowledge D istillation (SHKD) is proposed to generate representative features from the input and output location of models, and applicable to various teacher and student architectures. Specifically, the additional mixup task is first applied to the same batch of training images, which enriches the high-order feature information transferred by the model. And then, a top-k guided specification filter is added to the teacher model, to reduce the bias induced throughout the knowledge transfer process. Finally, the SHKD utilizes the same and different teacher–student model pairs of the deep neural network, which can be easily adjusted to the computational resources available in the current world. Extensive experiments are conducted to examine the efficiency, and show that the SHKD outperforms state-of-the-art knowledge distillation methods. With the SHKD, the high performance of models will be maintained, and the gap between the teacher model and the student model can be reduced, making the construction of deep neural networks more convenient to the unknown scenarios. The source code is available at https://github.com/lambett/SHKD. [ABSTRACT FROM AUTHOR]

Activation functions play an important role in deep learning models by introducing non-linearity to the output of a neuron, enabling the network to learn complex patterns and non-linear relationships in data and make predictions on more complex tasks. Deep learning models' most commonly used activation functions are Purelin, Sigmoid, Tansig, Rectified Linear Unit (ReLU), and Exponential Linear Unit (ELU), which exhibit limitations such as non-differentiability, vanishing gradients, and neuron inactivity with negative values. These functions are typically defined over a finite range, and their outputs are integers or real numbers. Using fractional calculus in designing activation functions for neural networks has shown promise in improving the performance of deep learning models in specific applications. These activation functions can capture more complex non-linearities than traditional integer-order activation functions, improving performance on tasks such as image classification and time series prediction. This paper focuses on deriving and testing linear and non-linear fractional-order forms of activation functions and their variants. The linear activation function includes Purelin. In contrast, the non-linear activation functions are Binary Step, Sigmoid, Tansig, ReLU, ELU, Gaussian Error Linear Unit (GELU), Hexpo, and their variants. Besides, the standard formula has been implemented and used in developing the fractional-order linear activation function. Furthermore, various expansion series, such as Euler and Maclaurin, have been used to design non-linear fractional-order activation functions and their variants. The single- and multi-layer fractional-order neural network models have been developed using the designed fractional-order activation functions. The simulation study uses developed fractional-order neural network models for predicting the Texas wind turbine systems' generated power. The performance of single and multi-layer fractional-order neural network models has been evaluated by changing the activation functions in the hidden layer while keeping the Purelin function constant at the output layer. Experiments on neural network models demonstrate that the designed fractional-order activation functions outperform traditional functions like Sigmoid, Tansig, ReLU, ELU, and their variants, effectively addressing limitations. • Activation functions are derived into fractional functions using fractional calculus. • The fractional neural network models have been made by adopting derived functions. • A Case study has been done on the Texas wind turbine with the models to predict the power. • The models' performance has been evaluated using the fractional functions. [ABSTRACT FROM AUTHOR]

Domain adaptation provides the possibility of utilizing the knowledge gained from an auxiliary domain to accomplish the task in another related domain. In this paper, a common feature representation of these domains is learned through a new proposed idea to reduce inter-domain differences more precisely which results in higher accuracy for unsupervised domain adaptation. To decrease these divergences, the proposed method finds a subspace that reduces data distribution discrepancy between domains. To achieve this goal, minimization of the divergence of the class conditional probability distributions is considered. By considering class conditional distributions, more discriminative information of data is preserved compared to applying marginal distributions. Also, an explicit parametric distribution of the source and target domains is considered to reduce the discrepancy between the two domains data which results in higher accuracy compared to the other relevant domain adaptation methods. Experimental studies on benchmark image classification tasks confirmed our assumptions and indicated the significant improvement given by the proposed method compared to the other state-of-the-art methods. • Aligning the feature distribution of the source and target domains explicitly. • Introducing a new loss function to impose the common distribution on the features. • Reducing the class conditional probability distribution distance between the domains. [ABSTRACT FROM AUTHOR]