Search

Attention is an important component of modern Convolutional Neural Networks (CNNs) that has been shown to improve baseline model performance for a wide variety of tasks. Attention has shown specific promise in the classification and segmentation of mammograms, but we have a limited understanding of why attention improves performance in these domains. In this paper, we present a robust comparison of different combinations of baseline models and attention methods at two resolutions for whole mammogram classification of masses and calcifications. We find that attention generally helps to improve baseline model performance. However, the extent of improvement is governed by a combination of model architecture and the statistical characteristics of the data. Specifically, we show that high amounts of pooling and model complexity may result in decreased performance for data with high variability. To better understand the effect of attention on mammogram classification, we used LayerCAM, a hierarchical Class Activation Map (CAM) approach, to visualize where the network pays attention in the input image. This research provides statistical evidence that attention can improve the correlation between model performance and LayerCAM activation in the region of interest (ROI). However, these correlations are weak and variable, indicating that improvements in model performance due to attention are not necessarily caused by increased model activation near the ROI. Overall, our work provides novel insights to help guide future efforts in incorporating attention-based mechanisms for mammogram classification. • Attention generally improves mammogram classification. • Newer attention methods don't consistently perform better. • Improvements due to attention depend on model architecture/complexity and characteristics of the image data. • Attention doesn't necessarily improve focus on the region of interest. • CNNs outperform transformers for high resolution mammogram classification. [ABSTRACT FROM AUTHOR]

In Semi-supervised Video Object Segmentation (SVOS), there is a critical emphasis on enhancing the memory and readout mechanisms for frame matching, especially in relation to temporal dynamics. Current methods predominantly use 2D CNNs for encoding video frames, which unfortunately neglects the crucial aspect of addressing temporal variations in individual frames and their associated masks during the encoding process. One potential solution would be to implement temporal models such as 3D CNNs instead of 2D CNNs, but this significantly increases computational requirements, making it impractical for real-world SVOS applications. In this paper, we introduce the Grouped Temporal Recalibration with Attention for Convolutional Encoders (G-TRACE), a novel plug-and-play module that is compatible with various existing SVOS frameworks. G-TRACE uses hierarchical memory-centric attention and integrates effortlessly with 2D CNNs, offering a novel approach to temporal modeling that operates orthogonally to traditional frame matching methods. Extensive evaluations on four widely-used benchmarks demonstrate that our method consistently delivers significant performance improvements over various baseline models. • Captures Temporal Variations: Effectively handles previously overlooked time-based changes in frame encoding. • Innovates Frame Encoding: Improves spatial and temporal context capture, potentially replacing self-attention mechanisms. • Hierarchical Memory Grouping: Introduces a two-tier strategy for efficient feature capture and memory utilization. • Enhances Feature Quality: Provides a richer understanding of video sequences through advanced memory-based attention. • Empirical Superiority: Demonstrates consistent performance improvements across key SVOS benchmarks. [ABSTRACT FROM AUTHOR]

Glaucoma is a progressive eye condition that can lead to permanent vision loss. Therefore, on-time detection of glaucoma is critical for making an effective treatment plan. In recent years, enormous attempts have been made to develop automated glaucoma classification systems using CNNs through images. In contrast, limited methods have been proposed for diagnosing different glaucoma stages. It is mainly owing to the lack of large publicly available labeled datasets. Also, fundus images exhibit a high inter-stage resemblance, redundant features and minute size variations of lesions, making the conventional CNNs difficult to classify multiple stages of glaucoma accurately. To address these challenges, this paper proposes a novel adapter and enhanced self-attention based CNN framework named AES-Net for effective classification of glaucoma stages. In particular, we propose a spatial adapter module on top of the backbone network for learning better feature representations and an enhanced self-attention module (ESAM) to capture global feature correlations among the relevant channels and spatial positions. The ESAM assists in capturing stage-specific and detailed-lesion features from the fundus images. Extensive experiments on two multi-stage glaucoma datasets indicate that our AES-Net surpasses CNN-based existing approaches. The Grad-CAM ++ visualization maps further confirm the effectiveness of our AES-Net. • A spatial adapter and enhanced self-attention based CNN, AES-Net, is proposed. • An enhanced self-attention module is proposed to learn class-specific features. • A comparative analysis with state-of-the-art attention mechanisms is performed. • Results on two datasets indicate the effectiveness of AES-Net over current methods. • The visualization results demonstrate the explainability and interpretability of the model. [ABSTRACT FROM AUTHOR]

Image recognition is an important task in computer vision with broad applications. In recent years, with the advent of deep learning, lightweight convolutional neural network (CNN) has brought new opportunities for image recognition, which allows high-performance recognition algorithms to run on resource-constrained devices with strong representation and generalization capabilities. This paper first presents an overview of several classical lightweight CNN models. Then, a comprehensive review is provided on recent image recognition techniques using lightweight CNN. According to the strategies applied to optimize image recognition performance, existing methods are classified into three categories: (1) model compression, (2) optimization of lightweight network, and (3) combining Transformer with lightweight network. In addition, some representative methods are tested on three commonly used datasets for performance comparison. Finally, technical challenges and future research trends in this field are discussed. • A comprehensive review of recent advances in lightweight CNN for image recognition. • Classification according to different optimization directions. • Present the evaluation results of different representative lightweight CNN methods. • Discuss future research trends of image recognition. [ABSTRACT FROM AUTHOR]

Although deep convolutional neural networks have achieved remarkable success in removing synthetic fog, it is essential to be able to process images taken in complex foggy conditions, such as dense or non-homogeneous fog, in the real world. However, the haze distribution in the real world is complex, and downsampling can lead to color distortion or loss of detail in the output results as the resolution of a feature map or image resolution decreases. Moreover, the over-stacking of convolutional blocks might increase the model complexity. In addition to the challenges of obtaining sufficient training data, overfitting can also arise in deep learning techniques for foggy image processing, which can limit the generalization abilities of the model, posing challenges for its practical applications in real-world scenarios. Considering these issues, this paper proposes a Transformer-based wavelet network (WaveletFormerNet) for real-world foggy image recovery. We embed the discrete wavelet transform into the Vision Transformer by proposing the WaveletFormer and IWaveletFormer blocks, aiming to alleviate texture detail loss and color distortion in the image due to downsampling. We introduce parallel convolution in the Transformer block, which allows for the capture of multi-frequency information in a lightweight mechanism. Such a structure reduces computational expenses and improves the effectiveness of the network. Additionally, we have implemented a feature aggregation module (FAM) to maintain image resolution and enhance the feature extraction capacity of our model, further contributing to its impressive performance in real-world foggy image recovery tasks. Through extensive experiments on real-world fog datasets, we have demonstrated that our WaveletFormerNet achieves superior performance compared to state-of-the-art methods, as shown through quantitative and qualitative evaluations of minor model complexity. Additionally, our satisfactory results on real-world dust removal and application tests showcase the superior generalization ability and improved performance of WaveletFormerNet in computer vision-related applications compared to existing state-of-the-art methods, further confirming our proposed approach's effectiveness and robustness. Our code is available at https://github.com/shengli666666/WaveletFormerNet. [Display omitted] • Combining Vision Transformer and wavelet transform alleviates image detail loss. • Parallel convolution captures frequency information in a lightweight mechanism. • The feature aggregation module enhances the model feature extraction capability. • Experimental results demonstrate WaveletFormerNet's superior dehazing performance. [ABSTRACT FROM AUTHOR]

Moving image recognition has become a well-explored problem in computer vision. However, it is difficult for the traditional convolutional neural network (CNN) model to effectively capture timing information in motion. For better use of video sequence features and to improve the accuracy of action recognition, Therefore, this paper proposes a Three-dimensional CNN (3DCNN) model based on Gated Recurrent Unit (GRU) with an attention mechanism. The model leverages 3DCNN for deep feature extraction from video frames, employs GRU to capture the temporal dynamics of feature sequences and incorporates an attention mechanism to emphasize key frames, which improves moving image recognition. Demonstrating superior accuracy in'Cross-Subject' and'Cross-View' evaluations, our model surpasses standard benchmarks with accuracies of 83.2% and 87.3% respectively. • We combined 3 DCNN and GRU to classify the sequences of the motion images. • The framework adds attention mechanisms to improve recognition capabilities. • Experimental results show that the action recognition accuracy is increased by 6%. [ABSTRACT FROM AUTHOR]

Wild Facial Expression Recognition (FER) task has been a long-standing challenge due to the various forms of uncertainty exist in expression data. When expression data is fed into a convolutional neural network (CNN), the model's estimated parameters also become uncertain. This uncertainty gives rise to concerns regarding the reliability of the recognition results. To quantify these uncertainties and achieve robust performance in the presence of noisy data, this paper introduces a novel model for Wild Facial Expression Recognition: the Bayesian Convolutional Neural Network with Perturbed Multi-Branch Structure (BPMB). This model aims to address uncertainty issues, enabling the network's decisions to become more deterministic with increasing training accuracy. Specifically, BPMB incorporates variational inference (VI) to introduce a probability distribution for the weights. A variational approximation to the true posterior is derived using Bayes by Backprop, involving two convolution operations: one for classification and another for quantifying uncertainty. Furthermore, an exploration is conducted into a lightweight multi-branch structure that leverages Dropout as a random generator to introduce perturbations during the training process, enhancing the model's robustness while extracting deep features. Extensive experiments validate the superiority of the proposed BPMB algorithm over the majority of existing mainstream algorithms on three widely utilized wild datasets. • An effective method for measuring uncertainty in facial expression recognition. • Variational inference derives a variational approximation to the true posterior. • Disturbance of multi branch structure enhances the robustness of the model while extracting deep features. • A study provides a probabilistic framework for convolutional neural networks. [ABSTRACT FROM AUTHOR]

Audio-visual saliency prediction (AVSP) is a task that aims to model human attention patterns in the perception of auditory and visual scenes. Given the challenges associated with perceiving and combining multi-modal saliency features from videos, this paper presents a multi-sensory framework for AVSP. This framework is designed to extract audio, motion and image saliency features and integrate them effectively, which can then serve as a general architecture for the AVSP task. To obtain multi-sensory information, we develop a three-stream encoder that extracts audio, motion and image saliency features. In particular, we utilize a pre-trained encoder with knowledge related to image saliency to extract saliency features for each frame. The image saliency features are then incorporated with motion features using a spatial attention module. For motion features, 3D convolutional neural networks (CNNs) like S3D are commonly used in AVSP models. However, these networks are unable to effectively capture the global motion relationship in videos. To tackle this problem, we incorporate Transformer- and MLP-based motion encoders into the AVSP models. To learn joint audio-visual representations, an audio-visual fusion block is exploited to enhance the correlation between audio and visual motion features under the supervision of a cosine similarity loss in a self-supervised manner. Finally, a multi-stage decoder integrates audio, motion and image saliency features to generate the final saliency map. We evaluate our methods on six audio-visual eye-tracking datasets. Experimental results demonstrate that our method achieves compelling performance compared to the state-of-the-art methods. The source code is available at https://github.com/oraclefina/MSPI. • A three-stream multisensory framework for audio-visual saliency prediction. • Image saliency model offers image saliency for feature fusion with dynamic saliency. • A self-supervised audio-visual fusion block fuses auditory and visual features. • Our method is examined with various motion encoders of 3D CNN, Transformer and MLP. [ABSTRACT FROM AUTHOR]

Recently, convolutional neural networks (CNNs), which obtain powerful deep features in an end-to-end manner, have achieved powerful performance in remote sensing scene classification. However, the average or maximum pooling operations defined in the spatial domain and coarser-resolution features with high levels cannot extract reliable features and clear boundaries for small-scale targets in remote sensing scene imagery. This paper attempts to address these problems and proposes a multi-domain sematic high-order network for scene classification, named MSHNet. First, wavelet-spatial and detachable pooling blocks defined in the wavelet and spatial domains are inserted at the end of the convolutional block to learn the features in a more structural fusion manner. Second, multi-scale and multi-resolution semantic embedding modules are proposed to take full advantage of the complementary information and effectively maintain the spatial structures of learned deep features. Third, we employ a factorized bilinear coding approach to obtain compact and discriminative second-order features. MSHNet is thoroughly evaluated on two publicly available benchmarks, i.e., AID (Aerial Image Dataset) and NWPU-RESISC45 (Northwestern Polytechnical University-Remote Sensing Image Scene Classification 45). The extensive results illustrate that our MSHNet is competitive with other related multi-scale deep neural networks. [Display omitted] • Wavelet-spatial and detachable pooling maintain structures of learned features. • Factorized bilinear coding method with semantic features captures rich statistics. • MSHNet obtains competitive performance on NWPU-RESISC45 and AID datasets. [ABSTRACT FROM AUTHOR]

Shadow detection is a challenging problem in computer vision due to the high variability in lighting conditions, object shapes, and scene layouts. Despite the positive results achieved by some existing technologies, the problem becomes particularly challenging with complex and heterogeneous images where shadow-casting objects coexist and shadows can have different depths, scales, and morphologies. As a result, more advanced and accurate solutions are still needed to deal with this type of complexities. To address these challenges, this paper proposes a novel deep learning model, called the Cross-Attentional Dual Decoder Network (CADDN), to improve shadow detection by using fine-grained image reconstruction features. Unlike other existing methods, the CADDN uses an innovative encoder-decoder architecture with two decoder segments that work together to reconstruct the input images and their corresponding shadow masks. In this way, the features used to reconstruct the original input image can be used to support the shadow detection process itself. The proposed model also incorporates a cross-attention mechanism to weight the most relevant features for detecting shadows and skip connections with noise to improve the quality of the transferred features. The experimental results, including several benchmark image datasets and state-of-the-art detection methods, demonstrate the suitability of the presented approach for detecting shadows in computer vision applications. [Display omitted] [ABSTRACT FROM AUTHOR]

Hypercomplex convolutional neural networks (HCNNs) have recently been used to improve deep learning architectures due to their ability to share weights across input channels and thus improve the cohesiveness of learned representations within the layers. Much work has been done to study the effect of hypercomplex representation in CNNs. However, to date, none of these models have used fully hypercomplex architectures, meaning that some of the layers in these earlier networks used conventional real-valued calculations that did not engage cross-channel weight sharing. Here, we introduce and study full hypercomplex CNNs by ensuring all layers perform hypercomplex calculations. For the earlier HCNNs, the real-valued layers are found in: (I) in the front end of the networks, (II) in the back end of the networks, and (III) in the residual blocks of the networks. The present research examines the performance of HCNNs when the modules mentioned above are replaced with hypercomplex equivalents. These representational networks have outperformed and shown state-of-the-art results compared to previous hypercomplex models (on some specific datasets). A disadvantage of these representational networks is that they consume high computational costs. To reduce this cost, novel separable hypercomplex networks (SHNNs) are proposed. They are created by factoring a quaternion convolutional module into two consecutive separable vectormap convolutional modules. As the successive layers of deep HCNNs perform local hierarchical grouping on increasingly abstract features, these groupings are responsible for the long-distance interaction problems found in the dense layers. To handle these problems, researchers have applied attention mechanisms in hypercomplex space. This paper offers a perspective on the basic concepts of representational networks and attention-based representational networks with the focus on (1) reviewing deep full HCNNs, including hypercomplex-based fully connected dense layer; (2) analyzing hypercomplex-based separable concept; (3) describing how the hypercomplex spatial layer of residual bottleneck block is replaced with an attention layer; and (4) providing a comprehensive summary comparing the performance of recent deep HCNNs, deep convolutional neural networks, and original attention based networks. We compare the performance of the networks in terms of accuracy and the number of trainable parameters on several image classification datasets. • Hypercomplex CNNs with and without attention introduce cross-channel weight sharing. • Cross-channel weight sharing enables discovery of a new class of input correlations. • Hypercomplex networks can be further optimized without loss of accuracy. • Toward this goal, separable hypercomplex operations are introduced. [ABSTRACT FROM AUTHOR]