Search

Attention is an important component of modern Convolutional Neural Networks (CNNs) that has been shown to improve baseline model performance for a wide variety of tasks. Attention has shown specific promise in the classification and segmentation of mammograms, but we have a limited understanding of why attention improves performance in these domains. In this paper, we present a robust comparison of different combinations of baseline models and attention methods at two resolutions for whole mammogram classification of masses and calcifications. We find that attention generally helps to improve baseline model performance. However, the extent of improvement is governed by a combination of model architecture and the statistical characteristics of the data. Specifically, we show that high amounts of pooling and model complexity may result in decreased performance for data with high variability. To better understand the effect of attention on mammogram classification, we used LayerCAM, a hierarchical Class Activation Map (CAM) approach, to visualize where the network pays attention in the input image. This research provides statistical evidence that attention can improve the correlation between model performance and LayerCAM activation in the region of interest (ROI). However, these correlations are weak and variable, indicating that improvements in model performance due to attention are not necessarily caused by increased model activation near the ROI. Overall, our work provides novel insights to help guide future efforts in incorporating attention-based mechanisms for mammogram classification. • Attention generally improves mammogram classification. • Newer attention methods don't consistently perform better. • Improvements due to attention depend on model architecture/complexity and characteristics of the image data. • Attention doesn't necessarily improve focus on the region of interest. • CNNs outperform transformers for high resolution mammogram classification. [ABSTRACT FROM AUTHOR]

In Semi-supervised Video Object Segmentation (SVOS), there is a critical emphasis on enhancing the memory and readout mechanisms for frame matching, especially in relation to temporal dynamics. Current methods predominantly use 2D CNNs for encoding video frames, which unfortunately neglects the crucial aspect of addressing temporal variations in individual frames and their associated masks during the encoding process. One potential solution would be to implement temporal models such as 3D CNNs instead of 2D CNNs, but this significantly increases computational requirements, making it impractical for real-world SVOS applications. In this paper, we introduce the Grouped Temporal Recalibration with Attention for Convolutional Encoders (G-TRACE), a novel plug-and-play module that is compatible with various existing SVOS frameworks. G-TRACE uses hierarchical memory-centric attention and integrates effortlessly with 2D CNNs, offering a novel approach to temporal modeling that operates orthogonally to traditional frame matching methods. Extensive evaluations on four widely-used benchmarks demonstrate that our method consistently delivers significant performance improvements over various baseline models. • Captures Temporal Variations: Effectively handles previously overlooked time-based changes in frame encoding. • Innovates Frame Encoding: Improves spatial and temporal context capture, potentially replacing self-attention mechanisms. • Hierarchical Memory Grouping: Introduces a two-tier strategy for efficient feature capture and memory utilization. • Enhances Feature Quality: Provides a richer understanding of video sequences through advanced memory-based attention. • Empirical Superiority: Demonstrates consistent performance improvements across key SVOS benchmarks. [ABSTRACT FROM AUTHOR]

Glaucoma is a progressive eye condition that can lead to permanent vision loss. Therefore, on-time detection of glaucoma is critical for making an effective treatment plan. In recent years, enormous attempts have been made to develop automated glaucoma classification systems using CNNs through images. In contrast, limited methods have been proposed for diagnosing different glaucoma stages. It is mainly owing to the lack of large publicly available labeled datasets. Also, fundus images exhibit a high inter-stage resemblance, redundant features and minute size variations of lesions, making the conventional CNNs difficult to classify multiple stages of glaucoma accurately. To address these challenges, this paper proposes a novel adapter and enhanced self-attention based CNN framework named AES-Net for effective classification of glaucoma stages. In particular, we propose a spatial adapter module on top of the backbone network for learning better feature representations and an enhanced self-attention module (ESAM) to capture global feature correlations among the relevant channels and spatial positions. The ESAM assists in capturing stage-specific and detailed-lesion features from the fundus images. Extensive experiments on two multi-stage glaucoma datasets indicate that our AES-Net surpasses CNN-based existing approaches. The Grad-CAM ++ visualization maps further confirm the effectiveness of our AES-Net. • A spatial adapter and enhanced self-attention based CNN, AES-Net, is proposed. • An enhanced self-attention module is proposed to learn class-specific features. • A comparative analysis with state-of-the-art attention mechanisms is performed. • Results on two datasets indicate the effectiveness of AES-Net over current methods. • The visualization results demonstrate the explainability and interpretability of the model. [ABSTRACT FROM AUTHOR]

Image recognition is an important task in computer vision with broad applications. In recent years, with the advent of deep learning, lightweight convolutional neural network (CNN) has brought new opportunities for image recognition, which allows high-performance recognition algorithms to run on resource-constrained devices with strong representation and generalization capabilities. This paper first presents an overview of several classical lightweight CNN models. Then, a comprehensive review is provided on recent image recognition techniques using lightweight CNN. According to the strategies applied to optimize image recognition performance, existing methods are classified into three categories: (1) model compression, (2) optimization of lightweight network, and (3) combining Transformer with lightweight network. In addition, some representative methods are tested on three commonly used datasets for performance comparison. Finally, technical challenges and future research trends in this field are discussed. • A comprehensive review of recent advances in lightweight CNN for image recognition. • Classification according to different optimization directions. • Present the evaluation results of different representative lightweight CNN methods. • Discuss future research trends of image recognition. [ABSTRACT FROM AUTHOR]

Although deep convolutional neural networks have achieved remarkable success in removing synthetic fog, it is essential to be able to process images taken in complex foggy conditions, such as dense or non-homogeneous fog, in the real world. However, the haze distribution in the real world is complex, and downsampling can lead to color distortion or loss of detail in the output results as the resolution of a feature map or image resolution decreases. Moreover, the over-stacking of convolutional blocks might increase the model complexity. In addition to the challenges of obtaining sufficient training data, overfitting can also arise in deep learning techniques for foggy image processing, which can limit the generalization abilities of the model, posing challenges for its practical applications in real-world scenarios. Considering these issues, this paper proposes a Transformer-based wavelet network (WaveletFormerNet) for real-world foggy image recovery. We embed the discrete wavelet transform into the Vision Transformer by proposing the WaveletFormer and IWaveletFormer blocks, aiming to alleviate texture detail loss and color distortion in the image due to downsampling. We introduce parallel convolution in the Transformer block, which allows for the capture of multi-frequency information in a lightweight mechanism. Such a structure reduces computational expenses and improves the effectiveness of the network. Additionally, we have implemented a feature aggregation module (FAM) to maintain image resolution and enhance the feature extraction capacity of our model, further contributing to its impressive performance in real-world foggy image recovery tasks. Through extensive experiments on real-world fog datasets, we have demonstrated that our WaveletFormerNet achieves superior performance compared to state-of-the-art methods, as shown through quantitative and qualitative evaluations of minor model complexity. Additionally, our satisfactory results on real-world dust removal and application tests showcase the superior generalization ability and improved performance of WaveletFormerNet in computer vision-related applications compared to existing state-of-the-art methods, further confirming our proposed approach's effectiveness and robustness. Our code is available at https://github.com/shengli666666/WaveletFormerNet. [Display omitted] • Combining Vision Transformer and wavelet transform alleviates image detail loss. • Parallel convolution captures frequency information in a lightweight mechanism. • The feature aggregation module enhances the model feature extraction capability. • Experimental results demonstrate WaveletFormerNet's superior dehazing performance. [ABSTRACT FROM AUTHOR]

Moving image recognition has become a well-explored problem in computer vision. However, it is difficult for the traditional convolutional neural network (CNN) model to effectively capture timing information in motion. For better use of video sequence features and to improve the accuracy of action recognition, Therefore, this paper proposes a Three-dimensional CNN (3DCNN) model based on Gated Recurrent Unit (GRU) with an attention mechanism. The model leverages 3DCNN for deep feature extraction from video frames, employs GRU to capture the temporal dynamics of feature sequences and incorporates an attention mechanism to emphasize key frames, which improves moving image recognition. Demonstrating superior accuracy in'Cross-Subject' and'Cross-View' evaluations, our model surpasses standard benchmarks with accuracies of 83.2% and 87.3% respectively. • We combined 3 DCNN and GRU to classify the sequences of the motion images. • The framework adds attention mechanisms to improve recognition capabilities. • Experimental results show that the action recognition accuracy is increased by 6%. [ABSTRACT FROM AUTHOR]

Wild Facial Expression Recognition (FER) task has been a long-standing challenge due to the various forms of uncertainty exist in expression data. When expression data is fed into a convolutional neural network (CNN), the model's estimated parameters also become uncertain. This uncertainty gives rise to concerns regarding the reliability of the recognition results. To quantify these uncertainties and achieve robust performance in the presence of noisy data, this paper introduces a novel model for Wild Facial Expression Recognition: the Bayesian Convolutional Neural Network with Perturbed Multi-Branch Structure (BPMB). This model aims to address uncertainty issues, enabling the network's decisions to become more deterministic with increasing training accuracy. Specifically, BPMB incorporates variational inference (VI) to introduce a probability distribution for the weights. A variational approximation to the true posterior is derived using Bayes by Backprop, involving two convolution operations: one for classification and another for quantifying uncertainty. Furthermore, an exploration is conducted into a lightweight multi-branch structure that leverages Dropout as a random generator to introduce perturbations during the training process, enhancing the model's robustness while extracting deep features. Extensive experiments validate the superiority of the proposed BPMB algorithm over the majority of existing mainstream algorithms on three widely utilized wild datasets. • An effective method for measuring uncertainty in facial expression recognition. • Variational inference derives a variational approximation to the true posterior. • Disturbance of multi branch structure enhances the robustness of the model while extracting deep features. • A study provides a probabilistic framework for convolutional neural networks. [ABSTRACT FROM AUTHOR]

Audio-visual saliency prediction (AVSP) is a task that aims to model human attention patterns in the perception of auditory and visual scenes. Given the challenges associated with perceiving and combining multi-modal saliency features from videos, this paper presents a multi-sensory framework for AVSP. This framework is designed to extract audio, motion and image saliency features and integrate them effectively, which can then serve as a general architecture for the AVSP task. To obtain multi-sensory information, we develop a three-stream encoder that extracts audio, motion and image saliency features. In particular, we utilize a pre-trained encoder with knowledge related to image saliency to extract saliency features for each frame. The image saliency features are then incorporated with motion features using a spatial attention module. For motion features, 3D convolutional neural networks (CNNs) like S3D are commonly used in AVSP models. However, these networks are unable to effectively capture the global motion relationship in videos. To tackle this problem, we incorporate Transformer- and MLP-based motion encoders into the AVSP models. To learn joint audio-visual representations, an audio-visual fusion block is exploited to enhance the correlation between audio and visual motion features under the supervision of a cosine similarity loss in a self-supervised manner. Finally, a multi-stage decoder integrates audio, motion and image saliency features to generate the final saliency map. We evaluate our methods on six audio-visual eye-tracking datasets. Experimental results demonstrate that our method achieves compelling performance compared to the state-of-the-art methods. The source code is available at https://github.com/oraclefina/MSPI. • A three-stream multisensory framework for audio-visual saliency prediction. • Image saliency model offers image saliency for feature fusion with dynamic saliency. • A self-supervised audio-visual fusion block fuses auditory and visual features. • Our method is examined with various motion encoders of 3D CNN, Transformer and MLP. [ABSTRACT FROM AUTHOR]

Recently, convolutional neural networks (CNNs), which obtain powerful deep features in an end-to-end manner, have achieved powerful performance in remote sensing scene classification. However, the average or maximum pooling operations defined in the spatial domain and coarser-resolution features with high levels cannot extract reliable features and clear boundaries for small-scale targets in remote sensing scene imagery. This paper attempts to address these problems and proposes a multi-domain sematic high-order network for scene classification, named MSHNet. First, wavelet-spatial and detachable pooling blocks defined in the wavelet and spatial domains are inserted at the end of the convolutional block to learn the features in a more structural fusion manner. Second, multi-scale and multi-resolution semantic embedding modules are proposed to take full advantage of the complementary information and effectively maintain the spatial structures of learned deep features. Third, we employ a factorized bilinear coding approach to obtain compact and discriminative second-order features. MSHNet is thoroughly evaluated on two publicly available benchmarks, i.e., AID (Aerial Image Dataset) and NWPU-RESISC45 (Northwestern Polytechnical University-Remote Sensing Image Scene Classification 45). The extensive results illustrate that our MSHNet is competitive with other related multi-scale deep neural networks. [Display omitted] • Wavelet-spatial and detachable pooling maintain structures of learned features. • Factorized bilinear coding method with semantic features captures rich statistics. • MSHNet obtains competitive performance on NWPU-RESISC45 and AID datasets. [ABSTRACT FROM AUTHOR]

Shadow detection is a challenging problem in computer vision due to the high variability in lighting conditions, object shapes, and scene layouts. Despite the positive results achieved by some existing technologies, the problem becomes particularly challenging with complex and heterogeneous images where shadow-casting objects coexist and shadows can have different depths, scales, and morphologies. As a result, more advanced and accurate solutions are still needed to deal with this type of complexities. To address these challenges, this paper proposes a novel deep learning model, called the Cross-Attentional Dual Decoder Network (CADDN), to improve shadow detection by using fine-grained image reconstruction features. Unlike other existing methods, the CADDN uses an innovative encoder-decoder architecture with two decoder segments that work together to reconstruct the input images and their corresponding shadow masks. In this way, the features used to reconstruct the original input image can be used to support the shadow detection process itself. The proposed model also incorporates a cross-attention mechanism to weight the most relevant features for detecting shadows and skip connections with noise to improve the quality of the transferred features. The experimental results, including several benchmark image datasets and state-of-the-art detection methods, demonstrate the suitability of the presented approach for detecting shadows in computer vision applications. [Display omitted] [ABSTRACT FROM AUTHOR]

• Novel Bilinear CNN based deep learning approach for facial expression recognition. • Extraction of second-order texture (Fine-grained) information by using dot-product & normalization layer. • Minimizing the intra-class variances within the expression. • It demonstrates the optimal balance between facial preprocessing and facial feature extraction component. • Experimenting on wild-environment (SFEW) dataset for real-world application. A facial expression recognition system has been proposed in this paper. The challenges of the facial expression recognition system lie due to low intra-class variance within a class of negative emotions, such as anger, disgust, and fear. A conventional Convolution Neural Network (CNN) model may extract discriminative features and has shown outstanding performance in various computer vision tasks. However, it is unable to extract the second-order feature information that demonstrates the interaction between features in the image of wild-environment datasets. This paper presents a novel method to solve the facial expression recognition problem by addressing several limitations concerning emotion recognition problems. A deep learning-based bilinear convolutional neural network framework has been proposed, termed an Fine-Grained Bilinear CNN (FgbCNN) model that consists of two branches with optimized CNN along with a normalization layer composed of batch-normalization, square-root normalization, L2-normalization, and drop-out layers. Here, local and holistic features have been aggregated using a dot-product layer to extract more discriminant features. Finally, experimenting with two wild-environments (SFEW 1.0 and SFEW 2.0) and two lab-controlled datasets (KDEF and CK+), it has been observed that the proposed model can minimize the intra-class variances and has attained outstanding performance compared to other state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Recently, convolutional neural network (CNN) has attracted much attention in different areas of computer vision, due to its powerful abstract feature representation. Visual object tracking is one of the interesting and important areas in computer vision that achieves remarkable improvements in recent years. In this work, we aim to improve both the motion and observation models in visual object tracking by leveraging representation power of CNNs. To this end, a motion estimation network (named MEN) is utilized to seek the most likely locations of the target and prepare a further clue in addition to the previous target position. Hence the motion estimation would be enhanced by generating a small number of candidates near two plausible positions. The generated candidates are then fed into a trained Siamese network to detect the most probable candidate. Each candidate is compared to an adaptable buffer, which is updated under a predefined condition. To take into account the target appearance changes, a weighting CNN (called WCNN) adaptively assigns weights to the final similarity scores of the Siamese network using sequence-specific information. Evaluation results on well-known benchmark datasets (OTB100, OTB50 and OTB2013) prove that the proposed tracker outperforms the state-of-the-art competitors. • This paper addresses the motion and observation models simultaneously. • This paper efficiently integrates the Siamese network into the sequence-specific updating. • Experimental results on public benchmark datasets verify that the proposed approach outperforms the state-of-the-art trackers. Unlabelled Image [ABSTRACT FROM AUTHOR]

Abstract Humans comprehend a natural scene at a single glance; painters and other visual artists, through their abstract representations, stressed this capacity to the limit. The performance of computer vision solutions matched that of humans in many problems of visual recognition. In this paper we address the problem of recognizing the genre (subject) in digitized paintings using Convolutional Neural Networks (CNN) as part of the more general dealing with abstract and/or artistic representation of scenes. Initially we establish the state of the art performance by training a CNN from scratch. In the next level of evaluation, we identify aspects that hinder the CNNs' recognition, such as artistic abstraction. Further, we test various domain adaptation methods that could enhance the subject recognition capabilities of the CNNs. The evaluation is performed on a database of 80,000 annotated digitized paintings, which is tentatively extended with artistic photographs, either original or stylized, in order to emulate artistic representations. Surprisingly, the most efficient domain adaptation is not the neural style transfer. Finally, the paper provides an experiment-based assessment of the abstraction level that CNNs are able to achieve. Highlights • We train from scratch a deep network to reach top performance in painting genre recognition. • We identify the style abstraction as the main limitation while trying to improve. • We test the potential for improvement by transfer from various domains. • Normal photographs (no adaptation) are as beneficial as style transfer. [ABSTRACT FROM AUTHOR]

Convolutional networks (ConvNets) are computationally expensive but well known for their performance on image data. One way to reduce their complexity is to explore inherited data sparsity. However, since the gradients involved in ConvNets require dynamic updates, applying data sparsity in the training step is not straightforward. Dictionary-based learning methods can be useful since they encode the original data in a sparse form. This paper proposes a new dictionary-based training paradigm for ConvNets by exploiting redundancy in the training data while keeping the distinctive features intact. The ConvNet is then trained on the reduced, sparse dataset. The new approach significantly reduces the training time without compromising accuracy. To the best of our knowledge, this is the first implementation of ConvNet on dictionary-based sparse training data. The proposed method is validated on three publicly available datasets â€"MNIST, USPS, and MNIST FASHION. The experimental results show a significant reduction of 4.5 times in the overall computational burden of vanilla ConvNet for all the datasets. Whereas the accuracy is intact at 97.21% for MNIST, 96.81% for USPS, and 88.4% for FASHION datasets. These results are comparable to state-of-the-art algorithms, such as ResNet-{18,34,50}, trained on the full training dataset. • ConvNets are powerful for images but computationally expensive. • Paper proposes dictionary-based training for ConvNets exploiting sparsity. • The proposed method preserves orthogonal features and reduces training time. • Results: 4.5x reduction in computational burden. • High accuracy: 97.21% MNIST, 96.81% USPS, 88.4% FASHION. [ABSTRACT FROM AUTHOR]

Hypercomplex convolutional neural networks (HCNNs) have recently been used to improve deep learning architectures due to their ability to share weights across input channels and thus improve the cohesiveness of learned representations within the layers. Much work has been done to study the effect of hypercomplex representation in CNNs. However, to date, none of these models have used fully hypercomplex architectures, meaning that some of the layers in these earlier networks used conventional real-valued calculations that did not engage cross-channel weight sharing. Here, we introduce and study full hypercomplex CNNs by ensuring all layers perform hypercomplex calculations. For the earlier HCNNs, the real-valued layers are found in: (I) in the front end of the networks, (II) in the back end of the networks, and (III) in the residual blocks of the networks. The present research examines the performance of HCNNs when the modules mentioned above are replaced with hypercomplex equivalents. These representational networks have outperformed and shown state-of-the-art results compared to previous hypercomplex models (on some specific datasets). A disadvantage of these representational networks is that they consume high computational costs. To reduce this cost, novel separable hypercomplex networks (SHNNs) are proposed. They are created by factoring a quaternion convolutional module into two consecutive separable vectormap convolutional modules. As the successive layers of deep HCNNs perform local hierarchical grouping on increasingly abstract features, these groupings are responsible for the long-distance interaction problems found in the dense layers. To handle these problems, researchers have applied attention mechanisms in hypercomplex space. This paper offers a perspective on the basic concepts of representational networks and attention-based representational networks with the focus on (1) reviewing deep full HCNNs, including hypercomplex-based fully connected dense layer; (2) analyzing hypercomplex-based separable concept; (3) describing how the hypercomplex spatial layer of residual bottleneck block is replaced with an attention layer; and (4) providing a comprehensive summary comparing the performance of recent deep HCNNs, deep convolutional neural networks, and original attention based networks. We compare the performance of the networks in terms of accuracy and the number of trainable parameters on several image classification datasets. • Hypercomplex CNNs with and without attention introduce cross-channel weight sharing. • Cross-channel weight sharing enables discovery of a new class of input correlations. • Hypercomplex networks can be further optimized without loss of accuracy. • Toward this goal, separable hypercomplex operations are introduced. [ABSTRACT FROM AUTHOR]

Stroke, particularly ischemic stroke, is a major cause of disability and one of the leading causes of adult mortality worldwide. Early and prompt management of stroke patients can reduce the severity of the disease. Doctors usually determine the severity of a stroke by focusing on the region of interest (ROI) in the MRI or CT scan images. An accurate and effective automatic image segmentation system can assist medical professionals as well as automatic detection and classification systems. Deep learning is the current advanced approach for dealing with machine learning and artificial intelligence. However, conventional deep learning requires a large amount of data for training, and the amount of labeled data in the medical field is limited. In this paper, we propose a few-shot learning strategy and integrate it with a base convolutional neural network model, which utilizes a self-attention mechanism to segment MRI for ischemic stroke. By combining the base model with self-attention, we can focus more on the ROI and disregard less important features. Additionally, the proposed system only selects slices with lesions and ignores unlesioned slices. This helps to improve efficiency and reduce the computational load by eliminating the need to tune unnecessary parameters. To achieve even better results, the system also combines two weighted images, FLAIR and DWI, as an early fusion process. Experiments have shown that this approach leads to higher performance compared to using the same system without fusion. The proposed system is evaluated using a publicly available dataset, ISLES 2015 SSIS, and compared with other state-of-the-art (SOTA) systems. It achieves a dice coefficient score of 0.68, which is significantly better than that of other SOTA systems. • An automatic ischemic stroke segmentation system using few-shot learning is proposed. • The proposed system selects only slices with lesions and excludes unlesioned slices. • We adopted prototypical networks with metric learning to achieve few-shot segmentation. • Two weighted MRIs, FLAIR, and DWI are fused in the few-shot learning paradigm. [ABSTRACT FROM AUTHOR]

Vegetation detection is a common procedure in remote sensing, but recently it has also been applied in robotics as an aid in navigation of autonomous vehicles. In this paper, we present a method for roadside vegetation detection intended for traffic infrastructure maintenance. While many published methods use Near Infrared images for vegetation detection, our method uses images from the visible spectrum which allows the use of a common color camera on-board a vehicle. Deep neural networks have proven to be a very promising machine learning technique and have shown excellent results in different computer vision problems. In this paper, we show that Fully Convolutional Neural Networks can be effectively used in a real-world application for detecting roadside vegetation. For training and testing purposes, we have created our own image database which contains roadside vegetation in various conditions. We present promising experimental results and a discussion of encountered problems in a real-world application as well as a comparison with several alternative approaches. [ABSTRACT FROM AUTHOR]

Due to the large intra-class variations and unbalanced training samples, the accuracy of existing algorithms used in defect classification of hot rolled steels is unsatisfactory. In this paper, a new hierarchical learning framework is proposed based on convolutional neural networks to classify hot rolled defects. Multi-scale receptive field is introduced in the new framework to extract multi-scale features, which can better represent defects than the feature maps produced by a single convolutional layer. A group of AutoEncoders are trained to reduce the dimension of the extracted multi-scale features which improve the generalization ability under insufficient training samples. Besides, to mitigate the deviation caused by fine-tuning the pre-trained model with images of different context, we add a penalty term in the loss function, which is to reconstruct the input image from the feature maps produced by the pre-trained model, to help network encode more effective and structured information. The experiments with samples captured from two hot rolled production lines showed that the proposed framework achieved a classification rate of 97.2% and 97% respectively, which are much higher than the conventional methods. • We proposed a new framework based on Inception-V4 integrated with multi-scale respective field. • AutoEncoders are trained to reduce the dimension of the multi-scale features extracted by the pre-trained model. • A penalty term in the loss function to reconstruct the input image from the feature maps was added. • The methods were tested with two different samples from online surface inspection system of steels. [ABSTRACT FROM AUTHOR]

Lipreading or Visual speech recognition is the process of decoding speech from speaker's mouth movements. It is used for people with hearing impairment, to understand patients attained with laryngeal cancer, people with vocal cord paralysis and in noisy environment. In this paper we aim to develop a visual-only speech recognition system based only on video. Our main targeted application is in the medical field for the assistance to laryngectomized persons. To that end, we propose Hahn Convolutional Neural Network (HCNN), a novel architecture based on Hahn moments as first layer in the Convolutional Neural Network (CNN) architecture. We show that HCNN helps in reducing the dimensionality of video images, in gaining training time. HCNN model is trained to classify letters, digits or words given as video images. We evaluated the proposed method on three datasets, AVLetters, OuluVS2 and BBC LRW, and we show that it achieves significant results in comparison with other works in the literature. • This work proposes a new architecture called Hahn Convolutional Neural Network. • The complexity is reduced enormously by minimizing number of parameters and layers. • The experiments are conducted on AVLetters, OuluVS2 and BBC LRW datasets. • The classification results are 59,23% on AVLetters, 93,72% on OuluVS2, 46,6% on BBC LRW. [ABSTRACT FROM AUTHOR]

[Display omitted] • Introduce capsule network for multi-focus image fusion. • Utilize structure information to help locate focus regions. • Design structure-guided flow module to integrate structure features. Existing deep learning based methods have shown their advantages in multi-focus image fusion task. However, most methods still suffer from inaccurate focus region detection. In this paper, we employ the property of part-whole relationships embedded by the Capsule Network (CapsNet) to solve the problem. Specifically, we introduce CapsNet in multi-focus image fusion task, and design a structure-guided flow module, which fully utilizes structure information to help locate focus regions. CapsNet is introduced to extract structure features by supervising gradient information of the image. Compared with traditional convolutional neural networks (CNNs), CapsNet takes into account the correlation of features from different positions, such that it encodes more compact features. Once structure features are obtained, a flow alignment module is introduced to learn flow field between structure and image features, and propagate effectively structure features to image features to make confident focus region detection. Experimental results show the proposed method achieves robust fusion performance on three publicly available multi-focus datasets, and outperforms or is comparable to the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Image segmentation of polyps can provide an important basis for the diagnosis of colorectal cancer and has a high clinical application value. Segmentation of polyp regions is very challenging due to the high similarity between polyps and background mucosal tissue, many existing methods have failed to produce satisfactory polyp segmentation results. Therefore, scholars have recently used transformer backbone networks to extract features, which captures global information better than CNNs, resulting in more accurate detection results, but their boundary results are still not accurate enough due to the lack of processing for the boundary. In this paper, we propose a multi-scale perceptual polyp segmentation network based on boundary guidance, to obtain higher segmentation accuracy in both regions and boundary. To increase the feature response region, we first propose a multi-scale global perception module to expand the receptive field and aggregate multi-scale contextual information to capture the primary location of polyps at local and global levels. Then, we design a boundary-guided feature enhancement module that utilizes contextual features to mine hidden polyp boundary and employs the boundary to guide region learning to improve segmentation boundary accuracy. Finally, we propose a complementary fusion module that uses higher-level features to filter out the background noise of lower-level features and fuses the features layer by layer. In particular, to refine the extracted features, a detail refinement module is designed to complement the spatial details to improve the segmentation performance. Extensive experiments using seven evaluation metrics on five publicly available polyp datasets have shown that the proposed a multi-scale perceptual polyp segmentation network based on boundary guidance outperforms most state-of-the-art models. • Propose multi-scale global perception module to expand the receptive field. • Propose boundary-guided feature enhancement module to solve boundary ambiguity. • Propose complementary fusion modules to correct false detection region. • Propose detail refinement module to supplement spatial detail information. [ABSTRACT FROM AUTHOR]

Due to the influence of light absorption and scattering in the water medium, underwater images often appear with different degrees of blur, color distortion, and so on. The physical model-based method needs to establish image degradation processes and improve image quality by assuming and estimating complex parameters, which limits the generalization ability and practical application effect of these methods. To solve the above problems, this paper proposes an image enhancement method of non-uniform scattering in turbid underwater based on neural network. First, the high-frequency and low-frequency components of underwater images were obtained through the difference of gaussian (DOG) and bilateral filtering (BF), respectively. Then, for the high-frequency component, pulse coupled neural networks (PCNN) to improve the edge details of the image. For low-frequency components, it is carried out in two steps: an adaptive automatic white balance and polynomial regression (AAWBPR) algorithm is proposed, which can effectively eliminate the color bias under different color temperatures and improve the brightness of images; the adaptive color and contrast enhancement (ACCE) method is used to further enhance the color and contrast of images. Finally, Decom-Net and the improved U-Net method are used to reduce the errors caused by the non-uniform scattering effect, weaken the noise interference, and enhance the overall effect of images. Extensive experiments show that the proposed method can effectively improve the quality of turbid underwater non-uniform scattering images, enhance the contrast of images, reduce the complexity of parameters, and has good generalization performance and practical application effects. • A suppression algorithm of heterogeneous scattering in turbid water is proposed. • An adaptive automatic white balance and polynomial regression algorithm is proposed. • It reduces the image deviation, and improve the problem of complex model parameters. • It is applied to image segmentation, saliency detection, and keypoint detection. • This method can get better training results in small dataset. [ABSTRACT FROM AUTHOR]

• A novel model for improving identification of Diabetic Foot Ulcers. • A bottleneck attention module focused on channel-wise and spatial important features. • Experimental results show that the proposed model outperforms state-of-the-art models. Diabetic Foot Ulcer (DFU) is a severe complication of diabetes, and it may cause lower limb amputation. However, the manual diagnosis of DFU is a complicated and expensive process. The primary objective of this work is to design an efficient Convolutional Neural Network (CNN) approach to identify DFU. Therefore, a novel CNN-based approach (AESPNet) is proposed in this paper, where convolution layers are stacked together in a parallel fashion and with an intermediate attention module to perform DFU vs -normal skin classification. The AESPNet consists of 2 blocks, where varying-sized kernel convolution layers are connected in a parallel fashion for better local and global feature abstraction. A bottleneck attention module is associated after every concatenation operation in the network. The Stochastic Gradient Descent (SGD) (with momentum) optimizer with 1 e - 2 learning rate is used to train the proposed network on a privately accessed DFU dataset. The results of the proposed approach are compared with other standard CNN-based schemes, namely AlexNet, VGG16, DenseNet121, and InceptionV3. It has been found that the proposed AESPNet outperforms state-of-the-art schemes with a sensitivity score of 98.44% and 0.98 F1-Scores. [ABSTRACT FROM AUTHOR]

• SIID-CNN detects signal modes' interference region in spectrogram images using CNN. • SIID-CNN treats interference detection as 3-classes image classification problem. • The training set relies on a local linear model for modes instantaneous frequency. • SIID-CNN classification is accurate and robust to the presence of moderate noise. Ridge curves retrieval in time–frequency (TF) domains is fundamental in many application fields as they convey most of information concerning the instantaneous frequencies of non-stationary signals. However, it represents a very hard task in the case of multicomponent signals having non-separable modes as they generate interference in TF domains. A preliminary detection of these interference regions may be then useful for the definition of effective strategies for ridge curve recovery. This paper introduces SIID-CNN (Spectrogram Image Interference Detection via CNN) , that is a novel approach based on machine learning for the automatic detection of interference regions in spectrogram images. Each spectrogram sample is suitably classified as interference, single mode or background by accounting for its relative information. Some critical problems, such as the training set size and the type of examples to use for populating the training set, are dealt with. Experimental results show that a local linear model for spectrogram image and a small training set can guarantee good classification rates for a wide class of non-stationary signals, even in the presence of moderate noise. [ABSTRACT FROM AUTHOR]

Automating polyps' detection and segmentation is very helpful in achieving early detection and potentially effective treatment of colon cancer. Because of the constant rise in processing power at a reasonable cost and the availability of training data, deep learning more than ever is becoming very powerful and increasingly widespread in many domains, including computer vision and medical image processing and segmentation. In this paper, we propose a novel method for polyp image segmentation using convolutional neural networks (CNNs) and utilizing multi-parallel U-Net encoder architecture. The proposed method uses multiple encoders that utilize pre-trained CNNs to enrich the extracted features. Skip connections from these parallel encoders are concatenated and propagated to the decoder in partially and fully meshed fashions allowing for data from shallow to deeper layers in the network to be exchanged and vice versa. We trained and tested the method on five publicly available datasets: Kvasir, CVC-Clinic DB, CVC-Colon DB, CVC-T, and ETIS-Larib, which are well-known benchmark datasets for polyp image segmentation. We tested the performance of the method on these five datasets using multiple testing scenarios including the effect of using attention, vision transformers, and multiple decoders. Experimental results showed that the proposed method outperforms other state-of-the-art methods on two of the used benchmark datasets and ranked second on a third. • A novel CNN-based method using parallel encoders for polyp segmentation is proposed. • Combinations of two and three parallel encoders, and decoders with different pre-trained networks are investigated. • The effect of attention and vision transformers is examined. • Five publicly available polyp image databases were used in the experiments. • Dual encoder with attention block and partial mesh skip connections achieved the best result. [ABSTRACT FROM AUTHOR]

• New architecture design for image segmentation tasks. • Incorporating the advantages of CNN and MLP. • Achieving better performance compared with state-of-the-art models and baseline models. Convolutional neural networks (CNN) are widely used in computer-aided diagnosis (CAD). However, there are several limitations of this structure for mass segmentation in mammograms, which may lead to incorrect segmentation results. Recently, multi-layer perceptron (MLP) based methods introduce a new way to solve computer vision (CV) problems. In this paper we propose U-shaped Sparse-MLP (USMLP), which is a MLP-based segmentation model with U-shaped architecture. Our proposed method consists of CNN layers and sparse MLP (sMLP) blocks. Detailed experiments on two public datasets show that our method achieves state-of-the-art performance while remaining efficiency, compared with other benchmarks. We hope the investigation of new network architecture can be beneficial to mass segmentation tasks as well as CAD systems. [ABSTRACT FROM AUTHOR]

Unsupervised domain adaptation addresses the problem of knowledge transformation from source domain to target domain, aiming to effectively alleviate data distribution mismatch and data labeling consumption. The issue of data distribution mismatches is widespread in actual agricultural visual tasks. Moreover, it is expensive and time-consuming to construct and label visual image data. For in-field cotton boll, its maturing status can greatly affect the yield and quality. Uneven distribution restrains the performance for maturing status recognition. Therefore, domain adaptation is essential for identifying cross-domain cotton boll maturity. Existing unsupervised domain adaptation methods obtain domain invariant feature for achieving domain alignment. However, the discriminability of features is less considered, which may result in unsatisfactory classification results. In this paper, an unsupervised domain adaptation method called discriminable feature enhancement(DFE-DA) is proposed to identify cross-domain cotton boll maturity. It enables to minimize intra-class distance by maximizing intra-domain density(MID) loss and realizes discriminable feature enhancement. The effectiveness of DFE-DA is verified on in-field cotton boll V2(IFCB-V2) dataset containing 2400 images. The experimental results demonstrate that DFE-DA has an average improvement of 12.8%, 10.3% and 7.6% compared with other methods in three different transfer tasks. Furthermore, the MID loss can cooperate well with other adversarial methods. To verity the robustness of DFE-DA, additional experiments conducted on the public benchmark Office-31 and Office-Home indicates it is comparable to the state-of-the-arts. [Display omitted] • A novel domain adaptation framework for cross-domain cotton boll maturity recognition. • Pulling same category samples closer to achieve discriminable feature enhancement. • Maximizing intra-domain density loss is designed to be general. • Experiments validate the effectiveness and robustness of our method. [ABSTRACT FROM AUTHOR]

• Image processing tehnique for water surface velocity prediction. • Temporal feature extraction for dynamic image sequence using XOR operator. • Machine learning method for image texture analysis. Water surface velocity (WSV) is one of the critical parameters in hydrology. The development of non-intrusive measurement of this parameter using cameras is increasing. Traditionally, measuring WSV using a camera utilizes tracking of moving objects on the water surfaces. Recently, another method has emerged that utilizes the movement of water ripples to estimate the WSV. This paper proposes a novel method for estimating WSV based on camera measurements. The authors call this method Temporal Gray-scale Pattern Image (TGPI) since it extracts Gray-scale patterns of pixels of water flow video in the temporal domain using the XOR operator and create new image. The new image pattern formed is then used to predict WSV using predictor. There are four predictors being compared, namely Multiple Input Linear Regression (MILR), Multiple Input Logistic Regression (MILgR), Convolutional Neural Network Regression (CNN-R), and Convolutional Neural Network Classification (CNN-K). CNN-R and CNN-K predict WSV directly from the results of TGPI. Meanwhile, MILR and MILgR predict WSV from five TGPI features. The five features are the Mean and Median of the Histogram, Mean and Median of the Histogram of Oriented Gradient (HOG), and the Maximum Mid-Value of the Fast Fourier Transform (FFT). MILR and CNN-R are predictors for regression problems, so the testing metrics for them are Trend-Line equation and R2 from the predicted WSV and actual values graph. Meanwhile, for MILgR and CNN-K, which are predictors for classification problems, the testing metrics are Confusion Matrix (CM), Accuracy, Precision, Recall, and F1-Score. To test the four methods without distinguishing their predictor types, a 2-dimensional histogram graph is used. The data-set used for training and testing is video footage of water flow with known WSV. The WSV measurement points used in this study are 1.7 m/s, 3.1 m/s, and 4.2 m/s. The video dataset and these three points are generated by the Mini Open Channel Water Flow Simulator (MOCWFS) developed by the author in this study. From the comparison results, it can be seen that the classification type predictor is superior to the regression type. For the regression type predictor, MILR is better than CNN-R. Meanwhile, for the classification type, CNN-K is superior to MILgR. The best accuracy produced by CNN-K is 98.4%. Although there are shortcomings, the TGPI method is quite feasible for predict Water Surface Velocity. [ABSTRACT FROM AUTHOR]

• Our proposed method combine pseudo-color enhancement to improve the performance. • We use the hybrid attention module to enhance the information interaction. • We propose a multi-scale channel attention module (MSCA) to achieve more information. Ultrasound image segmentation plays a vital role in the early diagnosis of human diseases. It helps diagnose many diseases, such as breast cancer, hemangioma, and other gynecological disorders. However, the intrinsic imaging characteristics of ultrasound images result in substantially lower resolution and clarity than CT, MRI, and other imaging modalities, and they are sensitive to interference from external influences. With its inherent artifacts, blurred lesion boundaries, and uneven intensity distribution, ultrasound images present a challenging task when it comes to segmenting lesion areas accurately. In recent years, convolutional neural networks (CNNs) have achieved remarkable results in medical image segmentation tasks. However, CNNs are limited in capturing the remote dependencies of the input image, leading to degraded accuracy in segmenting ultrasound lesions. In this paper, we developed a deep convolutional neural network that incorporates the pseudo-color enhancement algorithm and hybrid attention modules that enhance the network's ability to extract fine features and remote modeling capabilities. We propose a novel multi-scale channel attention-based decoder that efficiently uses the feature maps from the encoder as a complement and fuses them with the upsampled feature maps. The hybrid attention combination captures cross-channel interactions efficiently and enhances the context modeling capability, further improving the extraction of coarse and delicate features, and resulting in significant performance improvements. We found that the dice performance improved by 2.54%, 2.47%, 1.39%, 0.99%, and 1.23% on the BUL, BUSI, Hemangioma, BP, and VUI. Results from four public datasets and one self-collected dataset indicate that the proposed method outperforms other medical image segmentation methods for ultrasound image lesion segmentation. [ABSTRACT FROM AUTHOR]

Network pruning and knowledge distillation, as two effective network compression techniques, have drawn extensive attention due to their success in reducing model complexity. However, previous works regard them as two independent methods and combine them in an isolated manner rather than joint, leading to a sub-optimal optimization. In this paper, we propose a collaborative compression scheme named P runing-and- D istillation via C lustering (PDC), which integrates pruning and distillation into an end-to-end single-stage framework that takes both advantages of them. Specifically, instead of directly deleting or zeroing out unimportant filters within each layer, we reconstruct them based on clustering, which preserves the learned features as much as possible. The guidance from the teacher is integrated into the pruning process to further improve the generalization of pruned model, which alleviates the randomness caused by reconstruction to some extent. After convergence, we can equivalently remove reconstructed filters within each cluster through the proposed channel addition operation. Benefiting from such equivalence, we no longer require the time-consuming fine-tuning step to regain accuracy. Extensive experiments on CIFAR-10/100 and ImageNet datasets show that our method achieves the best trade-off between performance and complexity compared with other state-of-the-art algorithms. For example, for ResNet-110, we achieve a 61.5 % FLOPs reduction with even 0.14 % top-1 accuracy increase on CIFAR-10 and remove 55.2 % FLOPs with only 0.32 % accuracy drop on CIFAR-100. • Integrate pruning and distillation into a one-stage framework for collaborative compression. • Reconstruct redundant filters instead of removing to preserve learned knowledge. • Equivalently remove reconstructed filters by the channel addition operation. • Distill with the original model to avoid mismatch problem in vanilla distillation. [ABSTRACT FROM AUTHOR]

• A novel end-to-end network is proposed for fine-grained visual categorization. • The self-calibrated convolution is introduced to extract the salient features. • The gradual attention is proposed to find more local detailed features. • The inverse module is designed to boost the performance. Fine-grained visual categorization, which aims to identify the different subcategories of images within the same category, is a very challenging task due to the large intra-class differences and subtle inter-class variances. The existing methods mostly focus on the salient local regions and ignore other features which probably help to recognize the images more precisely. To address this issue, in this paper, we propose a novel end-to-end network composed of the self-calibrated convolution, gradual attention module and feature inverse module for fine-grained visual categorization. To extract the salient features, the self-calibrated convolution is exploited which can avoid the influence of irrelevant information and locate salient regions more accurately. In aiming to extract the discriminative features, we propose the gradual attention module which consists of alternate channel-spatial attention and hierarchical feature grouping. The gradual attention module can extract the subtle discriminative features gradually even when the semantic information of shallow stages is not rich. Moreover, we design the feature inverse module which forces the next stage of network to search for other different useful features by feature inverse. The gradual attention module combined with the feature inverse module is capable of finding more detailed regions and of benefit to improving classification performance. Finally, the stage features and fused features are jointly used for classification. The proposed method is evaluated on three classical fine-grained image datasets and compared with a number of state-of-the-art methods. Our method achieves 89.5 % , 95.2 % and 93.9 % accuracies on CUB-200–2011, Stanford Cars and FGVC-Aircraft datasets respectively. The experimental results demonstrate the effectiveness and superiority of the proposed method. [ABSTRACT FROM AUTHOR]

Text image captioning aims to understand the scene text in images for generating image captions. The key challenge of this task is to accurately and comprehensively understand the OCR tokens of scene text. Due to the dual modal of visual and textual features of scene text, expressing the multimodal semantic features of OCR tokens accurately is a challenging task. Additionally, since scene text cannot exist independently of specific objects and is always associated with its surroundings, establishing a scene graph centered around OCR tokens is also an important approach to understand its relationship with other objects in the image. In this paper, we propose a novel model named C lip- O CR and M aster Obj E ct (dubbed as COME) for text image captioning. First, we introduce a CLIP-OCR module to enhance the multimodal representation of OCR tokens. We separate the OCR representation into visual and textual items and narrow the similarity by contrastive learning. With the assistance of the CLIP-OCR module, we realize correlation alignment between different modes. Next, we propose the concept of master object for each OCR text and purify the OCR-oriented scene graph with it. The master object is defined as the object to which the OCR is attached, which bridges the semantic relationship between the OCR tokens and the image. We consider the master object as a proxy that connects OCR tokens and other regions in the image. By exploring the master object for each OCR token, we build a purified scene graph based on the master object and then enrich the visual embedding by the Graph Convolution Network (GCN). Furthermore, we cluster the OCR tokens and append the hierarchical information on the input embedding to provide a complete representation. Experiments on the TextCaps validation set and test set demonstrate the effectiveness of the proposed framework. • A new framework COME for text image captioning that understands OCR tokens of scene text. • A CLIP-OCR module to enhance the multimodal representation of OCR tokens. • The concept of master object to purified the relationships of OCR token. [ABSTRACT FROM AUTHOR]

The visual Transformer model based on self-attention has achieved better performance than convolutional neural networks in object detection tasks. However, existing visual Transformer models are typically heavy-weight to extract global features. In contrast, CNNs can extract features with fewer parameters and computational costs. To combine the advantages of convolutional processing at the local level with the advantages of the Transformer's global interaction, this paper proposes MCANet, a Hierarchical Cross-Fusion Lightweight Transformer Based on Multi-ConvHead Attention for Object Detection. To bi-directionally fuse local and global features, MCANet adds two improved transformers (MCA-Former) for global interaction and two novel feature fusion modules MCA-CSP. MCA-Former uses a novel self-attention computation method named Multi-ConvHead Attention(MCA) based on multi-scale depth-separable convolution, which reduces the computational cost by 2/3. Meanwhile, the number of model parameters is reduced to 9.49 M by using channel segmentation and multi-layer cross- fusion strategies. On the Pascal VOC and COCO datasets, the proposed model outperforms YOLOv4-Tiny in terms of AP by 2.43% and 1.8%, respectively. Additionally, MCANet is also superior to many latest lightweight object detection models. Results of various ablation experiments also verify the effectiveness of the proposed method. • A novel global attention computation method using multi-scale convolution. • A convolutional fusion Transformer module with a combination of linear and parallel. • A multi-level serial cross-fusion strategy is introduced into the backbone. • Experiments fully validate the effectiveness of the proposed modules. [ABSTRACT FROM AUTHOR]

Despite the tremendous achievements of deep convolutional neural networks (CNNs) in many computer vision tasks, understanding how they actually work remains a significant challenge. In this paper, we propose a novel two-step understanding method, namely Salient Relevance (SR) map , which aims to shed light on how deep CNNs recognize images and learn features from areas, referred to as attention areas , therein. Our proposed method starts out with a layer-wise relevance propagation (LRP) step which estimates a pixel-wise relevance map over the input image. Following, we construct a context-aware saliency map, SR map, from the LRP-generated map which predicts areas close to the foci of attention instead of isolated pixels that LRP reveals. In human visual system, information of regions is more important than of pixels in recognition. Consequently, our proposed approach closely simulates human recognition. Experimental results using the ILSVRC2012 validation dataset in conjunction with two well-established deep CNN models, AlexNet and VGG-16, clearly demonstrate that our proposed approach concisely identifies not only key pixels but also attention areas that contribute to the underlying neural network's comprehension of the given images. As such, our proposed SR map constitutes a convenient visual interface which unveils the visual attention of the network and reveals which type of objects the model has learned to recognize after training. The source code is available at https://github.com/Hey1Li/Salient-Relevance-Propagation. • A two-step visualization scheme for deep CNN understanding and interpretation is proposed. • The method reveals deep CNN model's perception of input images. • Experiment results prove the effectiveness of our proposed salient relevance map. [ABSTRACT FROM AUTHOR]

Abstract Describable visual facial attributes are now commonplace in human biometrics and affective computing, with existing algorithms even reaching a sufficient point of maturity for placement into commercial products. These algorithms model objective facets of facial appearance, such as hair and eye color, expression, and aspects of the geometry of the face. A natural extension, which has not been studied to any great extent thus far, is the ability to model subjective attributes that are assigned to a face based purely on visual judgments. For instance, with just a glance, our first impression of a face may lead us to believe that a person is smart, worthy of our trust, and perhaps even our admiration — regardless of the underlying truth behind such attributes. Psychologists believe that these judgments are based on a variety of factors such as emotional states, personality traits, and other physiognomic cues. But work in this direction leads to an interesting question: how do we create models for problems where there is only measurable behavior? In this paper, we introduce a convolutional neural network-based regression framework that allows us to train predictive models of crowd behavior for social attribute assignment. Over images from the AFLW face database, these models demonstrate strong correlations with human crowd ratings. Graphical abstract Computational modeling of social attributes allows us to predict what the crowd might say about a face image. In this image we graphically compare the attribute predictions for Julian Assange and Benedict Cumberbatch, who plays Assange in the movie The Fifth Estate , as well as the predictions for Edward Snowden and Joseph Gordon-Levitt, who plays Snowden in the movie Snowden. Specifically looking at these images, our models output similar predictions between the subjects and their actors. The radar plots above reflect the output of a face processing pipeline, where faces are detected, aligned, and then processed through a deep convolutional neural network regressor that models a particular social attribute. This regression framework is the main contribution of our work. For this image we display the predictions' z-scores with respect to the training data. Highlights • A novel dataset of over 6000 images annotated for all four traits of interest • The deployment of a crowd-sourced data collection regime for high level attributes • The comparison of different deep learning architectures for abstract modeling • A set of predictors that have high agreement with human annotators [ABSTRACT FROM AUTHOR]

Abstract Managing the water quality of freshwaters is a crucial task worldwide. One of the most used methods to biomonitor water quality is to sample benthic macroinvertebrate communities, in particular to examine the presence and proportion of certain species. This paper presents a benchmark database for automatic visual classification methods to evaluate their ability for distinguishing visually similar categories of aquatic macroinvertebrate taxa. We make publicly available a new database, containing 64 types of freshwater macroinvertebrates, ranging in number of images per category from 7 to 577. The database is divided into three datasets, varying in number of categories (64, 29, and 9 categories). Furthermore, in order to accomplish a baseline evaluation performance, we present the classification results of Convolutional Neural Networks (CNNs) that are widely used for deep learning tasks in large databases. Besides CNNs, we experimented with several other well-known classification methods using deep features extracted from the data. Highlights • We publish a database with 64 types of freshwater macroinvertebrates and more than 15,000 images. • CNNs outperforms a non-expert human in classification accuracy. • Several other well-known classifiers are applied using the features extracted from the CNNs. • In most cases, the classifiers cannot further improve the CNN results. [ABSTRACT FROM AUTHOR]

• A novel framework CCNet is devised for accurate medical image segmentation. • Complementary features are extracted by utilizing Transformer and CNN. • We create a global context refinement module to address semantic dilution issue. • A mutual attentive module is proposed to obtain fore-/background contrastive cues. • Extensive experiments demonstrate that CCNet achieves excellent performance. Existing convolutional neural networks (CNNs) have achieved remarkable performance in medical image segmentation tasks. However, they still fail to generalize well to unseen datasets due to the limited size and diversity of training data as well as distribution shifts. Meanwhile, CNN-based methods have inherent limitations in capturing global contexts and suffer semantic dilution issues in the decoder stage, which leads to suboptimal predictions especially under low inter-class discrepancy and complex backgrounds. In this paper, we propose a novel framework named CCNet that learns complementary and contrastive features for accurate segmentation. Firstly, a novel complementary feature extraction module is formulated to learn global–local features by coordinating Transformer and CNN-style parallel branches. Secondly, a global context refinement module is constructed to adaptively generate a set of layer-specific global maps, so as to remedy semantic dilution. Thirdly, a mutual attentive module is designed to alleviate background confusion, in which contrastive cues are mutually captured from the foreground and background view by cascaded dual attention blocks. Moreover, we implement synthetic data augmentation to deal with training data scarcity and distribution shifts, thereby improving the out-of-distribution generalization of our model. Extensive experiments demonstrate that our CCNet achieves outstanding performance in polyp, skin lesion, and nuclei segmentation tasks, outperforming the state-of-the-arts. [ABSTRACT FROM AUTHOR]

In the last few years, the multi-modal emotion recognition has become an important research issue in the affective computing community due to its wide range of applications that include mental disease diagnosis, human behavior understanding, human machine/robot interaction or autonomous driving systems. In this paper, we introduce a novel end-to-end multimodal emotion recognition methodology, based on audio and visual fusion designed to leverage the mutually complementary nature of features while maintaining the modality-specific information. The proposed method integrates spatial, channel and temporal attention mechanisms into a visual 3D convolutional neural network (3D-CNN) and temporal attention into an audio 2D convolutional neural network (2D-CNN) to capture the intra-modal features characteristics. Further, the inter-modal information is captured with the help of an audio-video (A-V) cross-attention fusion technique that effectively identifies salient relationships across the two modalities. Finally, by considering the semantic relations between the emotion categories, we design a novel classification loss based on an emotional metric constraint that guides the attention generation mechanisms. We demonstrate that by exploiting the relations between the emotion categories our method yields more discriminative embeddings, with more compact intra-class representations and increased inter-class separability. The experimental evaluation carried out on the RAVDESS (The Ryerson Audio-Visual Database of Emotional Speech and Song), and CREMA-D (Crowd-sourced Emotional Multimodal Actors Dataset) datasets validates the proposed methodology, which leads to average accuracy scores of 89.25% and 84.57%, respectively. In addition, when compared to state-of-the-art techniques, the proposed solution shows superior performances, with gains in accuracy ranging in the [1.72%, 11.25%] interval. • A multimodal emotion recognition framework with various self-attention mechanisms. • An audio-video fusion strategy which uses cross-attention. • A learnable emotional metric that extends the traditional triplet loss function. • An extensive objective evaluation is performed on RAVDESS and CREMA-D datasets. [ABSTRACT FROM AUTHOR]

Multimodal recognition of affective states is a difficult problem, unless the recording conditions are carefully controlled. For recognition “in the wild”, large variances in face pose and illumination, cluttered backgrounds, occlusions, audio and video noise, as well as issues with subtle cues of expression are some of the issues to target. In this paper, we describe a multimodal approach for video-based emotion recognition in the wild. We propose using summarizing functionals of complementary visual descriptors for video modeling. These features include deep convolutional neural network (CNN) based features obtained via transfer learning, for which we illustrate the importance of flexible registration and fine-tuning. Our approach combines audio and visual features with least squares regression based classifiers and weighted score level fusion. We report state-of-the-art results on the EmotiW Challenge for “in the wild” facial expression recognition. Our approach scales to other problems, and ranked top in the ChaLearn-LAP First Impressions Challenge 2016 from video clips collected in the wild. [ABSTRACT FROM AUTHOR]

Knowledge distillation has made remarkable achievements in model compression. However, most existing methods require the original training data, which is usually unavailable due to privacy and security issues. This paper proposes a conditional generative data-free knowledge distillation (CGDD) framework for training lightweight networks without real data. This framework realizes efficient knowledge distillation based on conditional image generation. Specifically, we treat the preset labels as ground truth to train a semi-supervised conditional generator. The trained generator can produce specified classes of training images. During training, we force the student model to extract the hidden knowledge in teacher feature maps, which provide crucial cues to the learning process. Meanwhile, we construct an adversarial training framework to promote distillation performance. The framework will help the student model to explore larger data space. To demonstrate the effectiveness of the proposed method, we conduct extensive experiments on different datasets. Compared with other data-free works, our method obtains state-of-the-art results on CIFAR100, Caltech101, and different versions of ImageNet datasets. The codes will be released. • A conditional generative data-free distillation framework is proposed. • The auxiliary class constraint is introduced to realize semi-supervised learning. • Trained student network can well imitate teacher attention. • We obtain state-of-the-art results on most experimental datasets. [ABSTRACT FROM AUTHOR]

YOLOv4-tiny is one of the most representative lightweight one-stage object detection algorithms. In this paper, we propose Mini-YOLOv4-tiny, an improved lightweight one-stage object detector based on the YOLOv4-tiny. Typical compression techniques address both memory optimization and computational complexity, but compromise model accuracy. For model compression and speedup, we selectively cut the width of the last convolutional layers. To increase model performance, we propose several improvements with minimal impact on memory and inference time. First, we replace the Leaky-RELU activation function with Mish and fine-tune the data augmentation parameters. To enlarge the receptive field of the network, we propose a modified Spatial Pyramid Pooling module with a reduced number of convolutional blocks and filters, thus adding fewer Floating Point Operations (FLOPs) during the inference process. We performed experiments on the PASCAL VOC and MS COCO datasets and evaluated the inference speed on NVidia Jetson Nano and Raspberry PI 4 (model B). The experimental results show that our methods achieve 72.1% mAP@0.5 and 23.4% mAP@ [0.5:0.95] on the PASCAL VOC and MS COCO datasets, achieving state-of-the-art results among lightweight object detectors. Compared with YOLOv4-tiny on MS COCO, our method has 37% fewer parameters and requires 19% fewer FLOPs, while improving the Intersection over Union (IoU) by 4.02% and the average precision (interval 0.50:0.95) by 2.8%. On PASCAL VOC, Mini-YOLOv4-tiny with an input size of 288 improves mAP@0.5 by 0.3%, while requiring 61% fewer FLOPS and running almost twice as fast. [Display omitted] • Mini-YOLOv4-tiny, an improved lightweight object detector based on YOLOv4-tiny. • We cut the width of the last convolutional layers for model compression and speedup. • Increase performance by switching to Mish activation function and parameters tuning. • New Spatial Pyramid Pooling module with a small number of Floating Point Operations. • MS-COCO: with fewer parameters, our method improves the average precision by 2.8%. [ABSTRACT FROM AUTHOR]

Cell tracking plays crucial role in biomedical and computer vision areas. As cells generally have frequent deformation activities and small sizes in microscope image, tracking the non-rigid and non-significant cells is quite difficult in practice. Traditional visual tracking methods have good performances on tracking rigid and significant visual objects, however, they are not suitable for cell tracking problem. In this paper, a novel cell tracking method is proposed by using Convolutional Neural Networks (CNNs) as well as multi-task learning (MTL) techniques. The CNNs learn robust cell features and MTL improves the generalization performance of the tracking. The proposed cell tracking method consists of a particle filter motion model, a multi-task learning observation model, and an optimized model update strategy. In the training procedure, the cell tracking is divided into an online tracking task and an accompanying classification task using the MTL technique. The observation model is trained by building a CNN to learn robust cell features. The tracking procedure is started by assigning the cell position in the first frame of a microscope image sequence. Then, the particle filter model is applied to produce a set of candidate bounding boxes in the subsequent frames. The trained observation model provides the confidence probabilities corresponding to all of the candidates and selects the candidate with the highest probability as the final prediction. Finally, an optimized model update strategy is proposed to enable the multi-task observation model for the variation of the tracked cell over the entire tracking procedure. The performance and robustness of the proposed method are analyzed by comparing with other commonly-used methods. Experimental results demonstrate that the proposed method has good performance to the cell tracking problem. [ABSTRACT FROM AUTHOR]

Crowd counting is an effective tool for situational awareness in public places. Automated crowd counting using images and videos is an interesting yet challenging problem that has gained significant attention in computer vision. Over the past few years, various deep learning methods have been developed to achieve state-of-the-art performance. The methods evolved over time vary in many aspects such as model architecture, input pipeline, learning paradigm, computational complexity, and accuracy gains etc. In this paper, we present a systematic and comprehensive review of the most significant contributions in the area of crowd counting. Although few surveys exist on the topic, our survey is most up-to date and different in several aspects. First, it provides a more meaningful categorization of the most significant contributions by model architectures, learning methods (i.e., loss functions), and evaluation methods (i.e., evaluation metrics). We chose prominent and distinct works and excluded similar works. We also sort the well-known crowd counting models by their performance over benchmark datasets. We believe that this survey can be a good resource for novice researchers to understand the progressive developments and contributions over time and the current state-of-the-art. [ABSTRACT FROM AUTHOR]

This paper addresses fully supervised action segmentation. Transformers have been shown to have large model capacity and powerful sequence modeling abilities, and hence seem quite suitable for capturing action grammar in videos. However, their performance in video understanding still lags behind that of temporal convolutional networks, or ConvNets for short. We hypothesize that this is because: (i) ConvNets tend to generalize better than Transformers, and (ii) Transformer's large model capacity requires significantly larger training datasets than existing action segmentation benchmarks. We specify a new hybrid model, TCTr, that combines the strengths from both frameworks. TCTr seamlessly unifies depth-wise convolution and self-attention in a principled manner. Also, TCTr addresses the Transformer's quadratic computational and memory complexity in the sequence length by learning how to adaptively estimate attention from local temporal neighborhoods, instead of all frames. Our experiments show that TCTr significantly outperforms the state of the art on the Breakfast, GTEA, and 50Salads datasets. [ABSTRACT FROM AUTHOR]

• Propose a collaborative learning framework for head pose estimation. • Learn complementary information from the landmark-based and landmark-free branches. • Design a Landmark-MLP-Mixer to obtain the head pose angles from facial landmarks. • Introduce a union loss function to optimize two complementary branches. • Conduct comparison experiments and ablation studies on various datasets. Head pose estimation is an important task in many real-world applications, such as human–computer interaction, driver monitoring, face localization and gaze estimation. In this paper, we present a novel collaborative learning framework based on Convolutional Neural Networks (CNNs) for head pose estimation. The proposed framework consists of a landmark-based branch and a landmark-free branch. The former first estimates facial landmarks and then follows the Landmark-MLP-Mixer module which models the complex nonlinear mapping relationship from facial landmarks to head pose angles. While the later adopts a label distribution learning strategy to estimate head pose. The two branches both dedicate themselves to head pose estimation task, and they collaborate with each other for mutual promotion and complementary semantic learning. Specifically, we introduce a dual-branch transfer module in the middle of the network to achieve explicit semantic interaction and introduce a multi-loss strategy that induces to implicit information interaction. We conduct extensive experiments on several popular benchmarks, including AFLW, AFLW2000 and BIWI, the results show that our method is competitive compared to other state-of-the-art methods. [ABSTRACT FROM AUTHOR]

• A novel joint-nullspace is carfully designed instead of feature space. • An adaptive loss function is developed to train the bounding box estimation. • Extensive experiments have been conducted to evaluate NullSpaceRDAR on six benchmarks. Recently, discriminative-based and Siamese-based trackers have achieved outstanding performance on most tracking benchmarks. However, these trackers use the pre-trained backbone networks that have been mainly designed for classification to extract target-specific features without taking into consideration the visual object tracking task. In this paper, we propose NullSpaceRDAR, a novel tracker that learns a robust target-specific feature representation specifically designed for object tracking. This feature representation is learned by projecting the traditional backbone feature space onto a novel discriminative nullspace that is used to regularize the backbone loss function. We refer to the discriminative nullspace herein as joint-nullspace. The same target features (i.e., target-specific) in the proposed joint-nullspace are collapsed into a single point, and different target-specific features are collapsed into different points. Consequently, the joint-nullspace forces the network to be sensitive to the object's variations from the same class (i.e., intra-class variations). Moreover, a modified adaptive loss function is developed for bounding box estimation to select the most suitable loss function from a super set family of loss functions based on the training data. This makes NullSpaceRDAR more robust to different challenges such as occlusions and background clutter. Extensive experiments have been conducted on six benchmarks to evaluate NullSpaceRDAR: OTB100, VOT variations (VOT2018, VOT2019, and VOT2020), LaSOT, TrackingNet, UAV123, and GOT10k. The results show that NullSpaceRDAR outperforms the state-of-the-art trackers. [ABSTRACT FROM AUTHOR]

• AGA-GAN uses facial descriptors to generate high-resolution facial images. • Performance increases with additional information about features like facial hair. • Coupling AGA-GAN with UNet gives superior hallucination results. • AGA-GAN and AGA-GAN+UNet, both translate well on downstream face verification tasks. The performance of facial super-resolution methods relies on their ability to recover facial structures and salient features effectively. Even though the convolutional neural network and generative adversarial network-based methods deliver impressive performances on face hallucination tasks, the ability to use attributes associated with the low-resolution images to improve performance is unsatisfactory. In this paper, we propose an Attribute Guided Attention Generative Adversarial Network which employs novel attribute guided attention (AGA) modules to identify and focus the generation process on various facial features in the image. Stacking multiple AGA modules enables the recovery of both high and low-level facial structures. We design the discriminator to learn discriminative features by exploiting the relationship between the high-resolution image and their corresponding facial attribute annotations. We then explore the use of U-Net based architecture to refine existing predictions and synthesize further facial details. Extensive experiments across several metrics show that our AGA-GAN and AGA-GAN + U-Net framework outperforms several other cutting-edge face hallucination state-of-the-art methods. We also demonstrate the viability of our method when every attribute descriptor is not known and thus, establishing its application in real-world scenarios. Our code is available at https://github.com/NoviceMAn-prog/AGA-GAN. [ABSTRACT FROM AUTHOR]

Monocular depth estimation is a classical but challenging task in the field of computer vision. In recent years, Convolutional Neural Network (CNN) based models have been developed to estimate high-quality depth map from a single image. Most recently, some Transformer based models have led to great improvements. All the researchers are looking for a better way to handle the global processing of information which is crucial for depth relation inference but of high computational complexity. In this paper, we take advantage of both the Transformer and CNN then propose a novel network architecture, called Rich Global Feature Guided Network (RGFN), with which rich global features are extracted from both encoder and decoder. The framework of the RGFN is the typical encoder-decoder for dense prediction. A hierarchical transformer is implemented as the encoder to capture multi-scale contextual information and model long-range dependencies. In the decoder, the Large Kernel Convolution Attention (LKCA) is adopted to extract global features from different scales and guide the network to recover fine depth maps from low spatial resolution feature maps progressively. What's more, we apply the depth-specific data augmentation method, Vertical CutDepth, to boost the performance. Experimental results on both the indoor and outdoor datasets demonstrate the superiority of the RGFN compared to other state-of-the-art models. Compared with the most recent method AdaBins, RGFN improves the RMSE score by 4.66% on the KITTI dataset and 4.67% on the NYU Depth v2 dataset. [ABSTRACT FROM AUTHOR]

Detecting small or tiny objects is always a difficult and challenging issue in computer vision. In this paper, we provide a latest and comprehensive survey of deep learning-based detection approaches from the perspective of small or tiny objects. Our survey is featured by thorough and exhaustive analysis of small or tiny object detection. We comprehensively introduce 30 existing datasets about small or tiny objects, and summarize different definitions of small or tiny objects based on different application scenarios, such as pedestrian detection, traffic signs detection, face detection, remote sensing target detection and object detection in common life. Then small or tiny object detection techniques are overviewed systematically from seven aspects, including super-resolution techniques, context-based information, multi-scale representation learning, anchor mechanism, training strategy, data augmentation, and schemes based on loss function. Finally, the detection performance of small or tiny objects on 12 popular datasets is analyzed in depth. Based on performance analysis, we also discuss the promising research directions in the future. We hope this survey could provide researchers guidance to catalyze understanding of small or tiny object detection and further facilitate research on small or tiny object detection systems. • Summarize the 30 datasets about small or tiny objects. • Provide definitions of small/tiny objects based on different application scenarios. • Systematically review small/tiny object detection techniques from seven aspects. • Present the evaluation results of different methods for small/tiny object detection. • Discuss future research directions of small/tiny object detection. [ABSTRACT FROM AUTHOR]

Crime detection and their prediction is a fundamental process to reduce criminal activities before they actually happen. Moreover, the detection method is vital since can it potentially can save the victim's life, avoid all-time strain, and harm to the public/private property. In addition, it can be useful in predicting the possible terrorist activities. Crime detection using deep learning models is an attention-grabbing research area. Detecting and reducing the criminal activities is imperative to develop a peaceful society. Video surveillance automates the hazardous situations and enables a law enforcement system to take effective steps towards public safety. In this paper, an end-to-end deep learning model is proposed which is based on Bi-directional gated recurrent unit (BiGRU) and Convolutional neural network (CNN) to detect and prevent criminal activities. The CNN extracts the spatial features from video frames whereas temporal and local motion features are extracted by the BiGRU from multiple frames CNN extracted features. The focused bag is created to select those video frames which indicate certain actions. Moreover, ranked-based loss is used to effectively detect and classify the suspicious activities. For classification of activities, various machine learning classifiers are used. The proposed deep learning video surveillance technique is able to track human trails and detect criminal events. The CAVIAR dataset is used to examine the proposed technique for video surveillance-based crime detection with a performance accuracy of almost 98.86%. The alerts received from the proposed technique can also be examined, demonstrates that the practiced video surveillance cameras systems can effectively detect unusual and criminal activities. In addition, the proposed technique showed considerable performance accuracy and outscored the related state-of-the-art (SOTA)DL models including CNN-LSTM, CNN, HMM, and DBN and achieved 21.88% absolute improvement in crime detection accuracy. • End-to-end Video Surveillance model for Crime detection and Prevention is proposed. • GRUs and SRUs are incorporated with CNN Architectural Changes in End-to-End fashion. • Bagged and Random Video Frames are selected to reduce the Complexity and Processing. [ABSTRACT FROM AUTHOR]