Search

The increasing availability of remote sensing data allows dealing with spatial-spectral limitations by means of pan-sharpening methods. However, fusing inter-sensor data poses important challenges, in terms of resolution differences, sensor-dependent deformations and ground-truth data availability, that demand more accurate pan-sharpening solutions. In response, this paper proposes a novel deep learning-based pan-sharpening model which is termed as the double-U network for self-supervised pan-sharpening (W-NetPan). In more details, the proposed architecture adopts an innovative W-shape that integrates two U-Net segments which sequentially work for spatially matching and fusing inter-sensor multi-modal data. In this way, a synergic effect is produced where the first segment resolves inter-sensor deviations while stimulating the second one to achieve a more accurate data fusion. Additionally, a joint loss formulation is proposed for effectively training the proposed model without external data supervision. The experimental comparison, conducted over four coupled Sentinel-2 and Sentinel-3 datasets, reveals the advantages of W-NetPan with respect to several of the most important state-of-the-art pan-sharpening methods available in the literature. The codes related to this paper will be available at https://github.com/rufernan/WNetPan. [ABSTRACT FROM AUTHOR]

• Providing an in-depth and comprehensive review related to underwater object detection. • Extensively exploring the correlation between underwater image enhancement and underwater object detection. • Analyzing the impact and augmentation forms of underwater image enhancement in underwater object detection. • Discussing the challenges, future trends and applications of underwater object detection. Underwater object detection is one of the most challenging research topics in computer vision technology. The complex underwater environment makes underwater images suffer from high noise, low visibility, blurred edges, low contrast and color deviation, which brings significant challenges to underwater object detection tasks. In underwater object detection tasks, traditional object detection methods often perform poorly in terms of accuracy and generalization capabilities. Underwater object detection requires accurate, stable, generalizable, real-time and lightweight detection models, for which many researchers have proposed various underwater object detection techniques based on deep learning. Although many outstanding results have been achieved on underwater object detection over the years, the research status of underwater object detection techniques are still lack of unified induction, and some existing problems need to be further probed from the latest perspective. In addition, previous reviews lack analysis on the relationship between underwater image enhancement and object detection. Therefore, this paper provides a comprehensive review of the current research challenges, future development trends, and potential applications of underwater object detection techniques. More importantly, this paper has explored the internal relationship between underwater image enhancement and object detection, and analyzed the possible implementation manners of underwater image enhancement in the object detection task in order to further enhance its benefits. The experiments show the performances of current underwater image enhancement and state-of-the-art object detection algorithms, point out their limitations, and indicate that there is not a strict positive correlation between underwater image enhancement and the accuracy improvement of object detection. The domain shift caused by underwater image enhancement cannot be ignored. This paper can be regarded as a guide for future works on underwater object detection. [ABSTRACT FROM AUTHOR]

Generative Adversarial Networks (GANs) are well-known tools for data generation and semi-supervised classification. GANs, with less labeled data, outperform Deep Neural Networks (DNNs) and Convolutional Neural Networks (CNNs) in classification tasks. The success of GANs in classification tasks motivates the development of GAN-based techniques for semi-supervised regression tasks. However, developing GANs for regression introduces two major challenges: (1) inherent instability in the GAN formulation and (2) performing regression and achieving stability simultaneously. This paper introduces techniques that show improvement in the GANs' regression capability. We bake a differentiable fuzzy logic system at multiple locations in a GAN. The fuzzy logic takes the output of either the generator or the discriminator to predict the output, y , and evaluate the generator's performance. We outline the results of applying the fuzzy logic system across multiple GANs and summarize each approach's efficacy. This paper shows that adding a fuzzy logic layer can enhance GAN's ability to perform regression; the most desirable injection location is problem-specific, and we show this through experiments over various datasets. Besides, we demonstrate empirically that the fuzzy-infused GANs are competitive with the DNNs. [ABSTRACT FROM AUTHOR]

• Processing Complex-valued PolSAR Data Using Complex-valued Convolutional Neural Network (CV-CNN). • Uses a Gaussian-type activation function (GTAF) that preserves the integrity of complex-valued operations. • Introduces learnable Gaussian parameters for GTAF, and designs two multi-channel activation methods. • The classification accuracy is better than that of existing state-of-the-art methods in three datasets. To process complex-valued information such as SAR signals conveniently, the complex-valued convolutional neural network (CV-CNN) has been proposed in recent years, and it has achieved great success in SAR image recognition. This paper proposes an activation function with learnable parameters based on the Gaussian-type activation function (GTAF) in CV-CNN to improve the utilization of information in the real and imaginary parts of the neuro. For the multi-channel input of the feature map, this paper discusses two ways to set the parameters of the Gaussian-type activation function. One is that all channels share the same parameters, called the channel-sharing Gaussian-type activation function (CSGTAF). The other is that each channel has its independent parameters, called the channel-exclusive Gaussian-type activation function (CEGTAF). In addition, this paper derives the backpropagation formula of both CSGTAF and CEGTAF in detail for the training process of CV-CNN. This paper performs experimental analysis on three L-band standard PolSAR datasets. The experimental results show that, compared with the traditional method and the Gaussian activation function with fixed parameters, both CSGTAF and CEGTAF achieve higher recognition accuracy, and the difference in the recognition effect of different targets in the same dataset is little. Both show good recognition performance and have good stability and versatility. [ABSTRACT FROM AUTHOR]

In this paper, a three-step model based on the integration of Deep Neural Networks (DNN) and Decision Models is introduced for image classification which is inspired by the human visual system. To make a decision about an object, many actions should be done in a hierarchical process in the brain. First, the retina receives visual stimuli and transfers them to the visual cortex in the brain. The information extracted in the visual cortex, is accumulated over time to select an appropriate response. Many of the current decision-making models do not show how each image is converted into useful information for the decision model. Some models have used neural networks to convert each image into the information needed in the decision-making model; however, the role of the retina is ignored among these models. In this paper, a combination of retina inspired filters, CNN-based description and accumulator-based decision model is used to classify images. This model's structure resembles the human brain due to the usage of the DoG filter bank as retina inspired filter in the first stage of it. This model shows a significant improvement in accuracy in comparison to other models; furthermore, its performance is acceptable even with the small sample training set. [ABSTRACT FROM AUTHOR]

• This paper proposes a filter pruning approach for CNN model compression by transferring the knowledge of unimportant filters to the filters of higher importance. • Before pruning unimportant filters, a custom regularizer is utilized for knowledge transfer, which increases the gap between L1-norms of important and unimportant filters. • The effect of the penalty imposed in the custom regularizer is analyzed to justify the need for knowledge transfer before pruning. • In order to validate the robustness of the proposed framework across different CNN architectures, we experiment with five popular CNNs, namely, LeNet-5, VGG-16, ResNet-56, ResNet-110, and ResNet-50. • Experiments are performed on three benchmark datasets MNIST, CIFAR-10, and ImageNet. • An improvement over the baseline in terms of accuracy is observed even after removing 95.15%, 62.28%, and 62.39% of the Floating Point OPerations (FLOPs) from architectures LeNet-5, ResNet-56, and ResNet-110, respectively. As the deep learning models have been widely used in recent years, there is a high demand for reducing the model size in terms of memory and computation without much compromise in the model performance. Filter pruning is a very widely adopted strategy for model compression. The existing filter pruning methods identify the unimportant filters and prune them without worrying about information loss. They try to recover the same by fine-tuning the remaining filters, limiting their performance. In this paper, we tackle this problem by utilizing the knowledge from unimportant filters before pruning to minimize information loss. First, the proposed method identifies the unimportant and important filters by exploiting the lower and higher importance, respectively, using the L 1 -norm of filters. Next, the proposed custom UFKT-Reg regularizer ( R ufkt ) transfers the knowledge from unimportant filters before pruning to remaining filters, notably to a fixed number of important filters. Hence, the proposed method minimizes information loss due to the removal of unimportant filters. The experiments are conducted using the three benchmark datasets, including MNIST, CIFAR-10, and ImageNet. The proposed filter pruning method outperforms many recent state-of-the-art filter pruning methods. An improvement over the baseline in terms of accuracy is observed even after removing 95.15%, 62.28%, and 62.39% of the Floating Point OPerations (FLOPs) from architectures LeNet-5, ResNet-56, and ResNet-110, respectively. After pruning 53.25% of FLOPS from ResNet-50, only 1.02% and 0.47% of drops are observed in top-1 and top-5 accuracies, respectively. The code used in this paper will be publicly available at (https://github.com/sarvanichinthapalli/UFKT). [ABSTRACT FROM AUTHOR]

• An overview of deep learning algorithms used in medical image segmentation is presented. • More than 150 papers applying deep learning to different medical applications are summarised. • Challenges and future directions in medical image segmentation are discussed. Deep learning (DL) algorithms have rapidly become a robust tool for analyzing medical images. They have been used extensively for medical image segmentation as the first and significant components of the diagnosis and treatment pipeline. Medical image segmentation is efficiently addressed by many types of deep neural networks, such as convolutional neural networks, fully convolutional network recurrent networks, adversarial networks, and U-shaped networks. This paper reviews the major DL models and applications pertinent to medical image segmentation and summarizes over 150 contributions to the field. Brief overviews of articles are provided by application area: anatomical structures such as organs, bones, and vessels, and abnormalities such as lesions and calcification. Moreover, we discuss current challenges and suggest directions for future research. [ABSTRACT FROM AUTHOR]

This paper presents a formal verification guided approach for a principled design and implementation of robust and resilient learning-enabled systems. We focus on learning-enabled state estimation systems (LE-SESs), which have been widely used in robotics applications to determine the current state (e.g., location, speed, direction, etc.) of a complex system. The LE-SESs are networked systems, composed of a set of connected components including: Bayes filters for state estimation, and neural networks for processing sensory input. We study LE-SESs from the perspective of formal verification, which determines the satisfiabilty of a system model against the specified properties. Over LE-SESs, we investigate two key properties – robustness and resilience – and provide their formal definitions. To enable formal verification, we reduce the LE-SESs to a novel class of labelled transition systems, named {PO}2-LTS in the paper, and formally express the properties as constrained optimisation objectives. We prove that the verification problems are NP-complete. Based on {PO}2-LTS and the optimisation objectives, practical verification algorithms are developed to check the satisfiability of the properties on the LE-SESs. As a major case study, we interrogate a real-world dynamic tracking system which uses a single Kalman Filter (KF) – a special case of Bayes filter – to localise and track a ground vehicle. Its perception system, based on convolutional neural networks, processes a high-resolution Wide Area Motion Imagery (WAMI) data stream. Experimental results show that our algorithms can not only verify the properties of the WAMI tracking system but also provide representative examples, the latter of which inspired us to take an enhanced LE-SESs design where runtime monitors or joint-KFs are required. Experimental results confirm the improvement in the robustness of the enhanced design. [ABSTRACT FROM AUTHOR]

In personal identity recognition systems, detecting a person's age, gender, and language using voice signal characteristics is a crucial issue, especially because of the importance of security considerations. Age, gender, and language classification problems are important in signal processing because they are used to analyze and understand human behavior, interactions, and preferences. This can be especially useful in the fields of human-computer interaction, psychology, and social science research. In this paper, a new system for detecting a speaker's age, gender, and language based on deep learning models is presented. Deep learning models have shown great efficacy in various fields of signal processing. For this paper, a range of deep models were tested, including convolutional neural networks (CNNs), recurrent neural network (RNN), and a fine-tuning ResNet34 architecture. Additionally, techniques such as transfer learning were applied to improve the efficiency of the proposed system. The input voice signals are preprocessed by applying the spectro-temporal transform to obtain additional features that can be fed to the ResNet34 model, which is designed specifically for the task of voice signal processing. The dataset used in this paper was sourced from the Mozilla common voice initiative, which is dedicated to advancing speech recognition and language identification technologies. The performance of the proposed algorithm was evaluated in the presence of Gaussian noise to determine its robustness. The experimental results demonstrated that the proposed algorithm significantly outperformed basic algorithms and other deep neural networks in terms of age and gender recognition from voice signals. [ABSTRACT FROM AUTHOR]

Recently, many efforts have been concentrated on land use land cover (LULC) classification due to rapid urbanization, environmental pollution, agriculture drought, frequent floods, and climate change. However, various aspects have attracted hyperspectral imaging due to there being informative discriminative features, such as spectral-spatial features. To this end, this paper is a comprehensive and systematic review of LULC classification using hyperspectral images by reviewing four significant research investigations. Moreover, the four investigations have addressed the following points: (1) the main components of the hyperspectral imaging, the modes of hyperspectral imaging with data acquisition methods, and the intrinsic differences between hyperspectral image and multispectral image, (2) the role of machine learning in LULC classification, and the standard deep learning methods: Convolution Neural Network (CNN), Stacked Autoencoder (SAE), Deep Belief Network (DBN), Recurrent Neural Network (RNN), and Generative Adversarial Network (GAN), (3) the standard benchmark hyperspectral datasets and the evaluation criteria, (4) the main challenges of LULC classification with the possible solutions for the limited training samples issue, the promising future directions, and finally the recent applications for LULC classification. [ABSTRACT FROM AUTHOR]

Feature pyramid network (FPN) is widely used in object detection in order to divide and conquer objects of different scales and to fuse high and low-level features, and it has achieved encouraging achievements in multi-scale object processing. However, due to the mismatch between receptive fields at different stages, the direct fusion of the two features from different receptive fields may be unable to achieve satisfactory results. Moreover, simple lateral connections in FPN may lead to loss of spatial relationships and details. To alleviate these problems, in this paper we propose a field-matching attention network (FMANet) for object detection. Particularly, we first propose a receptive field dilated module (RFDM), which is used to normalize receptive fields between features at different stages to the same scale. Furthermore, to capture the spatial informations and details, we build a dual attention module (DAM) by employing the spatial attention and channel attention. Utilizing both spatial and channel attention mechanisms simultaneously improves performance while maintaining speed. Finally, experimental results reveal that our proposed FMANet with DSPDarkNet-53 as backbone achieves a competitive detection performance. [ABSTRACT FROM AUTHOR]

Convolutional neural networks have been widely used in image semantic segmentation. However, continuous downsampling operations in convolutional neural networks (such as pooling or convolution with step size) reduce the initial image resolution and lose the spatial details of the image, resulting in blurred image segmentation results. To alleviate this problem, in this paper we propose a multi-stage context refinement network (MCRNet) for semantic segmentation. Specifically, we first construct a Lowest-resolution Chain Context Aggregation (LCCA) module to encode rich semantic information. For obtaining more spatial detail information, we further build a High-resolution Context Attention Refinement (HCAR) module consisting of context feature extraction and context feature refinement. Finally, MCRNet fuses the context information generated by LCCA and HCAR for pixel prediction. Experimental results on three challenging semantic segmentation datasets, namely PASCAL VOC2012, ADE20K and Cityscapes, reveals that our proposed MCRNet is effective. [ABSTRACT FROM AUTHOR]

Recent studies show that gradient descent can train a deep neural network (DNN) to achieve small training and test errors when the DNN is sufficiently wide. This result applies to various over-parameterized neural network models including fully-connected neural networks and convolutional neural networks. However, existing theory does not apply to graph convolutional networks (GCNs), as GCNs is built according to the topological structures of the data. It has been empirically observed that GCNs can outperform vanilla neural networks when the underlying graph captures geometric information of the data. However, there is few theoretical justification of such observation. In this paper, we establish theoretical guarantees of the high-probability convergence of gradient descent for training over-parameterized GCNs. Specifically, we introduce a novel measurement of the relation between the graph and the data, called the "graph disparity coefficient", and show that the convergence of GCN is faster when the graph disparity coefficient is smaller. Our analysis provides novel insights into how the graph convolution operation in a GCN helps training, and provides useful guidance for GCN training in practice. [ABSTRACT FROM AUTHOR]

Graph convolutional neural network (GCN) has drawn increasing attention and attained good performance in various computer vision tasks, however, there is a lack of a clear interpretation of GCN's inner mechanism. For standard convolutional neural networks (CNNs), class activation mapping (CAM) methods are commonly used to visualize the connection between CNN's decision and image region by generating a heatmap. Nonetheless, such heatmap usually exhibits semantic-chaos when these CAMs are applied to GCN directly. In this paper, we proposed a novel visualization method particularly applicable to GCN, Vertex Semantic Class Activation Mapping (VS-CAM). VS-CAM includes two independent pipelines to produce a set of semantic-probe maps and a semantic-base map, respectively. Semantic-probe maps are used to detect the semantic information from the semantic-base map to aggregate a semantic-aware heatmap. Qualitative results show that VS-CAM can obtain heatmaps where the highlighted regions match the objects much more precisely than CNN-based CAM. The quantitative evaluation further demonstrates the superiority of VS-CAM. [ABSTRACT FROM AUTHOR]

Filter pruning is proven to be an effective strategy in model compression. However, convolutional filter pruning methods usually pay all attention to evaluating filters' importance at a single layer, ignoring their collaborative relationship with corresponding filters of the next layer. In this paper, we propose novel consecutive layer collaborative filter similarity (CLCS) to make full use of the complete filter information and learn binary selection vectors to prune the redundant filters automatically. With learned selection vectors, the pruning ratio of each layer can be determined, and we can also calculate the FLOPs of the candidate pruned network at the current stage. Under the accuracy constraint and the FLOPs constraint, the selection vectors of each layer can be optimized to achieve a better trade-off between accuracy and efficiency. Extensive experiments on CIFAR-10 and ImageNet with multiple networks demonstrate the effectiveness of our proposed method. Specifically, we obtain 54.29% and 67.33% FLOPs reduction with 0.01% and 0.09% accuracy improvement for ResNet-56 and ResNet-110 on CIFAR-10, respectively. On ImageNet, we reduce FLOPs by nearly half compared to the ResNet-50 baseline with almost no loss of accuracy. Compared with state-of-the-art filter pruning methods, our approach also achieves superior results. [ABSTRACT FROM AUTHOR]

The goal of semantic segmentation is to segment the input image according to semantic information and predict the semantic category of each pixel from a given label set. With the gradual intellectualization of modern life, more and more applications need to infer relevant semantic information from images for subsequent processing, such as augmented reality, autonomous driving, video surveillance, etc. This paper reviews the state-of-the-art technologies of semantic segmentation based on deep learning. Because semantic segmentation requires a large number of pixel-level annotations, in order to reduce the fine-grained requirements of annotation and reduce the economic and time cost of manual annotation, this paper studies the works on weakly-supervised semantic segmentation. In order to enhance the generalization ability and robustness of the segmentation model, this paper investigates the works on domain adaptation in semantic segmentation. Many types of sensors are usually equipped in some practical applications, such as autonomous driving and medical image analysis. In order to mine the association between multi-modal data and improve the accuracy of the segmentation model, this paper investigates the works based on multi-modal data fusion semantic segmentation. The real-time performance of the model needs to be considered in practical application. This paper analyzes the key factors affecting the real-time performance of the segmentation model and investigates the works on real-time semantic segmentation. Finally, this paper summarizes the challenges and promising research directions of semantic segmentation tasks based on deep learning. [ABSTRACT FROM AUTHOR]

In this paper, we present an extensive review on deep learning methods for image restoration tasks. Deep learning techniques, led by convolutional neural networks, have received a great deal of attention in almost all areas of image processing, especially in image classification. However, image restoration is a fundamental and challenging topic and plays significant roles in image processing, understanding and representation. It typically addresses image deblurring, denoising, dehazing and super-resolution. There are substantial differences in the approaches and mechanisms in deep learning methods for image restoration. Discriminative learning based methods are able to deal with issues of learning a restoration mapping function effectively, while optimisation models based methods can further enhance the performance with certain learning constraints. In this paper, we offer a comparative study of deep learning techniques in image denoising, deblurring, dehazing, and super-resolution, and summarise the principles involved in these tasks from various supervised deep network architectures, residual or skip connection and receptive field to unsupervised autoencoder mechanisms. Image quality criteria are also reviewed and their roles in image restoration are assessed. Based on our analysis, we further present an efficient network for deblurring and a couple of multi-objective training functions for super-resolution restoration tasks. The proposed methods are compared extensively with the state-of-the-art methods with both quantitative and qualitative analyses. Finally, we point out potential challenges and directions for future research. [ABSTRACT FROM AUTHOR]

Reconstructing natural image from its corresponding compressive sensing (CS) measurements is an ill-posed problem. Learning accurate prior of desirable image is essential to solve this inverse problem in high quality, especially for CS reconstruction from noisy measurements. The existing learning-based methods cannot effectively simulate whole potential noise in the testing data during external learning, and cannot effectively exploit the information from internal testing data. In this paper, we present an effective convolutional neural network (CNN) based method for CS reconstruction from noisy measurements, which learns the deep prior from an external dataset and internal noisy testing data with Stein's unbiased risk estimator (SURE). Specifically, we first pre-train an arbitrary CNN for CS reconstruction with an external dataset to learn a common prior. Then, we utilize meta learning to find a generic initial parameter that is suitable for fast internal learning and adaption to various noisy measurements. Finally, we customize the learned network to learn a specific prior for each internal testing data under Gaussian noise or more general mixed Poisson-Gaussian noise. Experimental results show that the proposed method outperforms the state-of-the-art methods under both comprehensively quantitative metrics and perceptive quality. [ABSTRACT FROM AUTHOR]

Session-based recommendation aims to predict a user's next action based on a series of anonymous sequences and plays an essential role in various online applications, such as e-commerce and music applications. Recently, transformer-based models have obtained results that are competitive with or even surpass those of recurrent neural networks, because of the good performance of transformer models in capturing long-distance dependencies. However, a transformer has a limited ability to mine local contextual information, which can be regarded as collective features. Researchers are seeking to address this limitation by augmenting the contextual transition to boost session representation learning. Accordingly, in this paper, we enhance the capabilities of a transformer in a session-based recommendation task by introducing convolutional neural networks (CNNs) at the stage of aggregating the item features with long- and short-distance dependencies. We first borrow a self-attention module from the classic transformer model to explore the long-distance dependencies. We next propose horizontal and vertical convolutions for enhancing the local collective information and then obtain a session representation by integrating the two types of features. Extensive experiments on real-world datasets show that our method outperforms those that rely on a transformer or a CNN alone. [ABSTRACT FROM AUTHOR]

Filter pruning has drawn extensive attention due to its advantage in reducing computational costs and memory requirements of deep convolutional neural networks. However, most existing methods only prune filters based on their intrinsic properties or spatial feature maps, ignoring the correlation between filters. In this paper, we suggest the correlation is valuable and consider it from a novel view: the frequency domain. Specifically, we first transfer features to the frequency domain by Discrete Cosine Transform (DCT). Then, for each feature map, we compute a uniqueness score, which measures its probability of being replaced by others. This way allows to prune the filters corresponding to the low-uniqueness maps without significant performance degradation. Compared to the methods focusing on intrinsic properties, our proposed method introduces a more comprehensive criterion to prune filters, further improving the network compactness while preserving good performance. In addition, our method is more robust against noise than the spatial ones since the critical clues for pruning are more concentrated after DCT. Experimental results demonstrate the superiority of our method. To be specific, our method outperforms the baseline ResNet-56 by 0.38 % on CIFAR-10 while reducing the floating-point operations (FLOPs) by 47.4 %. In addition, a consistent improvement can be observed when pruning the baseline ResNet-110: 0.23 % performance increase and up to 71 % FLOPs drop. Finally, on ImageNet, our method reduces the FLOPs of the baseline ResNet-50 by 48.7 % with only 0.32 % accuracy loss. [ABSTRACT FROM AUTHOR]

• We propose a new graph convolution network with subgraph embedding. • It extracts local information with subgraphs partitioned by Girvan Newman algorithm. • Experiments confirm the superiority of predicting mutagenicity in aromatic hydrocarbons. • The accuracy is verified with comparison of nine SOTA deep learning models. • The proposed method prevents about 15 %p of GCN's information dilution. An aromatic hydrocarbon refers to an organic material having a carbon ring such as benzene and a functional group in the carbon ring. As the industry develops, natural pollution becomes harsh, new compounds emerge, and the exposure to aromatic hydrocarbons is continuously increasing. Predicting mutagenicity is one of the crucial issues in reducing the risk because these organisms may have properties that penetrate the DNA of living things to cause mutations. Recently, the accuracy of mutation prediction has improved due to the power of deep learning. However, most conventional methods do not consider the characteristics of molecular aromatic hydrocarbons, which dilutes local information and results in a severe deterioration of the prediction performance. In this paper, we propose a method of exploiting subgraph convolution neural networks that enables the extraction of local information of a graph by partitioning it to maintain the detailed information. For extracting the features of molecules, we use the Girvan Newman algorithm to partition the graph according to the carbon ring and functional group and obtain the embedding vectors of the subgraphs as well as the original graph with graph convolution network (GCN). The embedding vectors are combined to represent the whole graph information and predict mutagenicity. Experiments with MUTAG, NCI1 and NCI109, datasets for predicting mutagenicity of molecules in graph structure, confirm that we successfully segment carbon rings and functional groups from molecular graphs and predict mutations using the partitioned graphs, leading to a 2 %p performance improvement. In addition, the proposed method has prevented about 15 %p of information dilution in GCN, and an analysis of the latent space of graphs reveals that the subgraphs extracted maintain the local information appropriately. [ABSTRACT FROM AUTHOR]

The difficulties in both data acquisition and annotation substantially restrict the sample sizes of training datasets for 3D medical imaging applications. Therefore, it is non-trivial to build well-performing 3D convolutional neural networks from scratch. Previous efforts on 3D pre-training have frequently relied on self-supervised approaches, which use either predictive or contrastive learning on unlabeled data to build invariant 3D representations. However, because of the unavailability of large-scale supervision information, obtaining semantically invariant and discriminative representations from these learning frameworks remains problematic. In this paper, we revisit an innovative yet simple fully-supervised 3D network pre-training framework to take advantage of semantic supervision from large-scale 2D natural image datasets. With a redesigned 3D network architecture, reformulated natural images are used to address the problem of data scarcity and develop powerful 3D representations. Comprehensive experiments on five benchmark datasets demonstrate that the proposed pre-trained models can effectively accelerate convergence while also improving accuracy for a variety of 3D medical imaging tasks such as classification, segmentation, and detection. In addition, as compared to training from scratch, it can save up to 60% of annotation efforts. On the NIH DeepLesion dataset, it also achieves state-of-the-art detection performance, outperforming earlier self-supervised and fully-supervised pre-training approaches, as well as methods that do training from scratch. To facilitate further development of 3D medical models, our code and pre-trained model weights are publicly available at https://github.com/urmagicsmine/CSPR. [ABSTRACT FROM AUTHOR]

To facilitate the deployment of convolutional neural networks on resource-limited devices, neural network pruning, especially filter pruning, has been shown to be a promising approach. Precise channel importance evaluation requires the extraction of global feature information. In this paper, we propose a novel filter pruning method, VNGEP, which combines intra- and inter-channel information to evaluate channel importance comprehensively. We describe the feature map correlation information from a graph perspective. The feature map tensor is converted into an undirected weighted graph, where the edge weights are the cosine distance between feature maps. To characterize the individual information of the feature maps, we present the feature map energy matrix and embed it into the above undirected weighted graph. Ultimately, the filter importance is measured by the von Neumann graph entropy, thus identifying filters that are less informative and more replaceable by other filters. Moreover, we systematically investigate the computational feasibility and data sensitivity of the proposed scheme. Pruning experimental results for several models on different datasets show the superiority of our approach. For example, with ResNet110, our method achieves a 48.3% and 52.1% reduction in model size and FLOPs, respectively, along with a 1.07% improvement in accuracy over the baseline model on CIFAR10. With ResNet50, we achieve an accuracy improvement of 0.25% over the baseline model on ImageNet, while the model size and FLOPs are reduced by 40.8% and 44.8%, respectively. [ABSTRACT FROM AUTHOR]

Active contour models (ACMs) are popular and widely used for many image segmentation applications and obtain promising results. However, these methods are unable to achieve the highest performance in the presence of intensity inhomogeneity. To address this issue, this paper presents an ACM based on the combination of a convolutional neural network (CNN) and a texture descriptor approach. This study uses a CNN model to generate parameter maps more effectively for ACM. Compared to conventional global techniques, these parameter maps increase the speed of movement of the contour into the target. In this approach, the Local Word Directional Pattern (LWDP) is applied as the texture descriptor. LWDP is a texture descriptor that uses the angle between two gradients for exploring the texture structure inside the image. In the proposed method both the original image and the new image obtained by the LWDP texture descriptor (encoded image) are provided as inputs to the CNN. The experimental outcomes show that the proposed strategy consistently outperforms the state-of-the-art in accuracy and robustness for segmenting images with fuzzy boundaries and intensity inhomogeneity. [ABSTRACT FROM AUTHOR]

• Introduces the novel task of Sexual Facial Expression Recognition (SFER). • Triple-BigGAN; a conditional GAN with a generator, discriminator, and classifier. • SEA-Faces-30k; a new dataset with Erotic, Suggestive-Erotic, and Non-Erotic classes. • Evaluation using FER-2013, ExpW, MNIST, CIFAR-10, SVHN, and SEA-Faces-30k datasets. Automatic recognition of facial images showing erotic expressions can help to understand our social interaction and to detect non-appropriate images even when there is no nakedness present in them. This paper contemplates, for the first time, to exploit facial cues applied to automatic Sexual Facial Expression Recognition (SFER). With this goal, we introduce a new dataset named Sexual Expression and Activity Faces (SEA-Faces-30k) for SFER, which contains 30k manually labeled images under three categories: erotic, suggestive-erotic, and non-erotic. Deep Convolutional Neural Networks require large-scale annotated image datasets with diversity and variations to be properly trained. Unfortunately, gathering such a massive amount of data is not feasible in this area. Therefore, we present a new semi-supervised GAN framework named Triple-BigGAN, which learns a generative model and a classifier simultaneously. It learns both tasks in an end-to-end fashion while using unlabeled or partially labeled data. The Triple-BigGAN framework shows promising classification performance for the SFER task (i.e., 93.59 %) and other five benchmark datasets, i.e., FER-2013, CIFAR-10, Expression in-the-Wild (ExpW), Modified National Institute of Standards and Technology database (MNIST), and Street View House Numbers (SVHN). Next, we evaluated the quality of samples generated by Triple-BigGAN with a resolution of 256 × 256 pixels using Inception Score (IS) and Frechet Inception Distance (FID). Our approach obtained the best FID (i.e., 19.94 %) and IS (i.e., 97.98 %) scores on the SEA-Faces-30k dataset. Further, we empirically demonstrated that synthetic erotic face images generated by Triple-BigGAN could also help in improving the classification performance of deep supervised networks. [ABSTRACT FROM AUTHOR]

Infrared and visible image fusion aims to integrate complementary information from different types of images into one image. The existing image fusion methods are primarily based on convolutional neural network (CNN), which ignores long-range dependencies of images, resulting in the fusion network unable to generate images with good complementarity. Inspired by the importance of global information, we introduced the transformer technique into the CNN-based fusion network as a way to improve the entire image-level perception in complex fusion scenarios. In this paper, we propose an end-to-end image fusion framework based on transformer and hybrid feature extractor, which enables the network to focus on both global and local information, using the characteristics of transformer to compensate for the shortcomings of CNN itself. In our network, the dual-branch CNN module is used to extract the shallow features of images, and then the vision transformer module is used to obtain the global channel and spatial relationship in the features. Finally, the fusion results are obtained through the image reconstruction module. We calculate the loss in the features of different depths according to the different kinds of original images by using the pre-trained VGG19 network. The experimental results show the effectiveness of adding the vision transformer module. Compared with other traditional and deep learning methods, our method achieves state-of-the-art qualitative and quantitative experiments performance. [ABSTRACT FROM AUTHOR]

Analysis of point clouds through deep convolutional neural networks is an active area of research due to their massive real-world applications including autonomous driving, indoor navigation, robotics, virtual/augmented reality, unmanned aerial vehicles, and drone technology. However, capturing the fine-grained geometric and semantic properties for the underlying recognition task with raw unstructured point cloud is highly challenging due to the lack of explicit neighborhood relationship and sparsity among the points. In this paper, we have introduced a deep, hierarchical 3D point based architecture for object classification and part segmentation that is able to learn robust geometric features which remain invariant to both the geometry and orientation of the local patches. The proposed architecture consists of multiple layers of sampling, concentric annular convolution, pooling, and residual feature propagation blocks. In the skip connections of our deep residual design, we propose to use a combination of linear projection shortcut and nonlinear ReLU group normalization shortcut with batch normalization, to improve both the optimization landscape and the representational power. Our network achieves on par or even better than state-of-the-art results on synthetic and real-world object classification (i.e., ModelNet40 and ScanObjectNN) and part segmentation (i.e., ShapeNet-part) benchmark datasets. The implementation and results have been made available at ...https://github.com/Rabbia-Hassan/Deep_Annular_Residual_Feature_Learning_for_3dPointCloudsGitHub-link [ABSTRACT FROM AUTHOR]

The large data scale and computational resources required by Convolutional Neural Networks (CNNs) hinder the practical application on mobile devices. However, channel pruning has become one of the most efficient methods for addressing this problem, with many existing researches proving its practicability in the field of model compression. The current channel pruning methods mainly start with the perspective of assessing the importance of channels or manual setting of the evaluation criteria, which requires unnecessary human intervention and shows the lack of certain automaticity. In this paper, an effective automatic channel pruning (EACP) method for neural networks is proposed. Specifically, we adopt the k-means++ method to cluster filters with similar features hierarchically in each convolutional layer, forming an initial compact compression structure. Subsequently, we use an improved social group optimization (SGO) algorithm to iteratively search and optimize the compression process of the post-clustered structure to find the optimal compressed structure. The effectiveness of the proposed approach is tested with respect to three leading CNN models on two image classification datasets. In CIFAR-10, our method reduces the FLOPs of GoogLeNet by 58.10 % and improves the accuracy by 0.20 % compared to the baseline. [ABSTRACT FROM AUTHOR]

Recognizing skin cancer in time could greatly increase patients' chances of recovery. For this reason, in recent years, numerous decision support systems have been proposed to help dermatologists in this diagnosis. These systems are generally based on Convolutional Neural Networks and are used for both segmentation and classification of lesions. Although their main goal is to correctly recognize the lesions' type, the preliminary segmentation step has been shown to increase the performance of the classifier. In fact, this is not surprising because physicians also use information on the shape of the lesion to make a diagnosis. Thanks to the ISIC archive, a huge number of skin lesion images, along with the corresponding metadata (type, position, dimension, etc.), are publicly available to train a deep neural network, but, unfortunately, only a small fraction of them are labeled for segmentation. To overcome this limitation, in this paper, a weak supervised approach is proposed to extract the segmentation label maps from the entire ISIC archive. Moreover, to demonstrate the quality of the proposed approach, the generated supervisions were first compared with those available in ISIC and, then, used to train a segmentation network, whose performance was evaluated against that obtained using only the small set of ISIC label maps. To foster reproducibility and to promote future research in lesion segmentation and classification, the generated ISIC Weak Segmentation Map (ISIC_WSM) dataset has been released. As far as we know, this is the first dataset that contains segmentation supervisions for clinical images of skin lesions. [ABSTRACT FROM AUTHOR]

Segmentation of retinal vessel images is critical to the diagnosis of retinopathy. Recently, convolutional neural networks have shown significant ability to extract the blood vessel structure. However, it remains challenging to refined segmentation for the capillaries and the edges of retinal vessels due to thickness inconsistencies and blurry boundaries. In this paper, we propose a novel deep neural network for retinal vessel segmentation based on shared decoder and pyramid-like loss (SPNet) to address the above problems. Specifically, we introduce a decoder-sharing mechanism to capture multi-scale semantic information, where feature maps at diverse scales are decoded through a sequence of weight-sharing decoder modules. Also, to strengthen characterization on the capillaries and the edges of blood vessels, we define a residual pyramid architecture which decomposes the spatial information in the decoding phase. A pyramid-like loss function is designed to compensate possible segmentation errors progressively. Experimental results on public benchmarks show that the proposed method outperforms the backbone network and most state-of-the-art methods, especially in the regions of the capillaries and the vessel contours. In addition, performances on cross-datasets verify that SPNet shows stronger generalization ability. [ABSTRACT FROM AUTHOR]

Singing melody extraction from polyphonic musical audio is one of the most challenging tasks in music information retrieval (MIR). Recently, data-driven methods based on convolutional neural networks (CNNs) have achieved great success for this task. In the literature, harmonic relationship has been proven crucial for this task. However, few existing CNN-based singing melody extraction methods consider the harmonic relationship in the training stage. The state-of-the-art CNN based methods are not capable of capturing such long-dependency harmonic relationship due to limited receptive field and unacceptable computation cost. In this paper, we introduce a neural harmonic-aware network with gated attentive fusion (NHAN-GAF) for singing melody extraction. Specifically, in the 2-D spectrograms modeling branch, we propose to employ multiple parallel 1-D CNN kernels to capture the harmonic relations between 1–2 octaves along the frequency axis in the spectrogram. Considering the advantage of jointly using Time–Frequency (T-F) domain and time domain information, we use two-branch neural nets to learn discriminative representation for this task. A novel gated attentive fusion (GAF) network is suggested to encode potential correlations between the two branches and fuse the descriptors learned from raw waveform and T-F spectrograms. Moreover, the idea of GAF can be exploited to the multimedia applications with multimodal analysis. With the two proposed components, our proposed model is capable of learning the harmonic relationship in the spectrogram and better capturing the contextual but discriminative features for singing melody extraction. We use part of the vocal tracks of the RWC dataset and MIR-1 K dataset to train the model and evaluate the performance of the proposed model on the ADC 2004, MIREX 05 and MedleyDB datasets. The experimental results show that the proposed method outperforms the state-of-the-art ones. [ABSTRACT FROM AUTHOR]

The state-of-the-art performance for several real-world problems is currently reached by deep and, in particular, convolutional neural networks (CNN). Such learning models exploit recent results in the field of deep learning, leading to highly performing, yet very large neural networks with typically millions to billions of parameters. As a result, such models are often redundant and excessively oversized, with a detrimental effect on the environment in terms of unnecessary energy consumption and a limitation to their deployment on low-resource devices. The necessity for compression techniques able to reduce the number of model parameters and their resource demand is thereby increasingly felt by the research community. In this paper we propose the first extensive comparison, to the best of our knowledge, of the main lossy and structure-preserving approaches to compress pre-trained CNNs, applicable in principle to any existing model. Our study is intended to provide a first and preliminary guidance to choose the most suitable compression technique when there is the need to reduce the occupancy of pre-trained models. Both convolutional and fully-connected layers are included in the analysis. Our experiments involved two pre-trained state-of-the-art CNNs (proposed to solve classification or regression problems) and five benchmarks, and gave rise to important insights about the applicability and performance of such techniques w.r.t. the type of layer to be compressed and the category of problem tackled. [ABSTRACT FROM AUTHOR]

• We propose ReNAP for few-shot classification. • The APL module of ReNAP can learn class prototypes adaptively. • Experimental results demonstrate the superior classification performance of ReNAP. Traditional deep learning-based image classification methods often fail to recognize a new class that does not exist in the training dataset, particularly when the new class only has a small number of samples. Such a challenging and new learning problem is referred to as few-shot learning. In few-shot learning, the relation network (RelationNet) is a powerful method. However, in RelationNet and its state-of-the-art variants, the prototype of each class is obtained by a simple summation or average over the labeled samples. These simple sample statistics cannot accurately capture the distinct characteristics of the diverse classes of real-world images. To address this problem, in this paper, we propose the Relation Network with Adaptive Prototypical Learning method (ReNAP), which can learn the class prototypes adaptively and provide more accurate representations of the classes. More specifically, ReNAP embeds an adaptive prototypical learning module constructed by a convolutional network into RelationNet. Our ReNAP achieves superior classification performances to RelationNet and other state-of-the-art methods on four widely used benchmark datasets, FC100, CUB-200-2011, Stanford-Cars, and Stanford-Dogs. [ABSTRACT FROM AUTHOR]

• We propose a novel Transformers and CNNs Fusion Network for SOD, which can efficiently extract and combine global semantic context and local detail features. • We design a Bi-Enhancement Fusion Module to effectively fuse the obtained features from Transformers and CNNs branches to make predictions jointly. • We design multi-modality fusion modules (CMST and DEM) for Transformers and CNNs branches to effectively integrate RGB cues and depth cues for RGB-D SOD. Recently, Salient Object Detection (SOD) has achieved promising results thanks to the rapid evolution of CNN architectures. However, those CNN-based methods show limited capacity in modeling long-range interaction among pixels. In this paper, we propose to combine the merits from CNN and Transformer and design a unified model for RGB and RGB-D SOD. First, we represent the image with a sequence-to-sequence perspective and use a Transformer-based branch to express long-distance relationships of image tokens to obtain global semantic information and predict a coarse saliency map. Second, we employ a CNN-based branch to extract multi-scale local detail features to predict contour prediction for auxiliary supervision at each level. Finally, we propose the Bi-enhancement Fusion Module to fuse multi-scale cues from two branches to predict a more accurate saliency map. In addition, for RGB-D SOD, to obtain effective cross-modality features, we propose a Cross-modality Multi-Scale Transformer Module and a Depth-induced Enhancement Module to fuse RGB and depth cues in the Transformers branch and the CNNs branch, respectively. Experiments on both RGB and RGB-D SOD datasets demonstrate that our proposed model achieves satisfactory performance compared with state-of-the-art methods. [ABSTRACT FROM AUTHOR]

This paper considers the multi-label thoracic abnormality classification with chest X-ray images. In clinical settings, Chest X-ray imaging is a general diagnostic tool applied to visualize numerous thoracic pathological changes. While deep learning techniques have been extensively tested in this field, certain challenges persist. The data in existing thoracic abnormality datasets is insufficient, and some diseases are extremely imbalanced. Meanwhile, the dependencies between different labels are often ignored. To tackle these issues head-on, this paper introduces two crucial modules: the group-wise spatial attention (GWSA) module and the label co-occurrence dependency (LCD) module, integrated with DenseNet121 backbone. Specifically, GWSA enhances the spatial features within distinct groups while keeping the between-group feature discrimination. LCD models the correlations between different thoracic abnormalities to refine the predicted probabilities. In conjunction with the DenseNet121 backbone, these two modules reach an average AUC score of 0.818 on Chest X-ray14 dataset, achieving state-of-the-art. Source code is available at https://github.com/YujiaKCL/Chest-Xray14-GWSA-LCD. [ABSTRACT FROM AUTHOR]

As the smallest structural unit of feature mapping, the convolution kernel in a deep convolution neural networks (DCNN) convolutional layer is responsible for the input channel features to output channel features. A specific convolution kernel belongs to a specific group from the perspective of the input channel, and it belongs to a specific filter from the perspective of the output channel. If the input and output channels are simultaneously considered in the pruning process, the performance of the pruning model can be further improved. This paper proposes progressive kernel pruning with salient mapping of input-output channels, introduces the concept of input-output channel saliency and defines single-port salient mapping channels and dual-port salient mapping channels. This study demonstrates that single-port salient mapping channels can ensure that each input channel signal has a relatively strong convolution kernel mapped to the output channel, and vice versa. The dual-port salient mapping channel is a channel with high feature mapping abilities from both the input and output directions. Additionally, the average mapping ability measure index is defined, which is used to control the kernel pruning process of the single-port salient mapping channel to switch to the kernel pruning process of the dual-port salient mapping channel. The experimental results and analysis show that the method proposed in this paper can be used to effectively prune a heavyweight model and a lightweight model and can obtain a better accuracy under the conditions of higher compression ratio and acceleration ratios. For example, when VGG-16 is pruned on CIFAR-10, the compression ratio and acceleration ration are 91.00 × and 15.24 × , respectively, and the classification accuracy of the model decreased slightly by 0.22%. When ResNet-101 is pruned on ImageNet, the compression ratio and acceleration ratio are 3.90 × and 3.38 × , respectively, and the classification accuracy of the model decreased slightly by 0.48%. The proposed method is significantly better than state-of-the-art methods. [ABSTRACT FROM AUTHOR]

• This paper presents a memory augmented convolutional neural network. • This paper utilizes the augmenting SOM as memory for better network generalization. • This paper instantiates the distributed knowledge representation with SOM and CNN. The long short-term memory (LSTM) network underpins many achievements and breakthroughs especially in natural language processing fields. Essentially, it is endowed with certain memory capabilities to boost its performance. Currently, the volume and speed of big data generation are increasing exponentially, and such data require efficient models to acquire memory augmented knowledge. In this paper, we propose a memory augmented convolutional neural network (MACNN) with utilizing self-organizing maps (SOM) as the memory module. First, we depict the potential challenge about just applying solely a convolutional neural network (CNN) so as to highlight the advantage of augmenting SOM memory for better network generalization. Then, we dissert a corresponding network architecture incorporating memory to instantiate the distributed knowledge representation machanism, which tactically combines both SOM and CNN. Each component of the input vector is connected with a neuron in a two-dimensional lattice. Finally, we test the proposed network on various datasets and the experimental results reveal that MACNN can achieve competitive performance, especially for bioimages datasets. Meanwhile, we further illustrate the learned representations to interpret the SOM behavior and to comprehend the achieved results, which indicates that the proposed memory-incorporating model can exhibit the better performance. [ABSTRACT FROM AUTHOR]

[Display omitted] • We structure the GDN and LRN with translational invariance as the primary consideration. • We implement the multi-task strategy to improve the GDN's generalization. • The proposed method has robust translational invariance. Recently, intelligent cephalometric landmark detection systems that employ deep convolutional neural networks (DCNNs) have been widely developed. In this paper, we concentrate on two challenging difficulties of DCNNs based cephalometric landmark detection. First, previous DCNN methods usually decreased the resolution of high-resolution cephalometric images. Hence, the trained DCNNs missed the detailed local features of original images. Moreover, the detected landmarks' locations were recovered using a fixed ratio, so location error was increased after recovery. Second, the performance of cephalometric landmark detection is dependent on DCNNs' translational invariance because the target landmarks' locations are influenced by any shift of the cephalometric images. However, modern DCNNs have limited translational invariance, and previous DCNN methods did not take translational invariance into consideration. In this paper, we developed the widely used two-stage framework to resolve the above two challenges. In the first stage, we train a global detection network, which receives the whole images as input, to generate candidate landmarks. In the second stage, we split the input image into patches and train a local refine network, which receives image patches as input, to refine the landmarks' locations. The proposed local refine network can produce accurate landmarks' locations because it entirely focuses on the detailed local features of high-resolution cephalometric images. The global detection network and local refine network are structured with translational invariance as their primary consideration. The proposed method achieved robust performance on a private dataset and the public Automatic Cephalometric X-ray Landmark Detection Challenge 2015 dataset. [ABSTRACT FROM AUTHOR]

• Global Context Feature Extraction (GCFE) module is proposed, which can extract multi-scale global contextual information. • A Dual Scale Feature Aggregation (DSFA) module is proposed to fuse the features from different scales. • A learnable Gabor convolution layer is designed to enhance its ability to capture the edge and contour information of object. • A Spatial Attention (SA) mechanism is designed to further enhance the spatial structure information of object. Prohibited Object Detection (POD) in X-ray images plays an important role in protecting public safety. Automatic and accurate POD is required to relieve the working pressure of security inspectors. However, the existing methods cannot obtain a satisfactory detection accuracy, and especially, the problem of object occlusion also has not been solved well. Therefore, in this paper, according to the specific characteristics of X-ray images as well as low-level and high-level features of Convolutional Neural Network (CNN), different feature enhancement strategies have been elaborately designed for occluded POD. First, a learnable Gabor convolutional layer is designed and embedded into the low layer of the network to enhance the network's capability to capture the edge and contour information of object. A Spatial Attention (SA) mechanism is then designed to weight the output features of the Gabor convolutional layer to enhance the spatial structure information of object and suppress the background noises simultaneously. For the high-level features, Global Context Feature Extraction (GCFE) module is proposed to extract multi-scale global contextual information of object. And, a Dual Scale Feature Aggregation (DSFA) module is proposed to fuse these global features with those of another layer. To verify the effectiveness of the proposed modules, they are embedded into typical one-stage and two-stage object detection frameworks, i.e., Faster R-CNN and YOLO v5L, obtaining POD-F and POD-Y methods, respectively. The proposed methods have been extensively evaluated on three publicly available benchmark datasets, namely SIXray, OPIXray and WIXray. The experimental results show that, compared with existing methods, the proposed POD-Y method can achieve a state-of-the-art detection accuracy. And POD-F can also achieve a competitive detection performance among the two-stage detection methods. 1 1 https://github.com/Open-my-paper-code/POD. [ABSTRACT FROM AUTHOR]

Recent advances in multi-modal feature fusion boost the development of RGB-D salient object detection (SOD), and many remarkable RGB-D SOD models have been proposed. However, though some existing methods consider fusing the cross-level multi-modal features, they ignore the difference between inter-level having the multi-modal details in convolutional neural networks (CNNs) based RGB-D SOD. Therefore, exploring the correlations and differences of cross-level multi-modal features is a critical issue. In this paper, we present a novel depth-aware inverted refinement network (DAIR) to progressively guide the cross-level multi-modal features through backward propagation, which considerably preserves the different level details with multi-modal cues. Specifically, we innovatively design an end-to-end inverted refinement network to guide cross-level and cross-modal learning for revealing complementary relations of the cross-modal. The inverted refinement network also refines the low-level spatial details by the high-level global contextual cues. In particular, considering the difference of multi-modal and the effect of depth quality, a depth-aware intensified module (DAIM) is proposed with capturing the paired relationship of the pixel-level and inter-channel for the depth map. It promotes the representative capability of the depth details. Extensive experiments on nine challenging RGB-D SOD datasets demonstrate remarkable performance boosting of our proposed model against the fourteen state-of-the-art (SOTA) RGB-D SOD approaches. [ABSTRACT FROM AUTHOR]

Image-text matching aims to make a connection between visual and natural information. Some of the current methods have made great progress by using global alignment between images and sentences and local alignment between the image region and its corresponding word. However, the importance of the correlation between global alignment and local alignment is ignored to some extends. Therefore, in this paper, we propose a new region feature enhancement-based image text similarity inference network. Firstly, the image region feature is enhanced by graph convolutional neural network, which is used to find the correlation among different image region features with the production of features' sematic relationship. Secondly, we propose a vector-based similarity representation to describe local and global alignment with a more comprehensive way. Finally, a graph convolution neural network is introduced to construct a similarity graph for propagating correlation between local alignment and global alignment to every part. By testing on the MSCOCO and Flickr30k dataset, our proposed method shows great accuracy performance and competitiveness. [ABSTRACT FROM AUTHOR]

Low-light image enhancement studies how to improve the quality of images captured under poor lighting conditions, which is of real-world importance. Currently, convolutional neural network (CNN)-based methods with state-of-the-art performance have become the mainstream of research. However, most CNN-based methods improve the performance of the algorithm by increasing the width and depth of the neural network, which requires large computing device resources. In this paper, we propose a knowledge distillation method for low light image enhancement. The proposed method uses a teacher-student framework in which the teacher network tries to transfer the rich knowledge to the student network. The student network learns the knowledge of image enhancement under the supervision of ground truth images and under the guidance of the teacher network simultaneously. Knowledge transfer between the teacher-student network is accomplished by distillation loss based on attention maps. We designed a gradient-guided low-light image enhancement network that can be divided into an enhancement branch and a gradient branch, where the enhancement branch is learned under the guidance of the gradient branch to better preserve structural information. The teacher and student networks use a similar structure, but they have different model sizes. The teacher network has more parameters and more powerful learning capabilities than the student network. With the help of knowledge distillation, our approach can improve the performance of the student network without increasing the computational burden during the testing phase. The qualitative and quantitative experimental results demonstrate the superiority of our method compared to the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Fine-grained visual categorization is an important task in computer vision. Prior works on fine-grained visual categorization have paid much attention to addressing intra-class variation and inter-class similarity. However, they rarely study that task from the perspective of probability distribution. In this paper, we propose a novel refined probability distribution module based on deep convolutional neural network. Our module computes the probability of an image by fully utilizing the similarity information between images. Firstly, we use deep neural networks to obtain the initial probability distribution and extract features. Then, we build a network whose inputs are features for calculating image-to-image similarity scores. Finally, our module refines the initial probability distribution based on an effective batch random walk operation with similarity scores. Our module can be plugged into many deep convolutional neural networks. Experimental results show that our approach outperforms state-of-the-art methods on the CUB-200–2011, FGVC-Aircraft and Stanford Cars datasets respectively. [ABSTRACT FROM AUTHOR]

• Basic bibliometric indexes are utilized to characterize the global trends of medical image research from 1993 to 2022. • Network-based methods are employed to analyze the co-word, co-authorship, and co-citation relations. • Analysis shows the present medical image research is on an upward trend. • Medical image research starts to speed up after 2015 due to the application of convolutional neural networks. With the development of artificial intelligence and high-performance computing equipment, new technologies have huge effects on medical image research. However, it is difficult to find out when new research topics appear, who those authors with influences are, and how relevant publications influence the academic community. In order to catch up with global research trends, traditional methods of literature review are inadequate to acquire information. In this case, a data-driven analysis offers a new quantitative approach to studying global research trends. Specifically, this paper used several basic bibliometric indexes to characterize the global trends of medical image research from 1993 to 2022, including yearly output, active journals, important authors, active institutions, and main countries. Furthermore, we utilized network-based methods to analyze the internal relations of co-word, co-authorship, and co-citation, so as to discover academic hotspots and clarify global research trends. Finally, some conclusions are drawn as follow: (1) The present medical image research is on an upward trend. The number of publications on medical image surged since 2015 due to advances in deep learning. Deep learning and convolutional neural networks (CNNs) are both popular research keywords in recent years. (2) IEEE Transactions on Medical Imaging is the most influential journal in view of Total Local Citation Score (TLCS) and Total Global Citation Score (TGCS), followed by Medical Image Analysis. Neurocomputing and Information Fusion are well-recognized in local research community. (3) Van Ginneken B and Aerts HJWL are representative scholars in consideration of TLCS and TGCS. (4) The USA is a leading country in medical image research. Other influential countries include China, India, UK, Germany, France, Canada, Netherlands, Australia, Italy, South Korea, Switzerland, etc. Most important institutions are from these countries, including Harvard , UMich , Stanford , UPenn , UNC , CAS , SJTU , UCL , UofT , RU , etc. (5) Application of artificial intelligence technologies, especially CNNs, has dramatically promoted global studies of medical image since 2015. Interdisciplinary collaborations become popular among experts with different disciplines backgrounds. We can infer that medical image analysis and application based on deep learning will still be a flourishing field in the near future with the improvement of algorithms and the application of high-performance computing equipment. [ABSTRACT FROM AUTHOR]

• A novel deep learning model for laser-induced damage change detection is proposed. • The proposed model combines siamese CNN with visual attention mechanism. • A novel SoftmaxFocal loss is put forward to train the network. • Our model outperforms four different methods overall on three real datasets. With high-energy laser irradiating, the laser-induced damages may occur in the surfaces of optical elements in laser facilities. As the laser-induced damage changes can badly affect regular and healthy operation of laser facilities, it is essential to effectively detect real damage changes while suppressing meaningless and spurious changes in captured optical images. In order to achieve high-precision laser-induced damage change detection, this paper presents a novel deep learning model which exploits visual attention-based siamese convolutional neural network with SoftmaxFocal loss and significantly improves the performance of damage change detection. In the proposed model, an end-to-end classification network is designed and trained which fuses the spatial-channel domain collaborative attention modules into siamese convolutional neural network thus achieving more efficient feature extraction and representation. For the purpose of addressing the unbalanced distribution of hard and easy samples, a novel loss function which is termed as SoftmaxFocal loss is put forward to train the proposed network. The SoftmaxFocal loss creatively introduces an additive focusing term into original softmax loss which greatly enhances the online hard sample mining ability of the proposed model. Experiments conducted on three real datasets demonstrate the validity and superiority of the proposed model. [ABSTRACT FROM AUTHOR]

The combination of convolutional and recurrent neural networks is a promising framework. This arrangement allows the extraction of high-quality spatio-temporal features together with their temporal dependencies. This fact is key for time series prediction problems such as forecasting, classification or anomaly detection, amongst others. In this paper, the TSFE DL library is introduced. It compiles 22 state-of-the-art methods for both time series feature extraction and prediction, employing convolutional and recurrent deep neural networks for its use in several data mining tasks. The library is built upon a set of Tensorflow+Keras and PyTorch modules under the AGPLv3 license. The performance validation of the architectures included in this proposal confirms the usefulness of this Python package. [ABSTRACT FROM AUTHOR]

Unsupervised domain adaptation (UDA) has shown remarkable results in fault diagnosis under changing working conditions in recent years. However, most UDA methods do not consider the geometric structure of the data. Furthermore, the global domain adaptation technique is commonly applied, which ignores the relation between subdomains. This paper addresses mentioned challenges by presenting the novel deep subdomain adaptation graph convolution neural network (DSAGCN), which has two key characteristics: First, a graph convolution neural network (GCNN) is employed to model the structure of data. Second, adversarial domain adaptation and local maximum mean discrepancy (LMMD) methods are applied concurrently to align the subdomain's distribution and reduce structure discrepancy between relevant subdomains and global domains. CWRU, PU, and JNU bearing datasets are used to validate the DSAGCN method's superiority between comparison models. The experimental results demonstrate the significance of aligning structured subdomains along with domain adaptation methods to obtain an accurate data-driven model. [ABSTRACT FROM AUTHOR]

• New subspace-based method for image set recognition. • Theoretically-grounded integration of subspace representation into DNNs while keeping end-to-end trainability. • Our new method generalizes the classic learning subspace method. This paper addresses the problem of object recognition given a set of images as input (e.g., multiple camera sources and video frames). Convolutional neural network (CNN)-based frameworks do not exploit these sets effectively, processing a pattern as observed, not capturing the underlying feature distribution as it does not consider the variance of images in the set. To address this issue, we propose the Grassmannian learning mutual subspace method (G-LMSM), a NN layer embedded on top of CNNs that can process image sets more effectively and can be trained in an end-to-end manner. The image set is first represented by a low-dimensional input subspace and then this input subspace is matched with dictionary subspaces by a similarity of their canonical angles, an interpretable and easy to compute metric. The key idea of G-LMSM is that the dictionary subspaces are learned as points on the Grassmann manifold, optimized with Riemannian stochastic gradient descent. This learning is stable, efficient and theoretically well-grounded. We demonstrate the effectiveness of our proposed method on hand shape recognition, face identification, and facial emotion recognition. [ABSTRACT FROM AUTHOR]

Acquiring incomplete k-space matrices is an effective way to accelerate Magnetic Resonance Imaging (MRI). It is an important and challenging task to accurately reconstruct images from such under-sampled k-space matrices. On the one hand, neither image-domain oriented nor frequency-domain oriented deep Convolutional Neural Networks can simultaneously employ both frequency features and spatial features for cooperatively improving reconstruction accuracy. On the other hand, existing dual-domain reconstruction methods adopt heavy encoder-decoder frameworks, resulting in low efficiency and information loss in the process of pooling. To deal with these problems, in this paper, we propose a full-resolution dual-domain reconstruction network, called DIIK-Net. The DIIK-Net consists of a full-resolution frequency-domain branch, a full-resolution image-domain branch, and cross-domain interaction modules between the two branches. The first novelty of the proposed method is that the features of each block of frequency-domain branch are extracted by 1 × 1 filters, which reduces computational cost and captures rich contextual information. Due to the fact that an element in frequency domain conveys information of the whole image, 1 × 1 convolutional blocks are able to extract large contextual information with the interaction of image domain. The second novelty is that the image-domain branch consists of a very small number of 3 × 3 convolutional blocks and each block has very large field of perception due to integration of frequency domain. The third novelty lies in the simple and effective cross-domain interaction module. Experimental results on the challenging fastMRI dataset demonstrate that the proposed method is capable of achieving higher reconstruction accuracy with a few number of parameters. [ABSTRACT FROM AUTHOR]

In this paper, a low cost method for input size reduction without sacrificing accuracy is proposed, which reduces required computation resources for both training and inference of deep convolutional neural network (DCNN) in the steering control of self-driving cars. Efficient processing of DCNNs is becoming prominent challenge due to its huge computation cost, number of parameters and also inadequate computation resources on power efficient hardware devices and the proposed method alleviates the problem comparing the state of the art. The proposed method introduces feature density metric (FDM) as criterion to mask and filter out regions of input image that do not contain adequate amount of features. This filtering method prevents DCNN from useless calculations belongs to feature-free regions. Compared to PilotNet, the proposed method accelerates overall training and inference phases of end-to-end (ETE) deep steering control of self-driving cars up to 1.3× and 2.0× respectively. [ABSTRACT FROM AUTHOR]