Search

• The ADNMT uses two asymmetric networks to generate pseudo-labels for each other by clustering and iterative training. • Based on alternate training and mutual teaching to optimize the pseudo-labels. • The SCIC compensates the similarity distance of images captured by inter-camera. • The SSIC further suppresses the similarity distance by calculate the similarity distribution. Unsupervised person re-identification (Re-ID) has always been challenging in computer vision. It has received much attention from researchers because it does not require any labeled information and can be freely deployed to new scenarios. Most unsupervised person Re-ID research studies produce and optimize pseudo-labels by iterative clustering algorithms on a single network. However, these methods are easily affected by noisy labels and feature variations caused by camera shifts, which will limit the optimization of pseudo-labels. In this paper, we propose an Asymmetric Double Networks Mutual Teaching (ADNMT) architecture that uses two asymmetric networks to generate pseudo-labels for each other by clustering, and the pseudo-labels are updated and optimized by alternate training. Specifically, ADNMT contains two asymmetric networks. One network is a multiple granularity network, which extracts pedestrian features of multiple granularity that correspond to numerous classifiers, and the other network is a conventional backbone network, which extracts pedestrian features that correspond to a classifier. Furthermore, because the camera style changes seriously affect the generalization ability of the proposed model, this paper designs Similarity Compensation of Inter-Camera (SCIC) and Similarity Suppression of Intra-Camera (SSIC) according to the camera ID of the pedestrian images to optimize the similarity measure. Extensive experiments on multiple Re-ID benchmark datasets show that our proposed method achieves superior performance compared with the state‐of‐the‐art unsupervised person re-identification methods. [ABSTRACT FROM AUTHOR]

Deep neural networks have achieved outstanding performance in computer vision tasks. Convolutional Neural Networks (CNNs) typically operate in the spatial domain with raw images, but in practice, images are usually stored and transmitted in their compressed representation where JPEG is one of the most widely used encoder. Also, these networks are computationally intensive and slow. This paper proposes performing the learning and inference processes in the compressed domain in order to reduce the computational complexity and improve the speed of popular CNNs. For this purpose, a novel graph-based frequency channel selection method is proposed to identify and select the most important frequency channels. The computational complexity is reduced by retaining the important frequency components and discarding the insignificant ones as well as eliminating the unnecessary layers of the network. Experimental results show that the modified ResNet-50 operating in the compressed domain is up to 70% faster than the spatial-based traditional ResNet-50 while resulting in similar classification accuracy. Moreover, this paper proposes a preprocessing step with partial encoding to improve the resilience to distortions caused by low-quality encoded images. Finally, we show that training a network with highly compressed data can achieve a good classification accuracy with up to 93% reduction in the storage requirements of the training data. [ABSTRACT FROM AUTHOR]

In this paper we describe the design and the ideas motivating a new Continual Learning benchmark for Autonomous Driving (CLAD), that focuses on the problems of object classification and object detection. The benchmark utilises SODA10M, a recently released large-scale dataset that concerns autonomous driving related problems. First, we review and discuss existing continual learning benchmarks, how they are related, and show that most are extreme cases of continual learning. To this end, we survey the benchmarks used in continual learning papers at three highly ranked computer vision conferences. Next, we introduce CLAD-C, an online classification benchmark realised through a chronological data stream that poses both class and domain incremental challenges; and CLAD-D, a domain incremental continual object detection benchmark. We examine the inherent difficulties and challenges posed by the benchmark, through a survey of the techniques and methods used by the top-3 participants in a CLAD-challenge workshop at ICCV 2021. We conclude with possible pathways to improve the current continual learning state of the art, and which directions we deem promising for future research. • A review of current continual learning benchmarks and their relations to one-another. • Proposal of two new real-world continual learning benchmarks. • Review of the most successful ICCV'21 challenge entries on the new benchmarks. [ABSTRACT FROM AUTHOR]

Image Salient Object Detection (SOD) is a fundamental research topic in the area of computer vision. Recently, the multimodal information in RGB, Depth (D), and Thermal (T) modalities has been proven to be beneficial to the SOD. However, existing methods are only designed for RGB-D or RGB-T SOD, which may limit the utilization in various modalities, or just finetuned on specific datasets, which may bring about extra computation overhead. These defects can hinder the practical deployment of SOD in real-world applications. In this paper, we propose an end-to-end U nified T riplet D ecoder Net work, dubbed UTDNet, for both RGB-T and RGB-D SOD tasks. The intractable challenges for the unified multimodal SOD are mainly two-fold, i.e., (1) accurately detecting and segmenting salient objects, and (2) preferably via a single network that fits both RGB-T and RGB-D SOD. First, to deal with the former challenge, we propose the multi-scale feature extraction unit to enrich the discriminative contextual information, and the efficient fusion module to explore cross-modality complementary information. Then, the multimodal features are fed to the triplet decoder, where the hierarchical deep supervision loss further enable the network to capture distinctive saliency cues. Second, as to the latter challenge, we propose a simple yet effective continual learning method to unify multimodal SOD. Concretely, we sequentially train multimodal SOD tasks by applying Elastic Weight Consolidation (EWC) regularization with the hierarchical loss function to avoid catastrophic forgetting without inducing more parameters. Critically, the triplet decoder separates task-specific and task-invariant information, making the network easily adaptable to multimodal SOD tasks. Extensive comparisons with 26 recently proposed RGB-T and RGB-D SOD methods demonstrate the superiority of the proposed UTDNet. [ABSTRACT FROM AUTHOR]

In this paper, we investigate the problem of causal image classification with multi-label learning. As multi-label learning involves a diversity of supervision signals, it is considered a challenging issue to solve. Previous approaches have attempted to improve performance by identifying label-related image areas or exploiting the co-occurrence of labels. However, these methods are often characterized by complicated procedures, tedious computations, and a lack of intuitive interpretations. To overcome these limitations, we propose a novel approach that incorporates the concept of causal inference, which has been shown to be beneficial in other computer vision problems. Our method, called causal multi-label learning (CMLL), enables the selection of multiple objects from the original image through a multi-class attention module. These objects are then subjected to causal intervention to learn the causal relationships between different labels. Our proposed approach is both elegant and effective, with low computational cost and few parameters required for the multi-class causal intervention approach. Extensive tests and ablation studies demonstrate that the proposed method significantly improves prediction performance without a significant increase in training and inference times. • A Causal Multi-Label Learning (CMLL) mechanism mitigates spurious label associations. • A causal intervention strategy with multi-class attention maps is proposed. • Evaluations show CMLL's effectiveness on diverse datasets and baselines. [ABSTRACT FROM AUTHOR]

Anomaly detection (AD) plays a crucial role in various domains, including cybersecurity, finance, and healthcare, by identifying patterns or events that deviate from normal behavior. In recent years, significant progress has been made in this field due to the remarkable growth of deep learning models. Notably, the advent of self-supervised learning has sparked the development of novel AD algorithms that outperform the existing state-of-the-art approaches by a considerable margin. This paper aims to provide a comprehensive review of the current methodologies in self-supervised anomaly detection. We present technical details of the standard methods and discuss their strengths and drawbacks. We also compare the performance of these models against each other and other state-of-the-art anomaly detection models. Finally, the paper concludes with a discussion of future directions for self-supervised anomaly detection, including the development of more effective and efficient algorithms and the integration of these techniques with other related fields, such as multi-modal learning. • We present a cohesive overview of self-supervised methods for anomaly detection. • We categorize existing self-supervised anomaly detection algorithms. their proxy tasks. • We conduct a comprehensive performance comparison. • We discuss future directions in self-supervised anomaly detection. [ABSTRACT FROM AUTHOR]

The single image de-snowing task is an essential topic in computer vision, as images captured on snowy days degrade the performance of current vision-based intelligent systems. Existing methods build complex network structures with numerous parameters to pursue continuous performance improvement. Nonetheless, they generally ignore the negative impact of large memory consumption in real applications. This paper aims to address the above problem by making a trade-off between network capacity and performance. We propose two novel networks suitable for different application scenarios. For devices with small memory and requiring fast inference speed, we propose an extremely lightweight recursive network (XLRNet). XLRNet is constructed by a single recursive strategy and two novel lightweight modules. For devices with large memory and pursuing better de-snowing performance, we propose a coupled lightweight dual recursive network (CLDRNet). CLDRNet cascades two XLRNets by a novel dual recursive strategy and a novel dual coupled LSTM module (DC-LSTM). Extensive experiments demonstrate the effectiveness and superiority of our two models on three synthetic datasets and real-world datasets. [ABSTRACT FROM AUTHOR]

Image registration is a fundamental problem in computer vision and robotics. Recently, learning-based image registration methods have made great progress. However, these methods are sensitive to abnormal transformation and have insufficient robustness, which leads to more mismatched points in the actual environment. In this paper, we propose a new registration framework based on ensemble learning and dynamic adaptive kernel. Specifically, we first use a dynamic adaptive kernel to extract deep features at the coarse level to guide fine-level registration. Then we added an adaptive feature pyramid network based on the integrated learning principle to realize the fine-level feature extraction. Through different scale, receptive fields, not only the local geometric information of each point is considered, but also its low texture information at the pixel level is considered. According to the actual registration environment, fine features are adaptively obtained to reduce the sensitivity of the model to abnormal transformation. We use the global receptive field provided in the transformer to obtain feature descriptors based on these two levels. In addition, we use the cosine loss directly defined on the corresponding relationship to train the network and balance the samples, to achieve feature point registration based on the corresponding relationship. Extensive experiments on object-level and scene-level datasets show that the proposed method outperforms existing state-of-the-art techniques by a large margin. More critically, it has the best generalization ability in unknown scenes with different sensor modes. [ABSTRACT FROM AUTHOR]

Sum–product networks (SPNs) in deep probabilistic models have made great progress in computer vision, robotics, neuro-symbolic artificial intelligence, natural language processing, probabilistic programming languages, and other fields. Compared with probabilistic graphical models and deep probabilistic models, SPNs can balance the tractability and expressive efficiency. In addition, SPNs remain more interpretable than deep neural models. The expressiveness and complexity of SPNs depend on their own structure. Thus, how to design an effective SPN structure learning algorithm that can balance expressiveness and complexity has become a hot research topic in recent years. In this paper, we review SPN structure learning comprehensively, including the motivation of SPN structure learning, a systematic review of related theories, the proper categorization of different SPN structure learning algorithms, several evaluation approaches and some helpful online resources. Moreover, we discuss some open issues and research directions for SPN structure learning. To our knowledge, this is the first survey to focus specifically on SPN structure learning, and we hope to provide useful references for researchers in related fields. • SPNs are deep probabilistic graphical models with complete probabilistic semantics • SPNs are more tractable than PGMs • A reasonable structure of an SPN can balance expressiveness and complexity • Analyze SPN structure learning methods and forecast future research trends. [ABSTRACT FROM AUTHOR]

Deep neural networks have achieved great success in solving many machine learning and computer vision problems. In this paper, we propose a deep neural network called the Tucker network derived from the Tucker format and analyze its expressive power. The results demonstrate that the Tucker network has exponentially higher expressive power than the shallow network. In other words, a shallow network with an exponential width is required to realize the same score function as that computed by the Tucker network. Moreover, we discuss the expressive power between the hierarchical Tucker tensor network (HT network) and the proposed Tucker network. To generalize the Tucker network into a deep version, we combine the hierarchical Tucker format and Tucker format to propose a deep Tucker tensor decomposition. Its corresponding deep Tucker network is presented. Experiments are conducted on three datasets: MNIST, CIFAR-10 and CIFAR-100. The results experimentally validate the theoretical results and show that the Tucker network and deep Tucker network have better performance than the shallow network and HT network. [ABSTRACT FROM AUTHOR]

Human activity recognition (HAR) using drone-mounted cameras has attracted considerable interest from the computer vision research community in recent years. A robust and efficient HAR system has a pivotal role in fields like video surveillance, crowd behavior analysis, sports analysis, and human–computer interaction. What makes it challenging are the complex poses, understanding different viewpoints, and the environmental scenarios where the action is taking place. To address such complexities, in this paper, we propose a novel Sparse Weighted Temporal Attention (SWTA) module to utilize sparsely sampled video frames for obtaining global weighted temporal attention. The proposed SWTA is comprised of two parts. First, temporal segment network that sparsely samples a given set of frames. Second, weighted temporal attention, which incorporates a fusion of attention maps derived from optical flow, with raw RGB images. This is followed by a basenet network, which comprises a convolutional neural network (CNN) module along with fully connected layers that provide us with activity recognition. The SWTA network can be used as a plug-in module to the existing deep CNN architectures, for optimizing them to learn temporal information by eliminating the need for a separate temporal stream. It has been evaluated on three publicly available benchmark datasets, namely Okutama, MOD20, and Drone-Action. The proposed model has received an accuracy of 72.76%, 92.56%, and 78.86% on the respective datasets thereby surpassing the previous state-of-the-art performances by a margin of 25.26%, 18.56%, and 2.94%, respectively. [ABSTRACT FROM AUTHOR]

Residual Network (ResNet) achieves deeper and wider networks with high-performance gains, representing a powerful convolutional neural network architecture. In this paper, we propose architectural refinements to ResNet that address the information flow through several layers of the network, including the input stem, downsampling block, projection shortcut, and identity blocks. We will show that our collective refinements facilitate stable backpropagation by preserving the norm of the error gradient within the residual blocks, which can reduce the optimization difficulties of training very deep networks. Our proposed modifications enhance the learning dynamics, resulting in high accuracy and inference performance by enforcing norm-preservation throughout the network training. The effectiveness of our method is verified by extensive experimental results on five computer vision tasks, including image classification (ImageNet and CIFAR-100), video classification (Kinetics-400), multi-label image recognition (MS-COCO), object detection and semantic segmentation (PASCAL VOC). We also empirically show consistent improvements in generalization performance when applying our modifications over different networks to provide new insights and inspire new architectures. The source code is publicly available at: https://github.com/bharatmahaur/LeNo. [ABSTRACT FROM AUTHOR]

Inspired by the successful use of deep learning in computer vision, in this paper we introduce ForCNN , a novel deep learning method for univariate time series forecasting that mixes convolutional and dense layers in a single neural network. Instead of using conventional, numeric representations of time series data as input to the network, the proposed method considers visual representations of it in the form of images to directly produce point forecasts. Three variants of deep convolutional neural networks are examined to process the images, the first based on VGG-19, the second on ResNet-50, while the third on a self-designed architecture. The performance of the proposed approach is evaluated using time series of the M3 and M4 forecasting competitions. Our results suggest that image-based time series forecasting methods can outperform both standard and state-of-the-art forecasting models. • An image-based deep learning method for time series forecasting is proposed. • Numeric time series data are converted to images using simple line plots. • Deep convolutional neural networks are employed for producing forecasts from each image. • The proposed, image-based, method outperforms state-of-the-art forecasting methods. [ABSTRACT FROM AUTHOR]

Transformers have achieved great success in many artificial intelligence fields, such as computer vision (CV), audio processing and natural language processing (NLP). In speech emotion recognition (SER), transformer-based architectures usually compute attention in a token-by-token (frame-by-frame) manner, but this approach lacks adequate capacity to capture local emotion information and is easily affected by noise. This paper proposes a novel SER architecture, referred to as block and token self-attention (BAT), that splits a mixed spectrogram into blocks and computes self-attention by combining these blocks with tokens, which can alleviate the effect of local noise while capturing authentic sentiment expressions. Furthermore, we present a cross-block attention mechanism to facilitate information interaction among blocks while integrating a frequency compression and channel enhancement (FCCE) module to smooth the attention biases between blocks and tokens. BAT achieves 73.2% weighted accuracy (WA) and 75.2% unweighted accuracy (UA) on the Interactive Emotional Dyadic Motion Capture (IEMOCAP) dataset, surpassing the results of previously developed state-of-the-art approaches with the same dataset partitioning operation. Further experimental results reveal that our proposed method is also well suited for cross-database and cross-domain tasks, achieving 89% WA and 87.4% UA on Emo-DB and producing a top-1 recognition accuracy of 88.32% with only 15.01 Mb of parameters on the CIFAR-10 image dataset under a scenario with no data augmentation or pretraining. [ABSTRACT FROM AUTHOR]

Tensor completion has been widely used in computer vision and machine learning. Most existing tensor completion methods empirically assume the intrinsic tensor is simultaneous low-rank in all over modes. However, tensor data recorded from real-world applications may conflict with these assumptions, e.g., face images taken from different subjects often lie in a union of low-rank subspaces, which may result in a quite high rank or even full rank structure in its sample mode. To this aim, in this paper, we propose an imbalanced low-rank tensor completion method, which can flexibly estimate the low-rank incomplete tensor via decomposing it into a mixture of multiple latent tensor ring (TR) rank components. Specifically, each latent component is approximated using low-rank matrix factorization based on TR unfolding matrix. In addition, an effective proximal alternating minimization algorithm is developed and theoretically proved to maintain the global convergence property, that is, the whole sequence of iterates is convergent and converges to a critical point. Extensive experiments on both synthetic and real-world tensor data demonstrate that the proposed method achieves more favorable completion results with less computational cost when compared to the state-of-the-art tensor completion methods. [ABSTRACT FROM AUTHOR]

Text-to-image synthesis is a fundamental and challenging task in computer vision, which aims to synthesize realistic images from given descriptions. Recently, text-to-image synthesis methods have achieved great improvements in the quality of synthesized images. However, very few works have explored its application in the scenario of face synthesis, which is of great potentials in face-related applications and the public safety domain. On the other side, the faces generated by existing methods are generally of poor quality and have low consistency to the given text. To tackle this issue, in this paper, we build a novel end-to-end dual-channel generator based generative adversarial network, named DualG-GAN, to improve the quality of the generated images and the consistency to the text description. In DualG-GAN, to improve the consistency between the synthesized image and the input description, a dual-channel generator block is introduced, and a novel loss is designed to improve the similarity between the generated image and the ground-truth in three different semantic levels. Extensive experiments demonstrate that DualG-GAN achieves state-of-the-art results on SCU-Text2face dataset. To further verify the performance of DualG-GAN, we compare it with the current optimal methods on text-to-image synthesis tasks, where quantitative and qualitative results show that the proposed DualG-GAN achieves optimal performance in both Fréchet inception distance (FID) and R-precision metrics. As only a few works are focusing on text-to-face synthesis, this work can be seen as a baseline for future research. [ABSTRACT FROM AUTHOR]

There are several methods in the exploration of Convolutional Neural Networks' (CNNs') inner workings. However, in general, finding the inverse of the function performed by CNNs as a whole is an ill-posed problem. In this paper, we propose a method based on adjoint operators to reconstruct, given an arbitrary unit in the CNN (except for the first convolutional layer), its effective hypersurface in the input space. Since the reconstructed hyperplane (each point on the hypersurface) resides in the input space, we can easily visualize it. Our results show that the reconstructed hyperplane, when multiplied by the original input image, would give nearly the exact output value of that unit. We find that the CNN unit's decision process is largely conditioned on the input, and the corresponding reconstructed hypersurfaces are highly sensitive to adversarial noise, thus providing insights on why CNNs are susceptible to adversarial attack. [ABSTRACT FROM AUTHOR]

This paper proposes a new hierarchical approach to learning rate adaptation in gradient methods, called learning rate optimization (LRO). LRO formulates the learning rate adaption problem as a hierarchical optimization problem that minimizes the loss function with respect to the learning rate for current model parameters and gradients. Then, LRO optimizes the learning rate based on the alternating direction method of multipliers (ADMM). In the process of this learning rate optimization, LRO does not require any second-order information and probabilistic model, so it is highly efficient. Furthermore, LRO does not require any additional hyperparameters when compared to the vanilla gradient method with the simple exponential learning rate decay. In the experiments, we integrated LRO with vanilla SGD and Adam. Then, we compared their optimization performance with the state-of-the-art learning rate adaptation methods and also the most commonly-used adaptive gradient methods. The SGD and Adam with LRO outperformed all the competitors on the benchmark datasets in image classification tasks. [ABSTRACT FROM AUTHOR]

Although 3D human pose estimation has recently made strides, it is still difficult to precisely recreate a 3D human posture from a single image without the aid of 3D annotation for the following reasons. Firstly, the process of reconstruction inherently suffers from ambiguity, as multiple 3D poses can be projected onto the same 2D pose. Secondly, accurately measuring camera rotation without laborious camera calibration is a difficult task. While some approaches attempt to address these issues using traditional computer vision algorithms, they are not differentiable and cannot be optimized through training. This paper introduces two modules that explicitly leverage geometry to overcome these challenges, without requiring any 3D ground-truth or camera parameters. The first module, known as the relative depth estimation module, effectively mitigates depth ambiguity by narrowing down the possible depths for each joint to only two candidates. The second module, referred to as the differentiable pose alignment module, calculates camera rotation by aligning poses from different views. The use of these geometrically interpretable modules reduces the complexity of training and yields superior performance. By adopting our proposed method, we achieve state-of-the-art results on standard benchmark datasets, surpassing other self-supervised methods and even outperforming several fully-supervised approaches that heavily rely on 3D annotations. [ABSTRACT FROM AUTHOR]

Supervised learning-based image classification in computer vision relies on visual samples containing a large amount of labeled information. Considering that it is labor-intensive to collect and label images and construct datasets manually, Zero-Shot Learning (ZSL) achieves knowledge transfer from seen categories to unseen categories by mining auxiliary information, which reduces the dependence on labeled image samples and is one of the current research hotspots in computer vision. However, most ZSL methods fail to properly measure the relationships between classes, or do not consider the differences and similarities between classes at all. In this paper, we propose Adaptive Relation-Aware Network (ARAN), a novel ZSL approach that incorporates the improved triplet loss from deep metric learning into a VAE-based generative model, which helps to model inter-class and intra-class relationships for different classes in ZSL datasets and generate an arbitrary amount of high-quality visual features containing more discriminative information. Moreover, we validate the effectiveness and superior performance of our ARAN through experimental evaluations under ZSL and more practical GZSL settings on three popular datasets AWA2, CUB, and SUN. [ABSTRACT FROM AUTHOR]

Vision transformers (ViTs) have become one of the dominant frameworks for vision tasks in recent years because of their ability to efficiently capture long-range dependencies in image recognition tasks using self-attention. In fact, both CNNs and ViTs have advantages and disadvantages in vision tasks, and some studies suggest that the use of both may be an effective way to balance performance and computational cost. In this paper, we propose a new hybrid network based on CNN and transformer, using CNN to extract local features and transformer to capture long-distance dependencies. We also proposed a new feature map resolution reduction based on Discrete Cosine Transform and self-attention, named DCT-Attention Down-sample (DAD). Our DctViT-L achieves 84.8% top-1 accuracy on ImageNet 1K, far outperforming CMT, Next-ViT, SpectFormer and other state-of-the-art models, with lower computational costs. Using DctViT-B as the backbone, RetinaNet can achieve 46.8% mAP on COCO val2017, which improves mAP by 2.5% and 1.1% with less calculation cost compared with CMT-S and SpectFormer as the backbone. • We propose a DAD block for feature map compression and feature extraction using DCT. • We design new convolutional blocks and transformer blocks. • We propose a CNN and Transformer hybrid neural network family DctViT. • We experimentally demonstrate the superior performance of the DctViT network. [ABSTRACT FROM AUTHOR]

Vision Transformer (ViT) has performed remarkably in various computer vision tasks. Nonetheless, affected by the massive amount of parameters, ViT usually suffers from serious overfitting problems with a relatively limited number of training samples. In addition, ViT generally demands heavy computing resources, which limit its deployment on resource-constrained devices. As a type of model-compression method, model binarization is potentially a good choice to solve the above problems. Compared with the full-precision one, the model with the binarization method replaces complex tensor multiplication with simple bit-wise binary operations and represents full-precision model parameters and activations with only 1-bit ones, which potentially solves the problem of model size and computational complexity, respectively. In this paper, we investigate a binarized ViT model. Empirically, we observe that the existing binarization technology designed for Convolutional Neural Networks (CNN) cannot migrate well to a ViT's binarization task. We also find that the decline of the accuracy of the binary ViT model is mainly due to the information loss of the Attention module and the Value vector. Therefore, we propose a novel model binarization technique, called G roup S uperposition B inarization (GSB), to deal with these issues. Furthermore, in order to further improve the performance of the binarization model, we have investigated the gradient calculation procedure in the binarization process and derived more proper gradient calculation equations for GSB to reduce the influence of gradient mismatch. Then, the knowledge distillation technique is introduced to alleviate the performance degradation caused by model binarization. Analytically, model binarization can limit the parameter's search space during parameter updates while training a model. Therefore, the binarization process can actually play an implicit regularization role and help solve the problem of overfitting in the case of insufficient training data. Experiments on three datasets with limited numbers of training samples demonstrate that the proposed GSB model achieves state-of-the-art performance among the binary quantization schemes and exceeds its full-precision counterpart on some indicators. Code and models are available at: https://github.com/IMRL/GSB-Vision-Transformer. [ABSTRACT FROM AUTHOR]

Medical image segmentation faces challenges because of the small sample size of the dataset and the fact that images often have noise and artifacts. In recent years, diffusion models have proven very effective in image generation and have been used widely in computer vision. This paper presents a new feature map denoising module (FMD) based on the diffusion model for feature refinement, which is plug-and-play, allowing flexible integration into popular used segmentation networks for seamless end-to-end training. We evaluate the performance of the FMD module on four models, UNet, UNeXt, TransUNet, and IB-TransUNet, by conducting experiments on four datasets. The experimental data analysis shows that adding the FMD module significantly positively impacts the model performance. Furthermore, especially for small lesion areas and minor organs, adding the FMD module allows users to obtain more accurate segmentation results than the original model. [ABSTRACT FROM AUTHOR]

Convolutional Neural Networks (CNNs) have revolutionized image classification through their innovative design and training methodologies in computer vision. Differential convolutional neural network with simultaneous multidimensional filter realization improved the performance of the convolutional neural network with calculation cost drawback. This paper introduces logarithmic learning integration into the differential Convolutional neural network to overcome the drawback by supplying faster error minimization and convergence. This task is done by incorporating LogRelu activation, a Logarithmic Cost Function, and unique logarithmic learning method. The effectiveness of the proposed approaches are evaluated by using various datasets and SGD/Adam optimizers. The first step is the adaptation of LogRelu activation function to convolutional and differential convolutional neural networks. The experiment results show that LogRelu integration to convolutional neural network and differential convolutional neural network yields performance improvements ranging from 1.61% to 5.44%. The same integration on ResNet-18, ResNet-34, and ResNet-50 enhances top-1 accuracy in the range of 3.07% and 9.96%. In addition to LogRelu activation function, a Logarithmic Cost Function with logarithmic learning method is also proposed and adapted to differential convolutional neural network. These improvements lead to a new differential convolutional neural network named as Logarithmic Differential Convolutional Neural Network (LDiffCNN), It consistently outperforms standard CNN by increasing the accuracy up to 3.02%. Notably, Logarithmic Differential Convolutional Neural Network demonstrates reduced training iterations up to 38% with faster convergence. The experimental results proved the efficiency of the proposed approach. [ABSTRACT FROM AUTHOR]

Recent advances in unsupervised domain adaptation have shown that mitigating the domain divergence by extracting the domain-invariant features could significantly improve the generalization of a model with respect to a new data domain. However, current methodologies often neglect to retain domain private information, which is the unique information inherent to the unlabeled new domain, compromising generalization. This paper presents a novel method that utilizes mutual information to protect this domain-specific information, ensuring that the latent features of the unlabeled data not only remain domain-invariant but also reflect the unique statistics of the unlabeled domain. We show that simultaneous maximization of mutual information and reduction of domain divergence can effectively preserve domain-private information. We further illustrate that a neural estimator can aptly estimate the mutual information between the unlabeled input space and its latent feature space. Both theoretical analysis and empirical results validate the significance of preserving such unique information of the unlabeled domain for cross-domain generalization. Comparative evaluations reveal our method's superiority over existing state-of-the-art techniques across multiple benchmark datasets. [ABSTRACT FROM AUTHOR]

Although great success has been achieved in various computer vision tasks, deep neural networks (DNNs) suffer dramatic performance degradation when evaluated on out-of-distribution data. Domain generalization (DG) is proposed to handle this problem by learning domain-agnostic information from multiple source domains to generalize well on unseen target domains. Several methods resort to Fourier transform due to its simplicity and efficiency. They argue that amplitude spectra imply domain-specific information, which should be suppressed, while phase counterparts imply domain-agnostic information, which should be preserved. However, these methods only suppress the domain-specific information in source domains and neglect the relationship with target domains, leading to the persistence of the domain gap. Besides, these methods preserve domain-agnostic information by keeping phase components unchanged, causing them to be underutilized. In this paper, we propose Dual Branch Augmentation Module (DBAM) by leveraging Fourier transform and taking advantage of both amplitude and phase spectra. For the amplitude branch, we propose Inner-domain Amplitude Distribution Rectification (IADR) and Cross-domain Amplitude Dirichlet Mixup (CADM) to stabilize the training process and explore more feature space. In addition, we propose Test-time Amplitude Prototype Calibration (TAPC) to construct the connection between source and target domains during evaluation to further mitigate the domain gap. For the phase branch, we propose Random Symmetric Phase Perturbation (RSPP) to enhance the robustness for recognizing domain-agnostic information. With the combined contributions of the two branches, DBAM significantly surpasses other state-of-the-art (SOTA) methods. Extensive experiments on four benchmarks and further analysis demonstrate the effectiveness of DBAM. [ABSTRACT FROM AUTHOR]

Many large and complex deep neural networks have been shown to provide higher performance on various computer vision tasks. However, very little is known about the relationship between the complexity of the input data along with the type of noise and the depth needed for correct classification. Existing studies do not address the issue of common corruptions adequately, especially in understanding what impact these corruptions leave on the individual part of a deep neural network. Therefore, we can safely assume that the classification (or misclassification) might be happening at a particular layer(s) of a network that accumulates to draw a final correct or incorrect prediction. In this paper, we introduce a novel concept of corruption depth , which identifies the location of the network layer/depth until the misclassification persists. We assert that the identification of such layers will help in better designing the network by pruning certain layers in comparison to the purification of the entire network which is computationally heavy. Through our extensive experiments, we present a coherent study to understand the processing of examples through the network. Our approach also illustrates different philosophies of example memorization and a one-dimensional view of sample or query difficulty. We believe that the understanding of the corruption depth can open a new dimension of model explainability and model compression , where in place of just visualizing the attention map, the classification progress can be seen throughout the network. • A novel metric corruption depth is proposed to explain functioning of neural network. • The proposed research helps in deciphering the flow of classification decisions. • The proposed research overcomes the limitations of understanding example difficulty. • Corruption depth notation helps in pruning the networks while improving the accuracy. [ABSTRACT FROM AUTHOR]

Semantic segmentation is one of the essential prerequisites for computer vision tasks, but edge-precise segmentation stays challenging due to the potential lack of a proper model indicating the low-level relation between pixels. We have presented Refined UNet v2, a concatenation of a network backbone and a subsequent embedded conditional random field (CRF) layer, which coarsely performs pixel-wise classification and refines edges of segmentation regions in a one-stage way. However, the CRF layer of v2 employs a gray-scale global observation (image) to construct contrast-sensitive bilateral features, which is not able to achieve the desired performance on ambiguous edges. In addition, the naïve depth-wise Gaussian filter cannot always compute efficiently, especially for a longer-range message-passing step. To address the aforementioned issues, we upgrade the bilateral message-passing kernel and the efficient implementation of Gaussian filtering in the CRF layer in this paper, referred to as Refined UNet v3, which is able to effectively capture ambiguous edges and accelerate the message-passing procedure. Specifically, the inherited UNet is employed to coarsely locate cloud and shadow regions and the embedded CRF layer refines the edges of the forthcoming segmentation proposals. The multi-channel guided Gaussian filter is applied to the bilateral message-passing step, which improves detecting ambiguous edges that are hard for the gray-scale counterpart to identify, and fast Fourier transform-based (FFT-based) Gaussian filtering facilitates an efficient and potentially range-agnostic implementation. Furthermore, Refined UNet v3 is able to be extended to segmentation on multi-spectral datasets, and the corresponding refinement examination confirms the development of shadow retrieval. Experiments and corresponding results demonstrate that the proposed update can outperform its counterpart in terms of the detection of vague edges, shadow retrieval, and isolated redundant regions, and it is practically efficient in our TensorFlow implementation. The demo source code is available at https://github.com/92xianshen/refined-unet-v3. [ABSTRACT FROM AUTHOR]

Synthesizing photo-realistic images based on text descriptions is a challenging task in the field of computer vision. Although generative adversarial networks have made significant breakthroughs in this task, they still face huge challenges in generating high-quality visually realistic images consistent with the semantics of text. Generally, existing text-to-image methods accomplish this task with two steps, that is, first generating an initial image with a rough outline and color, and then gradually yielding the image within high-resolution from the initial image. However, one drawback of these methods is that, if the quality of the initial image generation is not high, it is hard to generate a satisfactory high-resolution image. In this paper, we propose SAM-GAN, Self-Attention supporting Multi-stage Generative Adversarial Networks, for text-to-image synthesis. With the self-attention mechanism, the model can establish the multi-level dependence of the image and fuse the sentence- and word-level visual-semantic vectors, to improve the quality of the generated image. Furthermore, a multi-stage perceptual loss is introduced to enhance the semantic similarity between the synthesized image and the real image, thus enhancing the visual-semantic consistency between text and images. For the diversity of the generated images, a mode seeking regularization term is integrated into the model. The results of extensive experiments and ablation studies, which were conducted in the Caltech-UCSD Birds and Microsoft Common Objects in Context datasets, show that our model is superior to competitive models in text-to-image synthesis. • SAM-GAN, a novel GAN-based model m, is proposed to improve the quality of generated images. • A multi-stage perceptual loss module is designed to generate high quality images consistent with text description. • A regularization is employed to improve the diversity of image backgrounds. • Extensive experiments and ablation studies show our model performs well for text-to-image task. [ABSTRACT FROM AUTHOR]

Deep Neural Networks (DNNs) have become popular for various applications in the domain of image and computer vision due to their well-established performance attributes. DNN algorithms involve powerful multilevel feature extractions resulting in an extensive range of parameters and memory footprints. However, memory bandwidth requirements, memory footprint and the associated power consumption of models are issues to be addressed to deploy DNN models on embedded platforms for real time vision-based applications. We present an optimized DNN model for memory and accuracy for vision-based applications on embedded platforms. In this paper we propose Quantization Friendly MobileNet (QF-MobileNet) architecture. The architecture is optimized for inference accuracy and reduced resource utilization. The optimization is obtained by addressing the redundancy and quantization loss of the existing baseline MobileNet architectures. We verify and validate the performance of the QF-MobileNet architecture for image classification task on the ImageNet dataset. The proposed model is tested for inference accuracy and resource utilization and compared to the baseline MobileNet architecture. The inference accuracy of the proposed QF-MobileNetV2 float model attained 73.36% and the quantized model has 69.51%. The MobileNetV3 float model attained an inference accuracy of 68.75% and the quantized model has 67.5% respectively. The proposed model saves 33% of time complexity for QF-MobileNetV2 and QF-MobileNetV3 models against the baseline models. The QF-MobileNet also showed optimized resource utilization with 32% fewer tunable parameters, 30% fewer MAC's operations per image and reduced inference quantization loss by approximately 5% compared to the baseline models. The model is ported onto the android application using TensorFlow API. The android application performs inference on the native devices viz. smartphones, tablets and handheld devices. Future work is focused on introducing channel-wise and layer-wise quantization schemes to the proposed model. We intend to explore quantization aware training of DNN algorithms to achieve optimized resource utilization and inference accuracy. [ABSTRACT FROM AUTHOR]

In this paper, we propose a new face de-identification method based on generative adversarial network (GAN) to protect visual facial privacy, which is an end-to-end method (herein, FPGAN). First, we propose FPGAN and mathematically prove its convergence. Then, a generator with an improved U-Net is used to enhance the quality of the generated image, and two discriminators with a seven-layer network architecture are designed to strengthen the feature extraction ability of FPGAN. Subsequently, we propose the pixel loss, content loss, adversarial loss functions and optimization strategy to guarantee the performance of FPGAN. In our experiments, we applied FPGAN to face de-identification in social robots and analyzed the related conditions that could affect the model. Moreover, we proposed a new face de-identification evaluation protocol to check the performance of the model. This protocol can be used for the evaluation of face de-identification and privacy protection. Finally, we tested our model and four other methods on the CelebA, MORPH, RaFD, and FBDe datasets. The results of the experiments show that FPGAN outperforms the baseline methods. • We proposed an end-to-end method for face de-identification, with one generator and dual discriminators. • We designed the pixel loss and content loss functions to retain partial links between the de-identified and the original images. • We improved the U-Net and used it as a generator to generate a sufficiently realistic face image. • We proposed new discriminators to improve the discrimination accuracy. • We applied the FPGAN to face de-identification of social robots and proposed the privacy protection system. [ABSTRACT FROM AUTHOR]

Image synthesis is currently one of the most addressed image processing topic in computer vision and deep learning fields of study. Researchers have tackled this problem focusing their efforts on its several challenging problems, e.g. image quality and size, domain and pose changing, architecture of the networks, and so on. Above all, producing images belonging to different domains by using a single architecture is a very relevant goal for image generation. In fact, a single multi-domain network would allow greater flexibility and robustness in the image synthesis task than other approaches. This paper proposes a novel architecture and a training algorithm, which are able to produce multi-domain outputs using a single network. A small portion of a dataset is intentionally used, and there are no hard-coded labels (or classes). This is achieved by combining a conditional Generative Adversarial Network (cGAN) for image generation and a Meta-Learning algorithm for domain switch, and we called our approach MetalGAN. The approach has proved to be appropriate for solving the multi-domain label-less problem and it is validated on facial attribute transfer, using CelebA dataset. [ABSTRACT FROM AUTHOR]

In the last few years, human age estimation from face images attracted the attention of many researchers in computer vision and machine learning fields. This is due to its numerous applications. In this paper, we propose a new architecture for age estimation based on facial images. It is mainly based on a cascade of classification trees ensembles, which are known recently as a Deep Random Forest. Our architecture is composed of two types of DRF. The first type extends and enhances the feature representation of a given facial descriptor. The second type operates on the fused form of all enhanced representations in order to provide a prediction for the age while taking into account the fuzziness property of the human age. While the proposed methodology is able to work with all kinds of image features, the face descriptors adopted in this work used off-the-shelf deep features allowing to retain both the rich deep features and the powerful enhancement and decision provided by the proposed architecture. Experiments conducted on six public databases prove the superiority of the proposed architecture over other state-of-the-art methods. [ABSTRACT FROM AUTHOR]

The machine vision based methods for micro-cracks detection of solar cells surface have become one of the main research directions with its efficiency and convenience. The existed methods are roughly classified into two categories: current viewing information based methods, prior knowledge based methods, however, the former usually adopt hand-designed features with poor generality and lacks the guidance of prior knowledge, the latter are usually implemented through the machine learning, and the generalization ability is also limited since the large-scale annotation dataset is scarce. To resolve above problems, a novel micro-cracks detection method via combining short-term and long-term deep features is proposed in this paper. The short-term deep features which represent the current viewing information are learned from the input image itself through stacked denoising auto encoder (SDAE), the long-term deep features which represent the prior knowledge are learned from a large number of natural scene images that people often see through convolutional neural networks (CNNs). The subjective and objective evaluations demonstrate that: 1) the performance of combining the short-term and long-term deep features is better than any of them alone, 2) the performance of proposed method is superior to the shallow learning based methods, 3) the proposed method can effectively detect various kinds of micro-cracks. [ABSTRACT FROM AUTHOR]

Humans are continuously exposed to a stream of visual data with a natural temporal structure. However, most successful computer vision algorithms work at image level, completely discarding the precious information carried by motion. In this paper, we claim that processing visual streams naturally leads to formulate the motion invariance principle, which enables the construction of a new theory of learning that originates from variational principles, just like in physics. Such principled approach is well suited for a discussion on a number of interesting questions that arise in vision, and it offers a well-posed computational scheme for the discovery of convolutional filters over the retina. Differently from traditional convolutional networks, which need massive supervision, the proposed theory offers a truly new scenario for the unsupervised processing of video signals, where features are extracted in a multi-layer architecture with motion invariance. While the theory enables the implementation of novel computer vision systems, it also sheds light on the role of information-based principles to drive possible biological solutions. [ABSTRACT FROM AUTHOR]

Salient object detection can be applied as a critical preprocessing step in many computer vision tasks. Recent studies of salient object detection mainly employed convolutional neural networks (CNNs) for mining high-level semantic properties. However, the existing methods can still be improved to find precise semantic information in different scenarios. In particular, in the two main methods employed for salient object detection, the patchwise detection models might ignore spatial structures among regions and the fully convolution-based models mainly consider semantic features in a global manner. In this paper, we proposed a salient object detection framework by embedding topological features into a deep neural network for extracting semantics. We segment the input image and compute weight for each region with low-level features. The weighted segmentation result is called a topological map and it provides an additional channel for the CNN to emphasize the structural integrity and locality during the extraction of semantic features. We also utilize the topological map for saliency refinement based on a conditional random field at the end of our model. Experimental results on six benchmark datasets demonstrated that our proposed framework achieves competitive performance compared to other state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Over the past decade, Deep Convolutional Neural Networks (DCNNs) have shown remarkable performance in most computer vision tasks. These tasks traditionally use a fixed dataset, and the model, once trained, is deployed as is. Adding new information to such a model presents a challenge due to complex training issues, such as "catastrophic forgetting", and sensitivity to hyper-parameter tuning. However, in this modern world, data is constantly evolving, and our deep learning models are required to adapt to these changes. In this paper, we propose an adaptive hierarchical network structure composed of DCNNs that can grow and learn as new data becomes available. The network grows in a tree-like fashion to accommodate new classes of data, while preserving the ability to distinguish the previously trained classes. The network organizes the incrementally available data into feature-driven super-classes and improves upon existing hierarchical CNN models by adding the capability of self-growth. The proposed hierarchical model, when compared against fine-tuning a deep network, achieves significant reduction of training effort, while maintaining competitive accuracy on CIFAR-10 and CIFAR-100. [ABSTRACT FROM AUTHOR]

Blind image deconvolution is one of the main low-level vision problems with wide applications. Many previous works manually design regularization to simultaneously estimate the latent sharp image and the blur kernel under maximum a posterior framework. However, it has been demonstrated that such joint estimation strategies may lead to the undesired trivial solution. In this paper, we present a novel perspective, using a stable feedback control system, to simulate the latent sharp image propagation. The controller of our system consists of regularization and guidance, which decide the sparsity and sharp features of latent image, respectively. Furthermore, the formational model of blind image is introduced into the feedback process to avoid the image restoration deviating from the stable point. The stability analysis of the system indicates the latent image propagation in blind deconvolution task can be efficiently estimated and controlled by cues and priors. Thus the kernel estimation used for image restoration becomes more precision. Experimental results show that our system is effective on image propagation, and can perform favorably against the state-of-the-art blind image deconvolution methods on different benchmark image sets and special blurred images. [ABSTRACT FROM AUTHOR]

Humans are highly efficient at visual search tasks by focusing selective attention on a small but relevant region of a visual scene. Recent results from biological vision suggest that surfaces of distinct physical objects form the basic units of this attentional process. The aim of this paper is to demonstrate how such surface-based attention mechanisms can speed up a computer vision system for visual search. The system uses fast perceptual grouping of depth cues to represent the visual world at the level of surfaces. This representation is stored in short-term memory and updated over time. A top-down guided attention mechanism sequentially selects one of the surfaces for detailed inspection by a recognition module. We show that the proposed attention framework requires little computational overhead (about 11 ms), but enables the system to operate in real-time and leads to a substantial increase in search efficiency. [ABSTRACT FROM AUTHOR]