Search

Learning from large-scale contrastive language-image pre-training like CLIP has shown remarkable success in a wide range of downstream tasks recently, but it is still under-explored on the challenging few-shot action recognition (FSAR) task. In this work, we aim to transfer the powerful multimodal knowledge of CLIP to alleviate the inaccurate prototype estimation issue due to data scarcity, which is a critical problem in low-shot regimes. To this end, we present a CLIP-guided prototype modulating framework called CLIP-FSAR, which consists of two key components: a video-text contrastive objective and a prototype modulation. Specifically, the former bridges the task discrepancy between CLIP and the few-shot video task by contrasting videos and corresponding class text descriptions. The latter leverages the transferable textual concepts from CLIP to adaptively refine visual prototypes with a temporal Transformer. By this means, CLIP-FSAR can take full advantage of the rich semantic priors in CLIP to obtain reliable prototypes and achieve accurate few-shot classification. Extensive experiments on five commonly used benchmarks demonstrate the effectiveness of our proposed method, and CLIP-FSAR significantly outperforms existing state-of-the-art methods under various settings. The source code and models are publicly available at https://github.com/alibaba-mmai-research/CLIP-FSAR. [ABSTRACT FROM AUTHOR]

Image stylization has attracted considerable attention from various fields. Although impressive results have been achieved, existing methods pay less attention to the preservation of outline and putting constraint on it when training, which makes generated images suffering from different degrees of distortion. To address this issue, we propose a dual attention mechanism based outline loss to enhance the restriction of outline consistency by incorporating an outline detection module and a dual attention module. Specifically, an outline detection module is used to detect outlines of the source image and the stylized image, which are further compared and enforced to be consistent with each other by a carefully-elaborated outline loss. Additionally, the dual attention module first guides the model to focus on regions of the source image whose style has the biggest difference from the target image during stylization based on the style attention feature map obtained by the auxiliary classifier. Then, an outline attention map is predicted to highlight regions where the outlines are prone to distort during stylization, which further facilitates the outline loss to execute stronger constraint on these regions. Experimental results show the superiority of our method compared to the existing state-of-the-art methods [ABSTRACT FROM AUTHOR]

Recently, crowd counting via estimating a density map has been widely studied. However, it still has a variety of issues to overcome, such as large-scale variation of population, complex background noise, perspective distortion, etc. The large-scale variation of heads will restrict the performance of crowd counting approaches, and the complex background noise will result in the background, such as leaf and mesh, being incorrectly recognized as heads. To maintain large-scale variation and generate a high-quality estimated density map, we propose a novel multi-scale fusion scale-aware attention network called multi-scale and gated spatial attention network (MGSNet). In MGSNet, the first 10 layers of VGG16 with Batch Normalization (BN) are utilized as backbone. Then, two branches, i.e., a large-scale branch and a scale–aware attention branch, are followed. The large-scale branch is used to overcome the large-scale variation of heads in crowd images, in which a Scale Information Aggregation Block (SIAB) is employed to extract multi-scale features by utilizing dilated convolution with different receptive fields. The scale-aware attention branch is used to address complex background noise in crowd scenes, in which a Gated Spatial Attention Block (GSAB) inspired by the Long Short-term Memory Networks (LSTM) is employed to fuse the previous information with different scales and retain the appropriate scale information of crowds. We demonstrate our proposed method on the ShanghaiTech (Part AB), UCF-CC-50 and UCF-QNRF datasets. The experimental results show its effectiveness over the state-of-the-art. [ABSTRACT FROM AUTHOR]

Low-level details and high-level semantics are both essential to the semantic segmentation task. However, to speed up the model inference, current approaches almost always sacrifice the low-level details, leading to a considerable decrease in accuracy. We propose to treat these spatial details and categorical semantics separately to achieve high accuracy and high efficiency for real-time semantic segmentation. For this purpose, we propose an efficient and effective architecture with a good trade-off between speed and accuracy, termed Bilateral Segmentation Network (BiSeNet V2). This architecture involves the following: (i) A detail branch, with wide channels and shallow layers to capture low-level details and generate high-resolution feature representation; (ii) A semantics branch, with narrow channels and deep layers to obtain high-level semantic context. The detail branch has wide channel dimensions and shallow layers, while the semantics branch has narrow channel dimensions and deep layers. Due to the reduction in the channel capacity and the use of a fast-downsampling strategy, the semantics branch is lightweight and can be implemented by any efficient model. We design a guided aggregation layer to enhance mutual connections and fuse both types of feature representation. Moreover, a booster training strategy is designed to improve the segmentation performance without any extra inference cost. Extensive quantitative and qualitative evaluations demonstrate that the proposed architecture shows favorable performance compared to several state-of-the-art real-time semantic segmentation approaches. Specifically, for a 2048 × 1024 input, we achieve 72.6% Mean IoU on the Cityscapes test set with a speed of 156 FPS on one NVIDIA GeForce GTX 1080 Ti card, which is significantly faster than existing methods, yet we achieve better segmentation accuracy. The code and trained models are available online at https://git.io/BiSeNet. [ABSTRACT FROM AUTHOR]

We present an effective blind image deblurring method based on a data-driven discriminative prior. Our work is motivated by the fact that a good image prior should favor sharp images over blurred ones. In this work, we formulate the image prior as a binary classifier using a deep convolutional neural network. The learned prior is able to distinguish whether an input image is sharp or not. Embedded into the maximum a posterior framework, it helps blind deblurring in various scenarios, including natural, face, text, and low-illumination images, as well as non-uniform deblurring. However, it is difficult to optimize the deblurring method with the learned image prior as it involves a non-linear neural network. In this work, we develop an efficient numerical approach based on the half-quadratic splitting method and gradient descent algorithm to optimize the proposed model. Furthermore, we extend the proposed model to handle image dehazing. Both qualitative and quantitative experimental results show that our method performs favorably against the state-of-the-art algorithms as well as domain-specific image deblurring approaches. [ABSTRACT FROM AUTHOR]

Recently, safe semi-supervised learning has attracted more and more attention in the machine learning field. Many methods are introduced to safely exploit unlabeled data by designing different safe mechanisms. However, they assume that the risk or safety degrees are equal for all unlabeled data. In this paper, we propose an adaptive safe semi-supervised learning framework where the safety degrees of different unlabeled data are different and adaptively computed. In this framework, a safety degree-based tradeoff term between supervised learning (SL) and semi-supervised learning (SSL) is incorporated into the objective function of SSL. Then the optimal problem is solved by using an alternating iterative strategy. In particular, we utilize Regularized Least Squares (RLS) and Laplacian RLS (LapRLS) for SL and SSL, respectively. Our experimental results on several datasets demonstrate that the performance of our algorithm is never significantly inferior to that of RLS and LapRLS and show the effectiveness of our proposed safety mechanism. [ABSTRACT FROM AUTHOR]

Low-rank regularization approximated by a nuclear norm has been proven its ability in image denoising. However, the nuclear norm is just a suboptimization of the rank norm, resulting in a big error when reducing noise. In this paper, a novel smooth and convex surrogate function, which is closer to the rank norm, is firstly proposed as a replacement of the prior nuclear norm. Then, the proposed surrogate function is approximated by its first-order Taylor expansion. Finally, a novel model called iterative weighted nuclear norm minimization scheme, solved by the single and effective alternating directions method of multipliers with a weighted singular-value thresholding operator, is formed for image denoising. Both quantitative and qualitative results obtained by applying advanced denoising methods to synthetic images will verify the effectiveness of the proposed method. Extensive application of these state-of-the-art methods to denoising of clinical X-ray cardiovascular angiograms further validates that our proposed approach performs better on reducing noise and preserving structures (especially capillaries), demonstrating that the method can yield clear X-ray angiograms with integral cardiovascular trees which are beneficial for clinicians to diagnose and analyze cardiovascular diseases. [ABSTRACT FROM AUTHOR]

The classification of hyperspectral image with a paucity of labeled samples is a challenging task. In this paper, we present a discriminant sparse representation (DSR) graph for semi-supervised learning (SSL) to address this problem. For graph-based methods, how to construct a graph among the pixels is the key to a successful classification. Our graph construction method contains two steps. Sparse representation (SR) method is first employed to estimate the probability matrix of the pairwise pixels belonging to the same class, and then this probability matrix is integrated into the SR graph, which can be obtained by solving an ℓ optimization problem, to form a DSR graph. Experiments on Hyperion and AVIRIS hyperspectral data show that our proposed method outperforms state of the art. [ABSTRACT FROM AUTHOR]

Covariance tracking has achieved impressive successes in recent years due to its competent region covariance-based feature descriptor. Although adopt fast integral image computation, covariance tracking's brute-force search strategy is still inefficient and it possibly leads to inconsecutive tracking trajectory and distraction. In this work, a generalized, adaptive covariance tracking approach with novel integral region computation and occlusion detection is proposed. The integral region is much faster than integral image and adaptive to the tracking target and tracking condition. The adaptive search window can be adjusted dynamically by simple occlusion detection. The integral image and the global covariance tracking can be seen just a special case of integral region and the proposed approach, respectively. The proposed approach unifies the local and global search strategies in an elegant way and smoothly switches between them according to the tracking conditions (i.e. occlusion distraction or sudden motion) which are judged by occlusion detector. It gets much better efficiency, robustness of distraction, and stable trajectory by local search in normal steady state, while obtains more abilities for occlusion and re-identification by enlarged search window (until to global search) in abnormal situation at the same time. Our approach shows excellent target representation ability, faster speed, and more robustness, which has been verified on some video sequences. [ABSTRACT FROM AUTHOR]

Recently, the local binary patterns (LBP) have been widely used in the texture classification. The LBP methods obtain the binary pattern by comparing the gray scales of pixels on a small circular region with the gray scale of their central pixel. The conventional LBP methods only describe microstructures of texture images, such as edges, corners, spots and so on, although many of them show good performances on the texture classification. This situation still could not be changed, even though the multi-resolution analysis technique is adopted by LBP methods. Moreover, the circular sampling region limits the ability of the conventional LBP methods in describing anisotropic features. In this paper, we change the shape of sampling region and get an extended LBP operator. And a multi-structure local binary pattern (Ms-LBP) operator is achieved by executing the extended LBP operator on different layers of an image pyramid. Thus, the proposed method is simple yet efficient to describe four types of structures: isotropic microstructure, isotropic macrostructure, anisotropic microstructure and anisotropic macrostructure. We demonstrate the performance of our method on two public texture databases: the Outex and the CUReT. The experimental results show the advantages of the proposed method. [ABSTRACT FROM AUTHOR]

Most variants of fuzzy c-means (FCM) clustering algorithms involving prior knowledge are generally based on the modification of the objective function or the clustering process. This paper proposes a new weighted semi-supervised FCM algorithm (SSFCM-HPR) that transforms the prior knowledge in the labeled samples into constraint conditions in terms of fuzzy membership degrees, assigns different weights according to the representativeness of the samples, and then uses the HPR multiplier to solve the clustering problem. The 'representativeness' of the labeled samples is decided by their distances to the cluster centers they belong to. In this paper, we take the ratio of the largest to the second largest fuzzy membership degree from a labeled sample as its weight. This algorithm not only retains the fuzzy partition of the labeled samples, which guarantees the effective guidance on the clustering process, but also can detect whether a sample is an outlier or not. Moreover, when part of the supervised information of the labeled samples is wrong, this algorithm can reduce the influence of the incorrectly labeled samples on the final clustering results. The experimental evaluation on synthetic and real data sets demonstrates the efficiency and effectiveness of our approach. [ABSTRACT FROM AUTHOR]

The local binary pattern (LBP) approach has been widely used in texture description. In this paper, we build a new framework to extract the binary patterns and propose a robust texture descriptor: multi-ring local binary pattern (MrLBP). The MrLBP algorithm creates patterns from several ringed areas and mainly contains two parts. One is the extra-ring local binary pattern operator that gets patterns from the mean values of different ringed areas. The other is the intra-ring local binary pattern operator that obtains patterns by counting the majority of binary values in every single ringed area. Moreover, the binary formation of each part of the MrLBP is obtained from two different aspects. The MrLBP method not only considers the binary relationship among pixels in a local region, but also focuses on the relationship between pixels in a local region and the whole image. This is a little different from the conventional LBP methods that only get the binary formation from the local gray scales differences. The experimental results on two public databases have validated the effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]

A biologically inspired object-based visual attention model is proposed in this paper. This model includes a training phase and an attention phase. In the training phase, all training targets are fused into a target class and all training backgrounds are fused into a background class. Weight vector is computed as the ratio of the mean target class saliency and the mean background class saliency for each feature. In the attention phase, for an attended scene, all feature maps are combined into a top-down salience map with the weight vector by a hierarchy method. Then, top-down and bottom-up salience map are fused into a global salience map which guides the visual attention. At last, the size of each salient region is obtained by maximizing entropy. The merit of our model is that it can attend a class target object which can appear in the corresponding background class. Experimental results indicate that: when the attended target object doesn’t always appear in the background corresponding to that in the training images, our proposed model is excellent to Navalpakkam’s model and the top-down approach of VOCUS. [ABSTRACT FROM AUTHOR]