Search

Stego-images are often contaminated by interchannel noise or active noise attack when communicating on the Web. And it is challenging to restore embedded image from corrupted stego-image. This paper studies a kNN-bit approximation algorithm to remove noises in embedded image. The proposed algorithm distinguishes reliable bits from extracted bits, and estimates pixel values by keeping reliable bits unchanged and correcting unreliable bits. Specifically, the 8th (highest) unreliable bit of a pixel can be approximated with its nearest neighbor pixels. And then, if an unreliable bit locates at any one of the $5^{th}\sim 7^{th}$ 5 t h ∼ 7 t h bits of a pixel, it is adjusted with two nearest neighbors of the pixel, where the pixel is in-between these two nearest neighbors. Finally, for other unreliable bits, each one is approximated by the maximum and minimum possible values of nearest neighbors of its pixel. We conduct experiments for illustrating the efficiency, and demonstrate that the proposed algorithm can recover the embedded images with good visual quality from corrupted stego-images. [ABSTRACT FROM AUTHOR]

Recently, remote sensing image captioning (RSIC) has drawn an increasing attention. In this field, the encoder–decoder-based methods have become the mainstream due to their excellent performance. In the encoder–decoder framework, the convolutional neural network (CNN) is used to encode a remote sensing image into a semantic feature vector, and a sequence model such as long short-term memory (LSTM) is subsequently adopted to generate a content-related caption based on the feature vector. During the traditional training stage, the probability of the target word at each time step is forcibly optimized to 1 by the cross entropy (CE) loss. However, because of the variability and ambiguity of possible image captions, the target word could be replaced by other words like its synonyms, and therefore, such an optimization strategy would result in the overfitting of the network. In this article, we explore the overfitting phenomenon in the RSIC caused by CE loss and correspondingly propose a new truncation cross entropy (TCE) loss, aiming to alleviate the overfitting problem. In order to verify the effectiveness of the proposed approach, extensive comparison experiments are performed on three public RSIC data sets, including UCM-captions, Sydney-captions, and RSICD. The state-of-the-art result of Sydney-captions and RSICD and the competitive results of UCM-captions achieved by TCE loss demonstrate that the proposed method is beneficial to RSIC. [ABSTRACT FROM AUTHOR]

Convolutional Neural Network (CNN) has shown their advantages in salient object detection. CNN can generate great saliency maps because it can obtain high-level semantic information. And the semantic information is usually achieved by stacking multiple convolutional layers and pooling layers. However, multiple pooling operations will reduce the size of the feature map and easily blur the boundary of the salient object. Therefore, such operations are not beneficial to generate great saliency results. To alleviate this issue, we propose a novel edge information-guided hierarchical feature fusion network (HFFNet). Our network fuses features hierarchically and retains accurate semantic information and clear edge information effectively. Specifically, we extract image features from different levels of VGG. Then, we fuse the features hierarchically to generate high-level semantic information and low-level edge information. In order to retain better information at different levels, we adopt a one-to-one hierarchical supervision strategy to supervise the generation of low-level information and high-level information respectively. Finally, we use low-level edge information to guide the saliency map generation, and the edge guidance fusion is able to identify saliency regions effectively. The proposed HFFNet has been extensively evaluated on five traditional benchmark datasets. The experimental results demonstrate that the proposed model is fairly effective in salient object detection compared with 10 state-of-the-art models under different evaluation indicators, and it is superior to most of the comparison models. [ABSTRACT FROM AUTHOR]

As a fundamental and challenging task in bridging language and vision domains, Image-Text Retrieval (ITR) aims at searching for the target instances that are semantically relevant to the given query from the other modality, and its key challenge is to measure the semantic similarity across different modalities. Although significant progress has been achieved, existing approaches typically suffer from two major limitations: (1) It hurts the accuracy of the representation by directly exploiting the bottom-up attention based region-level features where each region is equally treated. (2) It limits the scale of negative sample pairs by employing the mini-batch based end-to-end training mechanism. To address these limitations, we propose a Unified Semantic Enhancement Momentum Contrastive Learning (USER) method for ITR. Specifically, we delicately design two simple but effective Global representation based Semantic Enhancement (GSE) modules. One learns the global representation via the self-attention algorithm, noted as Self-Guided Enhancement (SGE) module. The other module benefits from the pre-trained CLIP module, which provides a novel scheme to exploit and transfer the knowledge from an off-the-shelf model, noted as CLIP-Guided Enhancement (CGE) module. Moreover, we incorporate the training mechanism of MoCo into ITR, in which two dynamic queues are employed to enrich and enlarge the scale of negative sample pairs. Meanwhile, a Unified Training Objective (UTO) is developed to learn from mini-batch based and dynamic queue based samples. Extensive experiments on the benchmark MSCOCO and Flickr30K datasets demonstrate the superiority of both retrieval accuracy and inference efficiency. For instance, compared with the existing best method NAAF, the metric R@1 of our USER on the MSCOCO 5K Testing set is improved by 5% and 2.4% on caption retrieval and image retrieval without any external knowledge or pre-trained model while enjoying over 60 times faster inference speed. Our source code will be released at https://github.com/zhangy0822/USER. [ABSTRACT FROM AUTHOR]

Existence of haze significantly degrades visual quality and hence negatively affects the performance of visual surveillance, video analysis, and human–machine interaction. To remove haze from a visual signal, in this paper, we propose a generative adversarial network for visual haze removal called HRGAN. HRGAN consists of a generator network and a discriminator network. A unified network jointly estimating transmission maps, atmospheric light, and haze-free images (called UNTA) is proposed as the generator network of HRGAN. Instead of being optimized by minimizing the pixel-wise loss, HRGAN is optimized by minimizing a novel loss function consisting of pixel-wise loss, perceptual loss, and adversarial loss produced by a discriminator network. Classical model-based image dehazing algorithms consist of three separate stages: 1) estimating transmission map; 2) estimating atmospheric light; and 3) restoring haze-free image by using an atmospheric scattering model to process the transmission map and atmospheric light. Such a separate scheme is not guaranteed to achieve optimal results. On the contrary, UNTA performs transmission map estimation and atmospheric light estimation simultaneously to obtain joint optimal solutions. The experimental results on both synthetic and real-world image databases demonstrate that HRGAN outperforms the state-of-the-art algorithms in terms of both effectiveness and efficiency. [ABSTRACT FROM AUTHOR]

Visual tracking is a very critical issue in computer vision and video processing. For Discriminative Correlation Filter (DCF)-based tracking methods, it is very essential and meaningful to adaptively incorporate reliable target and surrounding information from video frames. However, most existing DCF-based trackers solely rely on pre-defined and fixed constraints such as a binary mask or quadratic function-based regularization to improve the discrimination. Unfortunately, such attempts fail to adjust the constraints according to the change of tracking circumstance in the video sequence, and thus lead to the lack of reliability of learned filters. To mitigate these problems, we present a novel DCF-based tracking method that introduces an adaptive target-and-surrounding soft mask (ATSM) into the learning formula. The adaptive soft mask that is represented by float numbers contains the detail information for both target region and its surrounding information: first, for the background area, it introduces meaningful background information and suppressing uninformative one; second, for the target area inside the bounding box, it helps to focus on the reliable area and repress the rapidly changing area; third, the target-and-surrounding soft mask is adaptively adjusted based on the variations of the target and its surrounding during the tracking process. By jointly modeling the filter and the adaptive soft mask, our ATSM tracker achieves an efficient integration of meaningful information of both foreground and background and performs favorably against state-of-the-art algorithms on seven well-known benchmarks. [ABSTRACT FROM AUTHOR]

Multimodal emotion recognition in conversational videos (ERC) develops rapidly in recent years. To fully extract the relative context from video clips, most studies build their models on the entire dialogues which make them lack of real-time ERC ability. Different from related researches, a novel multimodal emotion recognition model for conversational videos based on reinforcement learning and domain knowledge (ERLDK) is proposed in this paper. In ERLDK, the reinforcement learning algorithm is introduced to conduct real-time ERC with the occurrence of conversations. The collection of history utterances is composed as an emotion-pair which represents the multimodal context of the following utterance to be recognized. Dueling deep-Q-network (DDQN) based on gated recurrent unit (GRU) layers is designed to learn the correct action from the alternative emotion categories. Domain knowledge is extracted from public dataset based on the former information of emotion-pairs. The extracted domain knowledge is used to revise the results from the RL module and is transformed into other dataset to examine the rationality. The experimental results on datasets show that ERLDK achieves the state-of-the-art results on weighted average and most of the specific emotion categories. [ABSTRACT FROM AUTHOR]

Remote sensing image captioning (RSIC), which aims at generating a well-formed sentence for a remote sensing image, has attracted more attention in recent years. The general framework for RSIC is the encoder–decoder architecture containing two submodels of encoder and decoder. Although the significant performance is obtained, the encoder–decoder architecture is a black-box model with a lack of explainability. To overcome this drawback, in this article, we propose a new explainable word–sentence framework for RSIC. The proposed word–sentence framework consists of two parts: word extractor and sentence generator, where the former extracts the valuable words in the given remote sensing image, while the latter organizes these words into a well-formed sentence. The proposed framework decomposes RSIC into a word classification task and a word sorting task, which is more in line with human intuitive understanding. On the basis of the word–sentence framework, some ablation experiments are conducted on the three public RSIC data sets of Sydney-captions, UCM-captions, and RSICD to explore the specific and effective network structures. In order to evaluate the proposed word–sentence framework objectively, we further conduct some comparative experiments on these three data sets and achieve comparable results in comparison with the encoder–decoder-based methods. [ABSTRACT FROM AUTHOR]

Extreme instance imbalance among categories and combinatorial explosion make the recognition of Human-Object Interaction (HOI) a challenging task. Few studies have addressed both challenges directly. Motivated by the success of few-shot learning that learns a robust model from a few instances, we formulate HOI as a few-shot task in a meta-learning framework to alleviate the above challenges. Due to the fact that the intrinsical characteristic of HOI is diverse and interactive, we propose a Semantic-guided Attentive Prototypes Network (SAPNet) framework to learn a semantic-guided metric space where HOI recognition can be performed by computing distances to attentive prototypes of each class. Specifically, the model generates attentive prototypes guided by the category names of actions and objects, which highlight the commonalities of images from the same class in HOI. In addition, we design two alternative prototypes calculation methods, i.e., Prototypes Shift (PS) approach and Hallucinatory Graph Prototypes (HGP) approach, which explore to learn a suitable category prototypes representations in HOI. Finally, in order to realize the task of few-shot HOI, we reorganize 2 HOI benchmark datasets with 2 split strategies, i.e., HICO-NN, TUHOI-NN, HICO-NF, and TUHOI-NF. Extensive experimental results on these datasets have demonstrated the effectiveness of our proposed SAPNet approach. [ABSTRACT FROM AUTHOR]

In many real-world applications, we are often confronted with high dimensional data which are represented by various heterogeneous views. How to cluster this kind of data is still a challenging problem due to the curse of dimensionality and effectively integration of different views. To address this problem, we propose two parameter-free weighted multi-view projected clustering methods which perform structured graph learning and dimensionality reduction simultaneously. We can use the obtained structured graph directly to extract the clustering indicators, without performing other discretization procedures as previous graph-based clustering methods have to do. Moreover, two parameter-free strategies are adopted to learn an optimal weight for each view automatically, without introducing a regularization parameter as previous methods do. Extensive experiments on several public datasets demonstrate that the proposed methods outperform other state-of-the-art approaches and can be used more practically. [ABSTRACT FROM AUTHOR]

Smoke density estimation from a single image is a totally new but highly ill-posed problem. To solve the problem, we stack several convolutional encoder-decoder structures together to propose a wave-shaped neural network, termed W-Net. Stacking encoder-decoders directly increases the network depth, leading to the enlargement of receptive fields for encoding more semantic information. To maximize the degrees of feature re-usage, we copy and resize the outputs of encoding layers to corresponding decoding layers, and then concatenate them to implement short-cut connections for improving spatial accuracy. The crests and troughs of W-Net are special structures containing abundant localization and semantic information, so we also use short-cut connections between these structures and decoding layers. Estimated smoke density is useful in many applications, such as smoke segmentation, smoke detection, disaster simulation. Experimental results show that our method outperforms existing methods on both smoke density estimation and segmentation. It also achieves satisfying results in visual detection of auto exhausts. [ABSTRACT FROM AUTHOR]

This paper addresses the problem of supervised video summarization by formulating it as a sequence-to-sequence learning problem, where the input is a sequence of original video frames, and the output is a keyshot sequence. Our key idea is to learn a deep summarization network with attention mechanism to mimic the way of selecting the keyshots of human. To this end, we propose a novel video summarization framework named attentive encoder–decoder networks for video summarization (AVS), in which the encoder uses a bidirectional long short-term memory (BiLSTM) to encode the contextual information among the input video frames. As for the decoder, two attention-based LSTM networks are explored by using additive and multiplicative objective functions, respectively. Extensive experiments are conducted on two video summarization benchmark datasets, i.e., SumMe and TVSum. The results demonstrate the superiority of the proposed AVS-based approaches against the state-of-the-art approaches, with remarkable improvements on both datasets. [ABSTRACT FROM AUTHOR]

Sufficient training examples are the fundamental requirement for most of the learning tasks. However, collecting well-labelled training examples is costly. Inspired by Zero-shot Learning (ZSL) that can make use of visual attributes or natural language semantics as an intermediate level clue to associate low-level features with high-level classes, in a novel extension of this idea, we aim to synthesise training data for novel classes using only semantic attributes. Despite the simplicity of this idea, there are several challenges. First, how to prevent the synthesised data from over-fitting to training classes? Second, how to guarantee the synthesised data is discriminative for ZSL tasks? Third, we observe that only a few dimensions of the learnt features gain high variances whereas most of the remaining dimensions are not informative. Thus, the question is how to make the concentrated information diffuse to most of the dimensions of synthesised data. To address the above issues, we propose a novel embedding algorithm named Unseen Visual Data Synthesis (UVDS) that projects semantic features to the high-dimensional visual feature space. Two main techniques are introduced in our proposed algorithm. (1) We introduce a latent embedding space which aims to reconcile the structural difference between the visual and semantic spaces, meanwhile preserve the local structure. (2) We propose a novel Diffusion Regularisation (DR) that explicitly forces the variances to diffuse over most dimensions of the synthesised data. By an orthogonal rotation (more precisely, an orthogonal transformation), DR can remove the redundant correlated attributes and further alleviate the over-fitting problem. On four benchmark datasets, we demonstrate the benefit of using synthesised unseen data for zero-shot learning. Extensive experimental results suggest that our proposed approach significantly outperforms the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Fine-grained image categorization is a challenging task aiming at distinguishing objects belonging to the same basic-level category, e.g., leaf or mushroom. It is a useful technique that can be applied for species recognition, face verification, and so on. Most of the existing methods either have difficulties to detect discriminative object components automatically, or suffer from the limited amount of training data in each sub-category. To solve these problems, this paper proposes a new fine-grained image categorization model. The key is a dense graph mining algorithm that hierarchically localizes discriminative object parts in each image. More specifically, to mimic the human hierarchical perception mechanism, a superpixel pyramid is generated for each image. Thereby, graphlets from each layer are constructed to seamlessly capture object components. Intuitively, graphlets representative to each super-/sub-category is densely distributed in their feature space. Thus, a dense graph mining algorithm is developed to discover graphlets representative to each super-/sub-category. Finally, the discovered graphlets from pairwise images are integrated into an image kernel for fine-grained recognition. Theoretically, the learned kernel can generalize several state-of-the-art image kernels. Experiments on nine image sets demonstrate the advantage of our method. Moreover, the discovered graphlets from each sub-category accurately capture those tiny discriminative object components, e.g., bird claws, heads, and bodies. [ABSTRACT FROM PUBLISHER]

Sparse representation-based visual tracking approaches have attracted increasing interests in the community in recent years. The main idea is to linearly represent each target candidate using a set of target and trivial templates, while imposing a sparsity constraint onto the representation coefficients. After we obtain the coefficients using \ell 1 -norm minimization methods, the candidate with the lowest error, when it is reconstructed using only the target templates and the associated coefficients, is considered as the tracking result. In spite of promising system performance widely reported, it is unclear if the performance of these trackers can be maximized. In addition, computational complexity caused by the dimensionality of the feature space limits these algorithms in real-time applications. In this paper, we propose a real-time visual tracking method based on structurally random projection (RP) and weighted least squares (WLS) techniques. In particular, to enhance the discriminative capability of the tracker, we introduce background templates to the linear representation framework. To handle appearance variations over time, we relax the sparsity constraint using a WLS method to obtain the representation coefficients. To further reduce the computational complexity, structurally RP is used to reduce the dimensionality of the feature space, while preserving the pairwise distances between the data points in the feature space. Experimental results show that the proposed approach outperforms several state-of-the-art tracking methods. [ABSTRACT FROM AUTHOR]

To fill the semantic gap between the predictive power of computational saliency models and human behavior, this paper proposes to predict where people look at using spatial-aware object-level cues. While object-level saliency has been recently suggested by psychophysics experiments and shown effective with a few computational models, the spatial relationship between the objects has not yet been explored in this context. We in this work for the first time explicitly model such spatial relationship, as well as leveraging semantic information of an image to enhance object-level saliency modeling. The core computational module is a graphlet-based (i.e., graphlets are moderate-sized connected subgraphs) deep architecture, which hierarchically learns a saliency map from raw image pixels to object-level graphlets (oGLs) and further to spatial-level graphlets (sGLs). Eye tracking data are also used to leverage human experience in saliency prediction. Experimental results demonstrate that the proposed oGLs and sGLs well capture object-level and spatial-level cues relating to saliency, and the resulting saliency model performs competitively compared with the state-of-the-art. [ABSTRACT FROM PUBLISHER]

We propose a novel method to model human actions by explicitly coding motion and structure features that are separately extracted from video sequences. Firstly, the motion template (one feature map) is applied to encode the motion information and image planes (five feature maps) are extracted from the volume of differences of frames to capture the structure information. The Gaussian pyramid and center-surround operations are performed on each of the six obtained feature maps, decomposing each feature map into a set of subband maps. Biologically inspired features are then extracted by successively applying Gabor filtering and max pooling on each subband map. To make a compact representation, discriminative locality alignment is employed to embed the high-dimensional features into a low-dimensional manifold space. In contrast to sparse representations based on detected interest points, which suffer from the loss of structure information, the proposed model takes into account the motion and structure information simultaneously and integrates them in a unified framework; it therefore provides an informative and compact representation of human actions. The proposed method is evaluated on the KTH, the multiview IXMAS, and the challenging UCF sports datasets and outperforms state-of-the-art techniques on action recognition [ABSTRACT FROM AUTHOR]

Due to the popularity of social media websites, extensive research efforts have been dedicated to tag-based social image search. Both visual information and tags have been investigated in the research field. However, most existing methods use tags and visual characteristics either separately or sequentially in order to estimate the relevance of images. In this paper, we propose an approach that simultaneously utilizes both visual and textual information to estimate the relevance of user tagged images. The relevance estimation is determined with a hypergraph learning approach. In this method, a social image hypergraph is constructed, where vertices represent images and hyperedges represent visual or textual terms. Learning is achieved with use of a set of pseudo-positive images, where the weights of hyperedges are updated throughout the learning process. In this way, the impact of different tags and visual words can be automatically modulated. Comparative results of the experiments conducted on a dataset including 370+images are presented, which demonstrate the effectiveness of the proposed approach. [ABSTRACT FROM AUTHOR]

The biologically inspired model (BIM) proposed by Serre presents a promising solution to object categorization. It emulates the process of object recognition in primates' visual cortex by constructing a set of scale- and position-tolerant features whose properties are similar to those of the cells along the ventral stream of visual cortex. However, BIM has potential to be further improved in two aspects: mismatch by dense input and randomly feature selection due to the feedforward framework. To solve or alleviate these limitations, we develop an enhanced BIM (EBIM) in terms of the following two aspects: 1) removing uninformative inputs by imposing sparsity constraints, 2) apply a feedback loop to middle level feature selection. Each aspect is motivated by relevant psychophysical research findings. To show the effectiveness of the EBIM, we apply it to object categorization and conduct empirical studies on four computer vision data sets. Experimental results demonstrate that the EBIM outperforms the BIM and is comparable to state-of-the-art approaches in terms of accuracy. Moreover, the new system is about 20 times faster than the BIM. [ABSTRACT FROM AUTHOR]