Search

Object detection based on RGB and infrared images has emerged as a crucial research area in computer vision, and the synergy of RGB-Infrared ensures the robustness of object-detection algorithms under varying lighting conditions. However, the RGB-IR image pairs captured typically exhibit spatial misalignment due to sensor discrepancies, leading to compromised localization performance. Furthermore, since the inconsistent distribution of deep features from the two modalities, directly fusing multi-modal features will weaken the feature difference between the object and the background, therefore interfering with the RGB-Infrared object-detection performance. To address these issues, we propose an adaptive dual-discrepancy calibration network (ADCNet) for misaligned RGB-Infrared object detection, including spatial discrepancy and domain-discrepancy calibration. Specifically, the spatial discrepancy calibration module conducts an adaptive affine transformation to achieve spatial alignment of features. Then, the domain-discrepancy calibration module separately aligns object and background features from different modalities, making the distribution of the object and background of the fusion feature easier to distinguish, therefore enhancing the effectiveness of RGB-Infrared object detection. Our ADCNet outperforms the baseline by 3.3% and 2.5% in m A P 50 on the FLIR and misaligned M3FD datasets, respectively. Experimental results demonstrate the superiorities of our proposed method over the state-of-the-art approaches. [ABSTRACT FROM AUTHOR]

To answer questions, visual question answering systems (VQA) rely on language bias but ignore the information of the images, which has negative information on its generalization. The mainstream debiased methods focus on removing language prior to inferring. However, the image samples are distributed unevenly in the dataset, so the feature sets acquired by the model often cannot cover the features (views) of the tail samples. Therefore, language bias occurs. This paper proposes a language bias-driven self-knowledge distillation framework to implicitly learn the feature sets of multi-views so as to reduce language bias. Moreover, to measure the performance of student models, the authors of this paper use a generalization uncertainty index to help student models learn unbiased visual knowledge and force them to focus more on the questions that cannot be answered based on language bias alone. In addition, the authors of this paper analyze the theory of the proposed method and verify the positive correlation between generalization uncertainty and expected test error. The authors of this paper validate the method's effectiveness on the VQA-CP v2, VQA-CP v1 and VQA v2 datasets through extensive ablation experiments. [ABSTRACT FROM AUTHOR]

Video caption aims to generate descriptive sentences about the video, and the most critical problem is how to achieve accurate word prediction with standardized and coherent syntax structure, which requires the model to thoroughly understand video content and precisely map them into corresponding sentence components. Many existing methods usually fuse different video features into a single visual feature for generating sentences. However, they ignore the word dataset prior information in the annotations (such as Part-Of-Speech) and they also ignore the association between sentence components and types of visual features. To solve these problems, we propose a POS-trends dynamic-aware model (PDA) to fully exploit the word dataset prior information in the captions to predict POS tag, so as to assist generating captions. We propose a POS feature extraction (PFE) module to use different filters to extract different POS-trends features, predict POS tags and fuse visual features. Furthermore, we propose a visual-dynamic-aware (VDA) module to dynamically adjust the mapping way of words and supplement the visual information into the local features. The fusion features provide directional visual information to generate correct words, and the predicted POS tags to guide the decoding process to generate a more standardized and coherent syntax structure. A large number of experiments based on MSVD, MSR-VTT and VATEX demonstrated that our method outperforms the state-of-the-art methods in BLEU-4, ROUGE-L, METEOR, CIDEr. Code can be available at: https://github.com/WangLanxiao/PDA-for-video-caption. [ABSTRACT FROM AUTHOR]

Single image dehazing is the key to enhancing image visibility in outdoor scenes, which facilitates human observation and computer recognition. The existing approaches generally utilize a one-shot strategy that indiscriminately applies the same filters to all local regions. However, due to neglecting inhomogeneous illumination and detail distortion, their dehazed results easily suffer from underfiltering or overfiltering across different regions. To tackle this issue, we propose a region-adaptive two-shot network (RATNet) that follows a coarse-to-fine framework. First, a lightweight subnetwork is applied to execute regular global filtering and obtain an initially restored image. Then, a two-branch subnetwork is put forward whose branches separately refine its illumination and detail. Eventually, we derive the final prediction by adaptively aggregating the results after illumination modification and detail restoration, whose region-variant weights are jointly optimized by maximizing the similarity between our fused result and haze-free counterpart. Extensive experiments validate the superiority of our proposed algorithm. [ABSTRACT FROM AUTHOR]

Object proposals are used in two-stage detectors, such as R-CNN, to generate detection results, including category predictions and refined bounding-boxes. As a result, classification scores are assigned to refined bounding-boxes rather than object proposals. However, this procedure ignores the discrepancy of data distribution between object proposals and refined bounding-boxes. We consider this discrepancy could limit the detection accuracy. Specifically, the foreground/background imbalance on object proposals and inaccurate information from low-IoU proposals could hinder the category prediction. In this paper, we propose a detector called the Multi-Path Detection Calibration Network (PDC-Net) to address this problem. The key idea behind PDC-Net is calibrating detection results from R-CNN by considering the statistical discrepancy between object proposals and refined bounding-boxes. PDC-Net is built on Faster R-CNN. The core component in PDC-Net is the multi-path detection head, in which the base detector (from Faster R-CNN) generates detection results from object proposals and multiple calibration detectors fix incorrect outputs from the base detector using refined bounding-boxes. Experiments reveal that PDC-Net can boost detection results. Our method could reach 83.1% and 43.3% mAP respectively on PASCAL VOC and MSCOCO benchmarks, which is comparable to several state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Pixel-level segmentation has been widely used to improve object detection. Most of the existing methods refine detection features by adding the constraint of the segmentation branch or by simply embedding high-level segmentation features into detection features within the local receptive field. However, noisy segmentation features are unavoidable in real-word applications and can easily cause false positives. To address this problem, we propose a novel hierarchical context embedding module to effectively embed segmentation features into detection features. The idea of this module is to capture hierarchical context information that includes local objects or parts and nonlocal context features by learning multiple attention maps, and subsequently utilize interdependencies between features to recalibrate noisy segmentation features. Furthermore, we use this module in the proposed gated encoder-decoder network that adaptively aggregates feature maps of different resolutions based on the gate mechanism so that we can embed multiscale segmentation feature maps into detection features for more accurate detection of objects of all sizes. Experimental results demonstrate the effectiveness of the proposed method on the Pascal VOC 2012Seg dataset, the Pascal VOC dataset and the MS COCO dataset. [ABSTRACT FROM AUTHOR]

Images acquired by outdoor vision systems easily suffer poor visibility and annoying interference due to the rainy weather, which brings great challenge for accurately understanding and describing the visual contents. Recent researches have devoted great efforts on the task of rain removal for improving the image visibility. However, there is very few exploration about the quality assessment of de-rained image, even it is crucial for accurately measuring the performance of various de-raining algorithms. In this paper, we first create a de-raining quality assessment (DQA) database that collects 206 authentic rain images and their de-rained versions produced by 6 representative single image rain removal algorithms. Then, a subjective study is conducted on our DQA database, which collects the subject-rated scores of all de-rained images. To quantitatively measure the quality of de-rained image with non-uniform artifacts, we propose a bi-directional feature embedding network (B-FEN) which integrates the features of global perception and local difference together. Experiments confirm that the proposed method significantly outperforms many existing universal blind image quality assessment models. To help the research towards perceptually preferred de-raining algorithm, we will publicly release our DQA database and B-FEN source code on https://github.com/wqb-uestc. [ABSTRACT FROM AUTHOR]

In High Efficiency Video Coding (HEVC), multiple-QP (quantization parameter) optimization can adapt to a local video content. However, the multiple-QP implementation in the HEVC reference software (HM 16.6) achieves the best QP value for each coding block with a large amount of computational complexity. To address this challenge, we propose a fast rate-constrained multiple-QP optimization approach for the HM platform. We first introduce a template-based transform coefficient selection method which can save the overall complexity of entropy coding. In addition, we model the multiple-QP determination as a new rate-constrained optimization problem, and finally, we get a feasible solution with a lower computation overhead. Experimental results show that our method dramatically reduces the average complexity under the all-intra, low-delay and random-access configuration. [ABSTRACT FROM AUTHOR]

Head detection plays an important role in localizing and identifying persons from visual data. Most existing methods treat head detection as a specific form of object detection. Head detection is nontrivial due to the considerable difficulty in building the local and global information under conditions of unconstrained pose and orientation. To address these issues, this paper presents an effective adaptive relational network to capture context information, which is greatly helpful to suppress missed detection. We show that the fundamental contextual properties, such as the global shape priors from different heads and the local adjacent relationship between the head and shoulders, can be systematically quantified by visual operators. Specifically, we propose a two-step search algorithm to quantify the global intergroup conflict with adaptive scale, pose and viewpoint. Meanwhile, a structured feature module is introduced to capture the local relation of intraindividual stability. Finally, the global priors and local relation are integrated seamlessly into a single-stage head detector that is end-to-end trainable. An extensive ablation analysis demonstrates the effectiveness of our approach. We achieve state-of-the-art results on two challenging datasets, i.e., HollywoodHeads and Brainwash. [ABSTRACT FROM AUTHOR]

Unmanned aerial vehicles are an essential component in the realization of Industry 4.0. With drones helping to improve industrial safety and efficiency in utilities, construction, and communication, there is an urgent need for drone-based intelligent applications. In this paper, we develop a unified framework to simultaneously detect and count vehicles from drone images. We first explore why the state-of-the-art detectors fail in highly dense drone scenes, which provides more appropriate insights. Then, we propose an effective loss to push the anchors toward matching the ground-truth boxes as much as possible, specifically designed for scale-adaptive anchor generation. Inspired by attention mechanisms in the human visual system, we maximize the mutual information between object classes and features by combining bottom-up cues with top-down attention mechanisms specifically designed for feature extraction. Finally, we build a counting layer with a regularized constraint related to the number of vehicles. Extensive experiments demonstrate the effectiveness of our approach. For both tasks, our proposed method achieves state-of-the-art results on all four challenging datasets. In particular, our results reduce error by a larger factor than previous methods. [ABSTRACT FROM AUTHOR]

The ultra high-definition (UHD) video format, which has recently become popular, aims to provide high spatial resolution, high temporal frame rate, high sample bit-depth, and wide pixel color gamut. Despite the continued development of global network capacities, it inevitably causes the increased bandwidth cost of catering to the requirement of delivering UHD video services. To address such challenges, this paper presents an improved super coding unit (SCU) method for UHD video coding in High Efficiency Video Coding (HEVC). Initially, the medium coding unit (MCU) is proposed to avoid unnecessary brute-force coding unit (CU) partitions of SCU. Furthermore, the SCU is proposed to be encoded by Direct-MCU and SCU-to-MCU modes: the Direct-MCU mode is intended to better adapt to the texture-rich region, which guarantees the compression efficiency by avoiding extra-size CU partition; the SCU-to-MCU mode is designed for the homogeneous region of UHD content, which saves the encoding time by skipping fine-grained CU partition search. Moreover, a learning-based fast SCU decision approach is proposed to speed up the determination process of Direct-MCU and SCU-to-MCU, where three representative handcrafted features are extracted. Experimental results show that our method achieves an affordable complexity and excellent coding efficiency (up to 7.30% Bjøntegaard Delta rate savings) in UHD video coding compared to recent HEVC reference software. [ABSTRACT FROM AUTHOR]

In this paper, we propose a spatial–spectral feature fusion model with a predictive feature weighting mechanism and demonstrate its applications to the problems of hyperspectral image classification and segmentation. To address these problems, we learn a set of 1-D convolutional local spectral filters and 2-D spatial–spectral filters that feed features into a fusion module, in an end-to-end fashion. We propose a lightweight predictive feature weighting component embedded in the fusion model and consider four design fusion options, i.e., by adding or concatenating features with equal or predicted weights. For the pixel classification task, the training input consists of image patches with labeled central pixels, whereas for the spatial segmentation task, it includes the label maps of image regions. The proposed networks have been evaluated on the Indian Pines, Pavia University, and Houston University for the classification problem and the SpaceNet data set for the spatial segmentation problem. The quantitative results favor the proposed approach over the state-of-the-art methods across all the four data sets. [ABSTRACT FROM AUTHOR]

In this paper, we propose a generative framework that unifies depth-based 3D facial pose tracking and face model adaptation on-the-fly, in the unconstrained scenarios with heavy occlusions and arbitrary facial expression variations. Specifically, we introduce a statistical 3D morphable model that flexibly describes the distribution of points on the surface of the face model, with an efficient switchable online adaptation that gradually captures the identity of the tracked subject and rapidly constructs a suitable face model when the subject changes. Moreover, unlike prior art that employed ICP-based facial pose estimation, to improve robustness to occlusions, we propose a ray visibility constraint that regularizes the pose based on the face model's visibility with respect to the input point cloud. Ablation studies and experimental results on Biwi and ICT-3DHP datasets demonstrate that the proposed framework is effective and outperforms completing state-of-the-art depth-based methods. [ABSTRACT FROM AUTHOR]

3-D human body reconstruction is an important research topic in computer vision. A 3-D human body model can be used in sports science, movie industry and personalized entertainment, especially virtual reality games. Most of depth-based 3-D reconstruction algorithms need multiple cameras surrounding the user and require the user to keep a specific pose strictly while capturing depth images. In this paper, we propose an algorithm to reconstruct the 3-D shape of human bodies using a single commodity depth camera. Our algorithm only needs two depth images of the front-facing and back-facing bodies. It also has strong operability since the proposed method is insensitive to the pose variations between the two depth images. We reconstruct 3-D shapes of front-facing and back-facing bodies from the two depth images, respectively, and stitch them together. We also propose a novel registration method, namely, “iterative mid-distance points,” which has fast convergence and robustness to the depth noise. The proposed method enables robust and easy-to-use human body reconstruction, and achieves higher accuracy than state-of-the-art methods. [ABSTRACT FROM AUTHOR]

This paper proposes a novel Hierarchical Parsing Net (HPN) for semantic scene parsing. Unlike previous methods, which separately classify each object, HPN leverages global scene semantic information and the context among multiple objects to enhance scene parsing. On the one hand, HPN uses the global scene category to constrain the semantic consistency between the scene and each object. On the other hand, the context among all objects is also modeled to avoid incompatible object predictions. Specifically, HPN consists of four steps. In the first step, we extract scene and local appearance features. Based on these appearance features, the second step is to encode a contextual feature for each object, which models both the scene-object context (the context between the scene and each object) and the interobject context (the context among different objects). In the third step, we classify the global scene and then use the scene classification loss and a backpropagation algorithm to constrain the scene feature encoding. In the fourth step, a label map for scene parsing is generated from the local appearance and contextual features. Our model outperforms many state-of-the-art deep scene parsing networks on five scene parsing databases. [ABSTRACT FROM AUTHOR]

A joint encryption and authentication scheme for HEVC compressed video is proposed in this work. It produces a HEVC format compliant video stream that permits the authentication process to be carried out irregardless of the video being in the encrypted or plaintext (i.e., decrypted) form. To achieve this separable property, one set of syntax elements within the HEVC standard is utilized to achieve authentication, while another set is exploited for encryption. Specifically, for authentication, information such as coding unit size in each video slice is utilized to generate the authentication code, which is then embedded into each video slice to detect the tampered regions. A two-level authentication method is implemented to facilitate the legitimacy verification process. On the other hand, for video encryption, syntax elements in the HEVC standard including Sign Bins, Transform Skip Bins and Suffix Bins are randomized to perceptually distort the video. Experiment results show that by using the proposed encryption modules, the perceptual quality of the plaintext video can be distorted, and the distorted video can be restored to the original HEVC compressed video. Furthermore, the proposed joint scheme is verified to be viable in detecting and localizing the tampered regions. Finally, a pragmatic comparison among the proposed and conventional joint schemes is performed. [ABSTRACT FROM AUTHOR]

Depth image-based rendering is an important technique for virtual view synthesis with limited 3-D data. However, occluded areas result in disocclusions in synthesized images. Filling of the disocclusions in a plausible manner is a critical task in virtual view synthesis. In addition to spatial consistency, temporal consistency of the filled regions also affects the visual quality. In this paper, we propose a novel codebook method called sprite cell for filling the disocclusions with high spatial and temporal consistency. Each codeword consists of a color vector, depth value, frame log, and the confidence score of the corresponding pixel. In contrast with the existing methods that reuse the filling results of previous frames without considering their accuracy, the proposed method estimates the confidence scores of the filling results to prevent temporal continuation of filling errors. Moreover, we introduce a method to correct the luminance of filled disocclusions that compensates for the change of scenes. The experimental results show that the proposed method achieves both objective and subjective improvements over the state-of-the-art methods. The sequences synthesized with the proposed method have higher spatio-temporal consistency. [ABSTRACT FROM PUBLISHER]

Object part segmentation is an important and challenging task in computer vision. The existing supervised part segmentation methods need pixel level training data which leads to a huge workload for the user. In this paper a weakly supervised part segmentation method is proposed which segments part regions from multiple images by only several seeds on an image. Two aspects such as seed propagation among multiple images and part generation from seeds are considered. The first aspect is to generate part seeds in each image in terms of seed propagation which is accomplished by part matching combined with latent object regions. We fuse the local part matching and global shape cosegmentation to avoid the noise propagation. The second aspect is to segment part regions from object regions and part seeds which is formulated as the object shape decomposition model. The shape convexity analysis and seed location are fused to accomplish the decomposition and the final part segmentation. The proposed method is verified on the PASCAL 2010 dataset Bird dataset Cat-Dog dataset and UCF Sports Actions dataset. Experimental results demonstrate the effectiveness of the proposed method with larger intersection over union (IOU) values compared with existing weakly supervised part generation methods. [ABSTRACT FROM PUBLISHER]

Due to the efficiency and effectiveness of hashing technologies, they have become increasingly popular in large-scale image semantic retrieval. However, existing hash methods suppose that the data distributions satisfy the manifold assumption that semantic similar samples tend to lie on a low-dimensional manifold, which will be weakened due to the large intraclass variation. Moreover, these methods learn hash functions by relaxing the discrete constraints on binary codes to real value, which will introduce large quantization loss. To tackle the above problems, this paper proposes a novel unsupervised hashing algorithm to learn efficient binary codes from high-level feature representations. More specifically, we explore nonnegative matrix factorization for learning high-level visual features. Ultimately, binary codes are generated by performing binary quantization in the high-level feature representations space, which will map images with similar (visually or semantically) high-level feature representations to similar binary codes. To solve the corresponding optimization problem involving nonnegative and discrete variables, we develop an efficient optimization algorithm to reduce quantization loss with guaranteed convergence in theory. Extensive experiments show that our proposed method outperforms the state-of-the-art hashing methods on several multilabel real-world image datasets. [ABSTRACT FROM PUBLISHER]

In this paper, we propose a novel no-reference image quality assessment method for the retargeted image based on the pairwise rank learning approach. Each retargeted image needs to be first represented as a feature vector, which not only captures the image characteristics but also is sensitive to distortions during the retargeting process. As such, we investigate and examine different image representations for their abilities depicting the perceptual quality of retargeted image. Based on the image representations, we resort to the pairwise rank learning approach to discriminate the perceptual quality between the retargeted image pairs. Experimental results demonstrate that the proposed method can effectively depict the perceptual quality of the retargeted image, which can even perform comparably with the full-reference quality assessment methods. [ABSTRACT FROM PUBLISHER]

We propose a real-time method to accurately track the human head pose in the 3-dimensional (3D) world. Using a RGB-Depth camera, a face template is reconstructed by fitting a 3D morphable face model, and the head pose is determined by registering this user-specific face template to the input depth video. [ABSTRACT FROM AUTHOR]

Video quality fluctuation plays a significant role in human visual perception, and hence, many rate control approaches have been widely developed to maintain consistent quality for video communication. This paper presents a novel rate control framework based on the Lagrange multiplier in high-efficiency video coding. With the assumption of constant quality control, a new relationship between the distortion and the Lagrange multiplier is established. Based on the proposed distortion model and buffer status, we obtain a computationally feasible solution to the problem of minimizing the distortion variation across video frames at the coding tree unit level. Extensive simulation results show that our method outperforms the rate control used in HEVC Test Model (HM) by providing a more accurate rate regulation, lower video quality fluctuation, and stabler buffer fullness. The average peak signal-to-noise ratio (PSNR) and PSNR deviation improvements are about 0.37 dB and 57.14% in the low-delay (P and B) video communication, where the complexity overhead is $\sim 4.44\%$ . [ABSTRACT FROM PUBLISHER]

In this paper, we extend the free-energy principle to video quality assessment (VQA) by incorporating with the recent psychophysical study on human visual speed perception (HVSP). A novel video quality metric, namely the free-energy principle inspired video quality metric (FePVQ), is therefore developed and applied to perceptual video coding optimization. The free-energy principle suggests that the human visual system (HVS) can actively predict “orderly” information and avoid “disorderly” information for image perception. Basically, “orderly” is associated with the skeletons and edges of objects, and “disorderly” mostly concerns textures in images. Based on this principle, an image is separated into orderly and disorderly regions, and processed differently in image quality assessment. For videos, visual attention, or fixation, is associated with the objects with significant motion according to HVSP, resulting in a motion strength factor in the FePVQ so that the free-energy principle is extended into spatio-temporal domain for VQA. In addition, we investigate the application of the FePVQ in perceptual rate distortion optimization (RDO). For this purpose, the FePVQ is realized with low computational cost by using the relative total variation model and the block-wise motion vectors of video coding to simulate the free-energy principle and the HVSP, respectively. The experimental results indicate that the proposed FePVQ is highly consistent with the HVS perception. The linear correlation coefficient and Spearman's rank-order correlation coefficient are up to 0.8324 and 0.8281 on the LIVE video database. Better perceptual quality of encoded video sequences is achieved by FePVQ-motivated RDO in video coding. [ABSTRACT FROM PUBLISHER]

The emerging high efficiency video coding (HEVC) standard has improved compression performance significantly in comparison with H.264/AVC. However, more intensive computational complexity has been introduced by adopting a number of new coding tools. In this paper, a fast inter CU decision is proposed based on the latent sum of absolute differences (SAD) estimation. Firstly, a two-layer motion estimation (ME) method is designed to take advantage of the latent SAD cost. The new ME method can obtain the SAD costs for both the upper CU and its sub-CUs. Secondly, a concept of motion compensation rate- distortion (R-D) cost is defined, and an exponential model is proposed to express the relationship between the motion compensation R-D cost and the SAD cost. Then, a fast CU decision approach is designed based on the exponential model. The fast CU decision is implemented by comparing a derived threshold with the SAD cost difference between the upper and sub SAD costs. Experimental results show that the proposed algorithm achieves an average of 52% and 58.4% reductions of the coding time at the cost of 1.61% and 2% bit-rate increases under the low delay and random access conditions, respectively. [ABSTRACT FROM PUBLISHER]