Search

This paper addresses the problem of interactive object segmentation with an input rectangle. We present a coarse-to-fine method from region-level segmentation to pixel-level segmentation. In the region-level segmentation, the best combination of adjacent refined superpixels is selected as the coarse segmentation result by measuring its global contrast and tightness degree. Subsequently, we use the coarse segmentation result to aid the construction of the energy function in the pixel-level segmentation. The result can be further refined due to the fusion of region-level and pixel-level segmentation. Experimental results demonstrate that our method can achieve better segmentation performance. [ABSTRACT FROM AUTHOR]

Object detection based on RGB and infrared images has emerged as a crucial research area in computer vision, and the synergy of RGB-Infrared ensures the robustness of object-detection algorithms under varying lighting conditions. However, the RGB-IR image pairs captured typically exhibit spatial misalignment due to sensor discrepancies, leading to compromised localization performance. Furthermore, since the inconsistent distribution of deep features from the two modalities, directly fusing multi-modal features will weaken the feature difference between the object and the background, therefore interfering with the RGB-Infrared object-detection performance. To address these issues, we propose an adaptive dual-discrepancy calibration network (ADCNet) for misaligned RGB-Infrared object detection, including spatial discrepancy and domain-discrepancy calibration. Specifically, the spatial discrepancy calibration module conducts an adaptive affine transformation to achieve spatial alignment of features. Then, the domain-discrepancy calibration module separately aligns object and background features from different modalities, making the distribution of the object and background of the fusion feature easier to distinguish, therefore enhancing the effectiveness of RGB-Infrared object detection. Our ADCNet outperforms the baseline by 3.3% and 2.5% in m A P 50 on the FLIR and misaligned M3FD datasets, respectively. Experimental results demonstrate the superiorities of our proposed method over the state-of-the-art approaches. [ABSTRACT FROM AUTHOR]

We propose a real-time method to accurately track the human head pose in the 3-dimensional (3D) world. Using a RGB-Depth camera, a face template is reconstructed by fitting a 3D morphable face model, and the head pose is determined by registering this user-specific face template to the input depth video. [ABSTRACT FROM AUTHOR]

Video quality fluctuation plays a significant role in human visual perception, and hence, many rate control approaches have been widely developed to maintain consistent quality for video communication. This paper presents a novel rate control framework based on the Lagrange multiplier in high-efficiency video coding. With the assumption of constant quality control, a new relationship between the distortion and the Lagrange multiplier is established. Based on the proposed distortion model and buffer status, we obtain a computationally feasible solution to the problem of minimizing the distortion variation across video frames at the coding tree unit level. Extensive simulation results show that our method outperforms the rate control used in HEVC Test Model (HM) by providing a more accurate rate regulation, lower video quality fluctuation, and stabler buffer fullness. The average peak signal-to-noise ratio (PSNR) and PSNR deviation improvements are about 0.37 dB and 57.14% in the low-delay (P and B) video communication, where the complexity overhead is $\sim 4.44\%$ . [ABSTRACT FROM PUBLISHER]

This paper presents a directional transform scheme for coding interprediction errors in block-based hybrid video coding. It proposes a two-layer transform structure, where the first layer uses discrete wavelet transform to compact the residue energy to the LL band and then the second layer uses 2-D nonseparable directional transforms to deal with the arbitrary edge directions in the four subbands. By doing this, the edges in a macroblock are efficiently compacted to a few coefficients and at the same time the overhead used to indicate the transform directions is affordable. Experimental results show that the proposed scheme provides peak signal-to-noise ratio gain up to 0.46 dB, compared with H.264/AVC High Profile. [ABSTRACT FROM PUBLISHER]

In this paper, we introduce a method to detect co-saliency from an image pair that may have some objects in common. The co-saliency is modeled as a linear combination of the single-image saliency map (SISM) and the multi-image saliency map (MISM). The first term is designed to describe the local attention, which is computed by using three saliency detection techniques available in literature. To compute the MISM, a co-multilayer graph is constructed by dividing the image pair into a spatial pyramid representation. Each node in the graph is described by two types of visual descriptors, which are extracted from a representation of some aspects of local appearance, e.g., color and texture properties. In order to evaluate the similarity between two nodes, we employ a normalized single-pair SimRank algorithm to compute the similarity score. Experimental evaluation on a number of image pairs demonstrates the good performance of the proposed method on the co-saliency detection task. [ABSTRACT FROM AUTHOR]

In this paper, we present a multiview approach to segment the foreground objects consisting of a group of people into individual human objects and track them across the video sequence. Depth and occlusion information recovered from multiple views of the scene is integrated into the object detection, segmentation, and tracking processes. Adaptive background penalty with occlusion reasoning is proposed to separate the foreground regions from the background in the initial frame. Multiple cues are employed to segment individual human objects from the group. To propagate the segmentation through video, each object region is independently tracked by motion compensation and uncertainty refinement, and the motion occlusion is tackled as layer transition. The experimental results implemented on both our sequences and other's sequence have demonstrated the algorithm's efficiency in terms of subjective performance. Objective comparison with a state-of-the-art algorithm validates the superior performance of our method quantitatively. [ABSTRACT FROM PUBLISHER]

Abstract: The proposed interpolation filter comprises two concatenating filters, adaptive pre-interpolation filter (APIF) and the normative interpolation filter in H.264/AVC. The former is applied only to the integer pixels in the reference frames; the latter generates all the sub-position samples, supported by the output of APIF. The convolution of APIF and the standard filter minimizes the motion prediction error on a frame basis. APIF preserves the merits of the adaptive interpolation filter (AIF) and the adaptive loop filter (ALF) in the key technical area (KTA) software and at the same time overcomes their drawbacks. The experimental results show that APIF outperforms either AIF or ALF. Compared with the joint use of AIF and ALF, APIF provides comparable performance, but has much lower complexity. [Copyright &y& Elsevier]

Most existing feature detectors assume no surface discontinuity within the keypoints' support regions and, hence, have little chance to match the keypoints located on or near the surface boundaries. These keypoints, though not many, are salient and representative. In this paper, we show that they can be successfully matched by using the proposed scale- and affine-invariant Fan features. Specifically, the image neighborhood of a keypoint is depicted by multiple fan subregions, namely Fan features, to provide robustness to surface discontinuity and background change. These Fan features are made scale-invariant by using the automatic scale selection method based on the Fan Laplacian of Gaussian (FLOG). Affine invariance is further introduced to the Fan features based on the affine shape diagnosis of the mirror-predicted surface patch. The Fan features are then described by Fan-SIFT, which is an extension of the famous scale-invariant feature transform (SIFT) descriptor. Experimental results of quantitative comparisons show that the proposed Fan feature has good repeatability that is comparable to the state-of-the-art features for general structured scenes. Moreover, by using Fan features, we can successfully match image structures near surface discontinuities despite significant scale, viewpoint, and background changes. These structures are complementary to those found by the traditional methods and are especially useful for describing weakly textured scenes, which is demonstrated in our experiments on image matching and object rendering. [ABSTRACT FROM AUTHOR]

Abstract: In this paper, we propose a novel Adaptive Block-size Transform (ABT) based Just-Noticeable Difference (JND) model for images/videos. Extension from 8×8 Discrete Cosine Transform (DCT) based JND model to 16×16 DCT based JND is firstly performed by considering both the spatial and temporal Human Visual System (HVS) properties. For still images or INTRA video frames, a new spatial selection strategy based on the Spatial Content Similarity (SCS) between a macroblock and its sub-blocks is proposed to determine the transform size to be employed to generate the JND map. For the INTER video frames, a temporal selection strategy based on the Motion Characteristic Similarity (MCS) between a macroblock and its sub-blocks is presented to decide the transform size for the JND. Compared with other JND models, our proposed scheme can tolerate more distortions while preserving better perceptual quality. In order to demonstrate the efficiency of the ABT-based JND in modeling the HVS properties, a simple visual quality metric is designed by considering the ABT-based JND masking properties. Evaluating on the image and video subjective databases, the proposed metric delivers a performance comparable to the state-of-the-art metrics. It confirms that the ABT-based JND consists well with the HVS. The proposed quality metric also is applied on ABT-based H.264/Advanced Video Coding (AVC) for the perceptual video coding. The experimental results demonstrate that the proposed method can deliver video sequences with higher visual quality at the same bit-rates. [Copyright &y& Elsevier]

Abstract: In this paper, we present an automatic algorithm to segment multiple objects from multi-view video. The Initial Interested Objects (IIOs) are automatically extracted in the key view of the initial frame based on the saliency model. Multiple objects segmentation is decomposed into several sub-segmentation problems, and solved by minimizing the energy function using binary label graph cut. In the proposed novel energy function, the color and depth cues are integrated with the data term, which is then modified with background penalty with occlusion reasoning. In the smoothness term, foreground contrast enhancement is developed to strengthen the moving objects boundary, and at the same time attenuates the background contrast. To segment the multi-view video, the coarse predictions of the other views and the successive frame are projected by pixel-based disparity and motion compensation, respectively, which exploits the inherent spatiotemporal consistency. Uncertain band along the object boundary is shaped based on activity measure and refined with graph cut, resulting in a more accurate Interested Objects (IOs) layer across all views of the frames. The experiments are implemented on a couple of multi-view videos with real and complex scenes. Excellent subjective results have shown the robustness and efficiency of the proposed algorithm. [Copyright &y& Elsevier]

Abstract: In this paper, we present an efficient external calibration technique for a typical multi-camera system. The technique is very handy in practice using a simple planar pattern. Based on homography, three different pair-wise estimation algorithms are proposed to recover the rigid rotation and translation between neighboring cameras. By registering the accurate and reliable partial calibrated structures, the complete calibration of a multi-camera system can be accomplished. In order to evaluate and compare the calibration results for different camera setups, a novel accuracy metric is introduced based on the deflection angles of projection rays. Experimental results using both simulated and real data are presented to verify the validity and performance of the proposed method. [Copyright &y& Elsevier]

Abstract: In this paper, we propose a novel two-pass rate control algorithm to achieve constant quality for H.264/AVC high definition video coding. With the first-pass collected rate and distortion information and the built model of scene complexity, the encoder can determine the expected distortion which could be achieved in the second-pass encoding under the target bit rate. According to the built linear distortion-quantizer (D-Q) model, before encoding one frame, the quantization parameter can be solved to realize constant quality encoding. After encoding one frame, the model parameters will be updated with linear regression method to ensure the prediction accuracy of the quantization parameter of next encoded frame with the same coding type. In order to obtain the expected distortion of each frame under the target bit rate, a GOP-level bit allocation scheme is also designed to adjust the target bit rate of each GOP based on the scene complexity of the GOP in the second-pass encoding. In addition, the effect of scene change on the updating of D-Q model is considered. The model will be re-initialized at the scene change to minimize modeling error. The experimental results show that compared with the latest two-pass rate control algorithm, our proposed algorithm can significantly improve the bit control accuracy at comparable coding performance in terms of constant quality and average PSNR. On average, the improvement of bit control accuracy achieved about 90%. [Copyright &y& Elsevier]

Abstract: Motion estimation and compensation in wavelet domain have received much attention recently. To overcome the inefficiency of motion estimation in critically sampled wavelet domain, the low-band-shift (LBS) method and the complete-to-overcomplete discrete wavelet transform (CODWT) method are proposed for motion estimation in shift-invariant wavelet domain. However, a major disadvantage of these methods is the computational complexity. Although the CODWT method has reduced the computational complexity by skipping the inverse wavelet transform and making the direct link between the critically sampled subbands and the shift-invariant subbands, the full search algorithm (FSA) increases it. In this paper, we proposed two fast multiresolution motion estimation algorithms in shift-invariant wavelet domain: one is the wavelet matching error characteristic based partial distortion search (WMEC-PDS) algorithm, which improves computational efficiency of conventional partial distortion search algorithms while keeping the same estimate accuracy as the FSA; another is the anisotropic double cross search (ADCS) algorithm using multiresolution-spatio-temporal context, which provides a significantly computational load reduction while only introducing negligible distortion compared with the FSA. Due to the multiresolution nature, both the proposed approaches can be applied to wavelet-based scalable video coding. Experimental results show the superiority of the proposed fast motion estimation algorithms against other fast algorithms in terms of speed-up and quality. [Copyright &y& Elsevier]

Abstract: In this paper, we review the recent advances in rate control techniques for video coding. The rate control algorithms recommended in the video coding standards are briefly described and analyzed. Recent advances, such as new concepts in rate-distortion modelling and quality constrained control, are presented. With these techniques, the rate control performance can be improved. The paper not only summarizes these recent rate control techniques but also provides explicit directions for future rate control algorithm design. [Copyright &y& Elsevier]

In this paper, we review the rate distortion tradeoff issues in real-time video coding and introduce a minimum variation (MINVAR) distortion criterion based approach. The MINVAR based rate distortion tradeoff framework provides a local optimization strategy as a rate control mechanism in real-time video coding applications by minimizing the distortion variation while the corresponding bit rate fluctuation is limited by utilizing the encoder buffer. The proposed approach aims to achieve a smooth decoded picture quality for pleasing human visual experience. The performance of the proposed method is evaluated with H.264. The experimental results demonstrate that using the proposed approach, the decoded picture quality is smoother than the traditional H.264 joint model (JM) rate control without sacrificing global quality such that a better subjective visual quality is guaranteed. [Copyright &y& Elsevier]

Abstract: In this paper, an object-based embedded image coding algorithm based on context classification and quadtree ordering in wavelet packet domain (OB-CCAQO) is proposed. To match up well with the quadtree-based embedded coder, a new cost function for the best basis selection is adopted in SA-DWPT (shape adaptive discrete wavelet packet transform). The significance probability of wavelet coefficient is estimated by convoluting a FIR filter matrix kernel with the significance states of neighboring coefficients, and then a context classifier based on Lloyd–Max algorithm is used to categorize the wavelet coefficients with the same or similar significance probability into several contexts. The significance state of wavelet coefficient with respect to a given threshold is encoded using adaptive arithmetic coder based on the classified context. In addition, combined with the structure of the best wavelet packet basis, a complete quadtree representation of wavelet packet coefficients is established to explore the ordering procedure of embedded object-based image coding. Experimental results show that the proposed object-based embedded image coder offers coding performance superior to the popular object-based coders and is also comparable to or better than the state-of-art embedded image coder when applied in the framed-based image coding. [Copyright &y& Elsevier]

Images acquired by outdoor vision systems easily suffer poor visibility and annoying interference due to the rainy weather, which brings great challenge for accurately understanding and describing the visual contents. Recent researches have devoted great efforts on the task of rain removal for improving the image visibility. However, there is very few exploration about the quality assessment of de-rained image, even it is crucial for accurately measuring the performance of various de-raining algorithms. In this paper, we first create a de-raining quality assessment (DQA) database that collects 206 authentic rain images and their de-rained versions produced by 6 representative single image rain removal algorithms. Then, a subjective study is conducted on our DQA database, which collects the subject-rated scores of all de-rained images. To quantitatively measure the quality of de-rained image with non-uniform artifacts, we propose a bi-directional feature embedding network (B-FEN) which integrates the features of global perception and local difference together. Experiments confirm that the proposed method significantly outperforms many existing universal blind image quality assessment models. To help the research towards perceptually preferred de-raining algorithm, we will publicly release our DQA database and B-FEN source code on https://github.com/wqb-uestc. [ABSTRACT FROM AUTHOR]

In this paper, we propose a spatial–spectral feature fusion model with a predictive feature weighting mechanism and demonstrate its applications to the problems of hyperspectral image classification and segmentation. To address these problems, we learn a set of 1-D convolutional local spectral filters and 2-D spatial–spectral filters that feed features into a fusion module, in an end-to-end fashion. We propose a lightweight predictive feature weighting component embedded in the fusion model and consider four design fusion options, i.e., by adding or concatenating features with equal or predicted weights. For the pixel classification task, the training input consists of image patches with labeled central pixels, whereas for the spatial segmentation task, it includes the label maps of image regions. The proposed networks have been evaluated on the Indian Pines, Pavia University, and Houston University for the classification problem and the SpaceNet data set for the spatial segmentation problem. The quantitative results favor the proposed approach over the state-of-the-art methods across all the four data sets. [ABSTRACT FROM AUTHOR]

Video caption aims to generate descriptive sentences about the video, and the most critical problem is how to achieve accurate word prediction with standardized and coherent syntax structure, which requires the model to thoroughly understand video content and precisely map them into corresponding sentence components. Many existing methods usually fuse different video features into a single visual feature for generating sentences. However, they ignore the word dataset prior information in the annotations (such as Part-Of-Speech) and they also ignore the association between sentence components and types of visual features. To solve these problems, we propose a POS-trends dynamic-aware model (PDA) to fully exploit the word dataset prior information in the captions to predict POS tag, so as to assist generating captions. We propose a POS feature extraction (PFE) module to use different filters to extract different POS-trends features, predict POS tags and fuse visual features. Furthermore, we propose a visual-dynamic-aware (VDA) module to dynamically adjust the mapping way of words and supplement the visual information into the local features. The fusion features provide directional visual information to generate correct words, and the predicted POS tags to guide the decoding process to generate a more standardized and coherent syntax structure. A large number of experiments based on MSVD, MSR-VTT and VATEX demonstrated that our method outperforms the state-of-the-art methods in BLEU-4, ROUGE-L, METEOR, CIDEr. Code can be available at: https://github.com/WangLanxiao/PDA-for-video-caption. [ABSTRACT FROM AUTHOR]

Instance segmentation has made great progress in recent years. However, current mainstream detection-based methods ignore the process of distinguishing different instances in the same detected region, making it hard to segment the correct instance when a detected region contains multiple instances. To address this problem, we propose an Instance-level Context Attention Network (ICANet) to generate more discriminative features for different instances based on the context from their respective instance scopes, called instance-level context. Specifically, given a detected region, we first propose an instance attention module to obtain the specific attention maps that focus on the relationships between pixel pairs from the same instance by learning an embedding space. With this type of attention map, the features from the same instance achieve mutual gains, while the features from different instances become more discriminative. Then, we propose a spatial attention module to incorporate spatial information and enhance feature representations based on both feature and spatial relations. Moreover, to obtain clearer attention maps, we further propose a weight clipping strategy to filter out the noise by cutting off the lower weight. We perform extensive experiments to verify the effectiveness of the proposed method. As reported in the results, our method steadily outperforms the baseline by over 1.5% on the COCO dataset using different backbones and 3.7% on the Cityscapes dataset, which demonstrates the effectiveness of our method. [ABSTRACT FROM AUTHOR]

In this paper, we propose a novel feature adaptive co-segmentation method that can learn adaptive features of different image groups for accurate common objects segmentation. We also propose image complexity awareness for adaptive feature learning. In the proposed method, the original images are first ranked according to the image complexities that are measured by superpixel changing cue and object detection cue. Then, the unsupervised segments of the simple images are used to learn the adaptive features, which are achieved using an expectation-minimization algorithm combining l 1-regularized least squares optimization with the consideration of the confidence of the simple image segmentation accuracies and the fitness of the learned model. The error rate of the final co-segmentation is tested by the experiments on different image groups and verified to be lower than the existing state-of-the-art co-segmentation methods. [ABSTRACT FROM AUTHOR]

Abstract: In this paper, a novel reduced-reference (RR) image quality assessment (IQA) is proposed by depicting the subband statistical characteristics in the reorganized discrete cosine transform (RDCT) domain. First, the block-based DCT coefficients are reorganized into a three-level coefficient tree, resulting in ten RDCT subbands. For the intra RDCT subband characteristic, the coefficient distribution of each RDCT subband is modeled by the generalized Gaussian density (GGD) function. The city-block distance (CBD) is employed to measure the modeling error between the actual distribution and the fitted GGD curve. For the inter RDCT subband characteristic, the mutual information (MI) is utilized to depict the dependencies between coefficient pairs in related RDCT subbands. Moreover, a frequency ratio descriptor (FRD) calculated in the RDCT domain is proposed to depict how the image energy distributes among different frequency components. The FRD values computed from both the reference and distorted images are jointly considered to derive a novel mutual masking strategy for simulating the texture masking property of the human visual system (HVS). By considering the GGD modeling of intra RDCT subband, MI of inter RDCT subbands, and FRD of the image, the proposed RR IQA is developed. Experimental results demonstrate that a small number of RR features is sufficient to represent the reference image for the perceptual quality analysis. The proposed method can outperform the state-of-the-art RR IQAs, and even the full-reference (FR) PSNR and SSIM. [Copyright &y& Elsevier]

Class Activation Maps (CAMs) visualize the pixels within an image that contribute to classify the image to a certain category, and can be used to localize the object regions from images that benefit to many tasks such as image segmentation and object detection. However, the object regions highlighted by the existing methods are usually small and local. We believe such drawback lies in the image-level labels, because the the image-level label prefers the network to capture the common regions intra-class and the discriminative region inter-class. Due to the variations of objects intra-class and the common region widely shared inter class, the regions based on image-level labels are usually small and local. Based on such observation, we propose a new strategy to use a set of lower-level labels called component labels to replace the image-level labels. The advantage is that the component regions have small feature variation intra-class and are non-overlapping inter-class, which leads to better CAMs generation through their simple combination. Furthermore, since the component labels are also shared by unknown classes, the proposed CAMs generation method can be easily extended to unknown classes, which facilitate the improvement of tasks related to new class processing, such as few-shot segmentation and detection. Specifically, the component labels are set based on the WordNet hierarchy firstly, which can also provides the relationships of classes. Besides, graph convolution networks (GCNs) are used as the classifiers, which can exactly describe not only the component, but also their structural relationships. Based on the component features, a feature fusion module is also designed to merge local component features into the global feature. Better CAM is finally obtained. The experiment section shows the effectiveness of our component labels in terms of better subjective and objective results compared with the existing CAM generation methods. Furthermore, we also show good generalization of our component label on unknown classes. [ABSTRACT FROM AUTHOR]

Visual quality consistency is one of the most important issues in video quality assessment. When people view a sequential video, they may have an unpleasant perceptual experience if the video has an inconsistent visual quality even though the average visual quality of the video is not compromised. Thus, consistent visual quality control is mostly expected in general video encoding with limited channel bandwidth and buffer resources. However, there still has not been enough study on such an issue. In this paper, a new objective visual quality metric (VQM) is proposed first, which can easily be incorporated into video coding for guiding video coding. Second, a VQM-based window model is proposed to handle the tradeoff between visual quality consistency and buffer constraint in video coding. Third, a window-level rate control algorithm is developed to accomplish visual quality control based on the above two proposals. Finally, experimental results prove that consistent visual quality, high rate-distortion efficiency, accurate bit control, and compliant buffer constraint can be achieved by the proposed rate control algorithm. [ABSTRACT FROM PUBLISHER]

In this paper, a novel reduced-reference (RR) video quality assessment (VQA) is proposed by exploiting the spatial information loss and the temporal statistical characteristics of the interframe histogram. From the spatial perspective, an energy variation descriptor (EVD) is proposed to measure the energy change of each individual encoded frame, which results from the quantization process. Besides depicting the energy change, EVD can further simulate the texture masking property of the human visual system (HVS). From the temporal perspective, the generalized Gaussian density (GGD) function is employed to capture the natural statistics of the interframe histogram distribution. The city-block distance (CBD) is used to calculate the histogram distance between the original video sequence and the encoded one. For simplicity, the difference image between adjacent frames is employed to characterize the temporal interframe relationship. By combining the spatial EVD together with the temporal CBD, an efficient RR VQA is developed. Evaluation on the subjective quality video database demonstrates that the proposed method outperforms the representative RR video quality metric and the full-reference VQAs, such as peak signal-to-noise ratio and structure similarity index in matching subjective ratings. This means that the proposed metric is more consistent with the HVS perception. Furthermore, as only a small number of RR features are extracted for representing the original video sequence (each frame requires only one parameter for describing EVD and three parameters for recording GGD), the RR features can be embedded into the video sequences or transmitted through the ancillary data channel, which can be used in the video quality monitoring system. [ABSTRACT FROM PUBLISHER]

Video quality assessment plays a fundamental role in video processing and communication applications. In this paper, we study the use of motion information and temporal human visual system (HVS) characteristics for objective video quality assessment. In our previous work, two types of spatial distortions, i.e., detail losses and additive impairments, are decoupled and evaluated separately for spatial quality assessment. The detail losses refer to the loss of useful visual information that will affect the content visibility, and the additive impairments represent the redundant visual information in the test image, such as the blocking or ringing artifacts caused by data compression and so on. In this paper, a novel full-reference video quality metric is developed, which conceptually comprises the following processing steps: 1) decoupling detail losses and additive impairments within each frame for spatial distortion measure; 2) analyzing the video motion and using the HVS characteristics to simulate the human perception of the spatial distortions; and 3) taking into account cognitive human behaviors to integrate frame-level quality scores into sequence-level quality score. Distinguished from most studies in the literature, the proposed method comprehensively investigates the use of motion information in the simulation of HVS processing, e.g., to model the eye movement, to predict the spatio-temporal HVS contrast sensitivity, to implement the temporal masking effect, and so on. Furthermore, we also prove the effectiveness of decoupling detail losses and additive impairments for video quality assessment. The proposed method is tested on two subjective quality video databases, LIVE and IVP, and demonstrates the state-of-the-art performance in matching subjective ratings. [ABSTRACT FROM AUTHOR]

Because of the outstanding contribution in improving compression efficiency, block-based quantization has been widely accepted in state-of-the-art image/video coding standards. However, false contour artifacts are introduced, which result in reducing the fidelity of the decoded image/video especially in terms of subjective quality. In this paper, a block-based decontouring method is proposed to reduce the false contour artifacts in the decoded image/video by automatically dithering its direct current (DC) value according to a composite model established between gradient smoothness and block-edge smoothness. Feature points on the model with the corresponding criteria in suppressing contour artifacts are compared to show a good consistency between the model and the actual processing effects. Discrete cosine transform (DCT)-based block level contour artifacts detection mechanism ensures the blocks within the texture region are not affected by the DC dithering. Both the implementation method and the algorithm complexity are analyzed to present the feasibility in integrating the proposed method into an existing video decoder on an embedded platform or system-on-chip (SoC). Experimental results demonstrate the effectiveness of the proposed method both in terms of subjective quality and processing complexity in comparison with the previous methods. [ABSTRACT FROM PUBLISHER]

Abstract: In this paper, we propose an automatic human body segmentation system which mainly consists of human body detection and object segmentation. Firstly, an automatic human body detector is designed to provide hard constraints on the object and background for segmentation. And a coarse-to-fine segmentation strategy is employed to deal with the situation of partly detected object. Secondly, background contrast removal (BCR) and self-adaptive initialization level set (SAILS) are proposed to solve the tough segmentation problems of the high contrast at object boundary and/or similar colors existing in the object and background. Finally, an object updating scheme is proposed to detect and segment new object when it appears in the scene. Experimental results demonstrate that our body segmentation system works very well in the live video and standard sequences with complex background. [Copyright &y& Elsevier]

Abstract: AVS1-P2 is the newest video standard of Audio Video coding Standard (AVS) workgroup of China, which provides close performance to H.264/AVC main profile with lower complexity. In this paper, a platform-independent software package with macroblock-based (MB-based) architecture is proposed to facilitate AVS video standard implementation on embedded system. Compared with the frame-based architecture, which is commonly utilized for PC platform oriented video applications, the MB-based decoder performs all of the decoding processes, except the high-level syntax parsing, in a set of MB-based buffers with adequate size for saving the information of the current MB and the neighboring reference MBs to minimize the on-chip memory and to save the time consumed in on-chip/off-chip data transfer. By modifying the data flow and decoding hierarchy, simulating the data transfer between the on-chip memory and the off-chip memory, and modularizing the buffer definition and management for low-level decoding kernels, the MB-based system architecture provides over 80% reduction in on-chip memory compared to the frame-based architecture when decoding 720p sequences. The storage complexity is also analyzed by referencing the performance evaluation of the MB-based decoder. The MB-based decoder implementation provides an efficient reference to facilitate development of AVS applications on embedded system. The complexity analysis provides rough storage complexity requirements for AVS video standard implementation and optimization. [Copyright &y& Elsevier]

Abstract: Compressed video bit stream is very sensitive to transmission errors, due to error propagation resulting from variable-length coding, differential coding, and motion compensated prediction. There exist many error-resilient techniques that can significantly improve the reconstructed video quality by reducing the spatial and temporal error propagation. In this paper, an error-resilient coding scheme that can limit the spatial error propagation with little overhead is proposed. It disables intra AC prediction and groups appropriate fixed length coded (FLC) syntaxes in a video packet (or slice) together to form a new partition. With intra AC prediction disabled, errors occurred in these FLC syntaxes will not cause spatial error propagation. This error resilient coding scheme essentially classifies the syntaxes into two categories according to whether that syntax will cause spatial error propagation when an error occurs. We evaluate the efficiency of the error-resilient coding scheme by comparing the reconstructed PSNR of the proposed coding scheme with that of the MPEG-4 coding scheme over the AWGN channel and the Rayleigh channel. Depending on the video sequence, channel coding rate and channel condition, our proposed data-partitioning scheme can achieve up to 1.8dB PSNR gain. [Copyright &y& Elsevier]

Abstract: In this paper, we study joint source channel coding for bitplane based video coding over wireless channels. We consider using frame-level intra-mode to stop error propagation and using unequal error protection (UEP) to combat channel errors. Our focus is on how to optimally select coding modes and find UEP solutions for bitplane based video coding. In particular, we propose an overall end-to-end rate–distortion (R–D) function, which considers not only the source distortion and the channel distortion introduced in the current frame but also the propagated channel distortion from the previous frames. Based on this end-to-end R–D function, we are able to find the optimal solutions for both mode selection and UEP so that an optimal tradeoff between efficiency and robustness can be achieved. Experimental results demonstrate the significant performance gain. [Copyright &y& Elsevier]

Abstract: It is a challenging problem to design an efficient real-time video transport system for the present best-effort Internet because of the strict bandwidth, loss and time constraints and the lack of quality of service (QoS) guarantees from the present IP networks. This paper presents an end-to-end system architecture based on the MPEG-4 Delivery Multimedia Integration Framework (DMIF) for transporting real-time live video over the Internet. The key features of the system include: (1) MPEG-4 DMIF-based terminal architecture; (2) combination of an end-to-end feedback control mechanism and a rate-adaptive encoding algorithm for the best use of the Internet; (3) a robust and efficient packetization scheme for the MPEG-4 bit-stream by using the resynchronization marker approach specified in the MPEG-4 standard; and (4) efficient error control algorithms adopted at the end systems for visual quality enhancement. Simulation and test results using the actual Internet showed that our system is capable of utilizing the available network resource and achieve good perceptual quality at the application level. [Copyright &y& Elsevier]

Object proposals are used in two-stage detectors, such as R-CNN, to generate detection results, including category predictions and refined bounding-boxes. As a result, classification scores are assigned to refined bounding-boxes rather than object proposals. However, this procedure ignores the discrepancy of data distribution between object proposals and refined bounding-boxes. We consider this discrepancy could limit the detection accuracy. Specifically, the foreground/background imbalance on object proposals and inaccurate information from low-IoU proposals could hinder the category prediction. In this paper, we propose a detector called the Multi-Path Detection Calibration Network (PDC-Net) to address this problem. The key idea behind PDC-Net is calibrating detection results from R-CNN by considering the statistical discrepancy between object proposals and refined bounding-boxes. PDC-Net is built on Faster R-CNN. The core component in PDC-Net is the multi-path detection head, in which the base detector (from Faster R-CNN) generates detection results from object proposals and multiple calibration detectors fix incorrect outputs from the base detector using refined bounding-boxes. Experiments reveal that PDC-Net can boost detection results. Our method could reach 83.1% and 43.3% mAP respectively on PASCAL VOC and MSCOCO benchmarks, which is comparable to several state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Referring Image Segmentation (RIS) aims to extract the object or stuff from an image according to the given natural language expression. As a representative multi-modal reasoning task, the main challenge of RIS lies in accurately understanding and aligning two types of heterogeneous data (i.e. image and text). Existing methods typically conduct this task via inexplicit cross-modal fusion toward the visual and linguistic features, which are separately extracted from different encoders and hard to capture accurate image–text alignment information due to their distinct latent representation structures. In this paper, we propose a Dual-Graph Hierarchical Interaction Network (DGHIN) to facilitate the explicit and comprehensive alignment between the image and text data. Firstly, two graphs are separately built for the initial visual and linguistic features extracted with different pre-trained encoders. By means of graph reasoning, we obtain a unified representation structure for different modalities to capture the intra-modal entities and their contexts, where each projected node incorporates the long-range dependencies into the latent representation. Then, the Hierarchical Interaction Module (HIM) is applied to the visual and linguistic graphs to extract comprehensive inter-modal correlations from the entity level and graph level, which not only capture the corresponding keywords and visual patches but also draws the whole sentence closer to the image region with the consistent context in the latent space. Extensive experiments on RefCOCO, RefCOCO+, G-Ref, and ReferIt demonstrate that the proposed DGHIN outperforms many state-of-the-art methods. Code is available at https://github.com/ZhaofengSHI/referring-DGHIN. • We propose a novel "part-unity-whole" paradigm for RIS. • We develop a DGHIN consisting of the DGM and a HIM. • DGM represents the image/text with a unified node and edge structure. • HIM captures comprehensive inter-modal correlations across the entity and graph level. • Experiments show that the proposed method achieves state-of-the-art performance. [ABSTRACT FROM AUTHOR]

Removing raindrops from a single image is a challenging problem due to the complex changes in shape, scale, and transparency among raindrops. Previous explorations have mainly been limited in two ways. First, publicly available raindrop image datasets have limited capacity in terms of modeling raindrop characteristics (e.g., raindrop collision and fusion) in real-world scenes. Second, recent deraining methods tend to apply shape-invariant filters to cope with diverse rainy images and fail to remove raindrops that are especially varied in shape and scale. In this paper, we address these raindrop removal problems from two perspectives. First, we establish a large-scale dataset named RaindropCityscapes, which includes 11,583 pairs of raindrop and raindrop-free images, covering a wide variety of raindrops and background scenarios. Second, a two-branch Multi-scale Shape Adaptive Network (MSANet) is proposed to detect and remove diverse raindrops, effectively filtering the occluded raindrop regions and keeping the clean background well-preserved. Extensive experiments on synthetic and real-world datasets demonstrate that the proposed method achieves significant improvements over the recent state-of-the-art raindrop removal methods. Moreover, the extension of our method towards the rainy image segmentation and detection tasks validates the practicality of the proposed method in outdoor applications. [ABSTRACT FROM AUTHOR]

Rectified activation units make an important contribution to the success of deep neural networks in many computer vision tasks. In this paper, we propose a Parametric Deformable Exponential Linear Unit (PDELU) and theoretically verify its effectiveness for improving the convergence speed of learning procedure. By means of flexible map shape, the proposed PDELU could push the mean value of activation responses closer to zero, which ensures the steepest descent in training a deep neural network. We verify the effectiveness of the proposed method in the image classification task. Extensive experiments on three classical databases (i.e., CIFAR-10, CIFAR-100, and ImageNet-2015) indicate that the proposed method leads to higher convergence speed and better accuracy when it is embedded into different CNN architectures (i.e., NIN, ResNet, WRN, and DenseNet). Meanwhile, the proposed PDELU outperforms many existing shape-specific activation functions (i.e., Maxout, ReLU, LeakyReLU, ELU, SELU, SoftPlus, Swish) and the shape-adaptive activation functions (i.e., APL, PReLU, MPELU, FReLU). [ABSTRACT FROM AUTHOR]

Thanks to its storage and computation efficiency, hashing as a kind of nearest neighbor search method can facilitate massive data processing in recent vision and learning studies. Particularly, deep supervised hashing methods have significantly improved the retrieval performance compared with non-deep supervised hashing methods. However, most existing deep supervised hashing methods approximate the similarity between two images with the hamming distance between the outputs of the same hash function, i.e., the symmetric strategy. Consequently, it is typically time-consuming to train these symmetric hashing methods, and these methods can hardly take full advantage of the supervised information in the large-scale database. In this paper, we propose a novel discriminative deep m etric learning approach for a symmetric d iscrete h ashing (ADMH) approach for supervised hashing learning. ADMH integrates an asymmetric strategy with a deep metric learning method to learn the hash function for the query images and the discrete hash codes for database images directly. More specifically, we train a deep neural network to extract the features of the query images. Subsequently, we introduce a metric learning scheme for learning the discrete discriminative hash codes of the database images directly. Finally, the feature learning procedure for generating the hash codes of the query images and the discrete coding procedure for generating the hash codes of the database images are integrated into an end-to-end learning framework. Extensive experiments on various benchmark datasets show that the proposed asymmetric deep hashing method outperforms the existing hashing methods. [ABSTRACT FROM AUTHOR]

Head detection plays an important role in localizing and identifying persons from visual data. Most existing methods treat head detection as a specific form of object detection. Head detection is nontrivial due to the considerable difficulty in building the local and global information under conditions of unconstrained pose and orientation. To address these issues, this paper presents an effective adaptive relational network to capture context information, which is greatly helpful to suppress missed detection. We show that the fundamental contextual properties, such as the global shape priors from different heads and the local adjacent relationship between the head and shoulders, can be systematically quantified by visual operators. Specifically, we propose a two-step search algorithm to quantify the global intergroup conflict with adaptive scale, pose and viewpoint. Meanwhile, a structured feature module is introduced to capture the local relation of intraindividual stability. Finally, the global priors and local relation are integrated seamlessly into a single-stage head detector that is end-to-end trainable. An extensive ablation analysis demonstrates the effectiveness of our approach. We achieve state-of-the-art results on two challenging datasets, i.e., HollywoodHeads and Brainwash. [ABSTRACT FROM AUTHOR]

Unmanned aerial vehicles are an essential component in the realization of Industry 4.0. With drones helping to improve industrial safety and efficiency in utilities, construction, and communication, there is an urgent need for drone-based intelligent applications. In this paper, we develop a unified framework to simultaneously detect and count vehicles from drone images. We first explore why the state-of-the-art detectors fail in highly dense drone scenes, which provides more appropriate insights. Then, we propose an effective loss to push the anchors toward matching the ground-truth boxes as much as possible, specifically designed for scale-adaptive anchor generation. Inspired by attention mechanisms in the human visual system, we maximize the mutual information between object classes and features by combining bottom-up cues with top-down attention mechanisms specifically designed for feature extraction. Finally, we build a counting layer with a regularized constraint related to the number of vehicles. Extensive experiments demonstrate the effectiveness of our approach. For both tasks, our proposed method achieves state-of-the-art results on all four challenging datasets. In particular, our results reduce error by a larger factor than previous methods. [ABSTRACT FROM AUTHOR]

The ultra high-definition (UHD) video format, which has recently become popular, aims to provide high spatial resolution, high temporal frame rate, high sample bit-depth, and wide pixel color gamut. Despite the continued development of global network capacities, it inevitably causes the increased bandwidth cost of catering to the requirement of delivering UHD video services. To address such challenges, this paper presents an improved super coding unit (SCU) method for UHD video coding in High Efficiency Video Coding (HEVC). Initially, the medium coding unit (MCU) is proposed to avoid unnecessary brute-force coding unit (CU) partitions of SCU. Furthermore, the SCU is proposed to be encoded by Direct-MCU and SCU-to-MCU modes: the Direct-MCU mode is intended to better adapt to the texture-rich region, which guarantees the compression efficiency by avoiding extra-size CU partition; the SCU-to-MCU mode is designed for the homogeneous region of UHD content, which saves the encoding time by skipping fine-grained CU partition search. Moreover, a learning-based fast SCU decision approach is proposed to speed up the determination process of Direct-MCU and SCU-to-MCU, where three representative handcrafted features are extracted. Experimental results show that our method achieves an affordable complexity and excellent coding efficiency (up to 7.30% Bjøntegaard Delta rate savings) in UHD video coding compared to recent HEVC reference software. [ABSTRACT FROM AUTHOR]

In this paper, we propose a generative framework that unifies depth-based 3D facial pose tracking and face model adaptation on-the-fly, in the unconstrained scenarios with heavy occlusions and arbitrary facial expression variations. Specifically, we introduce a statistical 3D morphable model that flexibly describes the distribution of points on the surface of the face model, with an efficient switchable online adaptation that gradually captures the identity of the tracked subject and rapidly constructs a suitable face model when the subject changes. Moreover, unlike prior art that employed ICP-based facial pose estimation, to improve robustness to occlusions, we propose a ray visibility constraint that regularizes the pose based on the face model's visibility with respect to the input point cloud. Ablation studies and experimental results on Biwi and ICT-3DHP datasets demonstrate that the proposed framework is effective and outperforms completing state-of-the-art depth-based methods. [ABSTRACT FROM AUTHOR]

• A novel 3D face reconstruction framework which is robust to expression and pose changes. • A high level feature, which has robustness to lighting variations. • we provide 3DMM parameters and 3D meshes for a 3D face database. This paper proposes a novel model fitting algorithm for 3D facial expression reconstruction from a single image. Face expression reconstruction from a single image is a challenging task in computer vision. Most state-of-the-art methods fit the input image to a 3D Morphable Model (3DMM). These methods need to solve a stochastic problem and cannot deal with expression and pose variations. To solve this problem, we adopt a 3D face expression model and use a combined feature which is robust to scale, rotation and different lighting conditions. The proposed method applies a cascaded regression framework to estimate parameters for the 3DMM. 2D landmarks are detected and used to initialize the 3D shape and mapping matrices. In each iteration, residues between the current 3DMM parameters and the ground truth are estimated and then used to update the 3D shapes. The mapping matrices are also calculated based on the updated shapes and 2D landmarks. HOG features of the local patches and displacements between 3D landmark projections and 2D landmarks are exploited. Compared with existing methods, the proposed method is robust to expression and pose changes and can reconstruct higher fidelity 3D face shape. [ABSTRACT FROM AUTHOR]

The generation of object proposals plays an important role in object detection. Most existing methods produce object proposals by using bottom-up cues, such as closed contour or superpixel. In this paper, we propose a novel method to improve the ranking of object proposals by combining bottom-up cues with top-down information of objectivity. Firstly, we utilize the bottom-up method to generate initial object proposals of the given test image. Then we retrieve its top-k similar images from training images set. Considering both appearance and spatial similarity between initial object proposals and the ground truth bounding boxes of these top-k similar images, we obtain the top-down guided scores of initial object proposals. Finally, the refined score of each initial object proposal is modeled as a fusion of the bottom-up score and the top-down score. Experiments show that our method achieves better performance compared with the state-of-art on the Pascal VOC2007 dataset. [ABSTRACT FROM AUTHOR]

In this paper, a novel reduced reference (RR) stereoscopic image quality assessment (SIQA) is proposed by characterizing the statistical properties of the stereoscopic image in the reorganized discrete cosine transform (RDCT) domain. Firstly, the difference image between the left and right view images is computed. Afterwards, the left and right view images, as well as the difference image, are decomposed by block-based discrete cosine transform (DCT). The DCT coefficients are further reorganized into a three-level coefficient tree, resulting in ten RDCT subbands. For each RDCT subband, the statistical property of the coefficient distribution is modeled by the generalized Gaussian density (GGD) function. And the mutual information (MI) and energy distribution ratio (EDR) are employed to depict the statistical properties across different RDCT subbands. Moreover, EDR can further model the mutual masking property of the human visual system (HVS). By considering the GGD modeling behavior within each RDCT subband and MI together EDR characterizing behavior across RDCT subbands, the statistical properties of the stereoscopic image are fully exploited, including the left view, right view, and the difference image. Experimental results demonstrated that the statistical properties of the difference image can well represent the perceptual quality of the stereoscopic image, which outperforms the representative RR quality metrics for stereoscopic image and even some full reference (FR) quality metrics. By considering the left view, right view, and difference image together, the performances of the proposed RR SIQA can be further improved, which presenting a more closely relationship between the quality metric output and human visual perception. [ABSTRACT FROM AUTHOR]

A multi-layer authentication scheme for HEVC compressed video is proposed. The combination of CU sizes, which is unique to HEVC and sensitive to video manipulation, is considered along with other elements in the HEVC coding standard to generate the authentication tag. Temporal dependency was enforced, where the authentication tag generated in one slice is embedded into its subsequent slice. By design, the authentication tag is repeatedly but selectively embedded into various elements in a HEVC video, including nonzero DCT coefficients, QP parameter values, and prediction modes, depending on the bit segment in the generated tag. Our scheme offers three layers of authentication to detect and localize the tampered regions in a HEVC video, as well as verifying the source/sender of the video using a shared secret key. Video sequences from various classes (resolutions) are considered to verify the performance of the proposed multi-layer authentication scheme. Results show that, at the expense of slight degradation in perceptual quality, the proposed scheme is robust against several common attacks. A functional comparison is performed between the proposed multi-layer authentication scheme and the conventional schemes. [ABSTRACT FROM AUTHOR]

In this paper, we present a hybrid human recognition system for surveillance. A Cascade Head–Shoulder Detector (CHSD) with human body model is proposed to find the face region in a surveillance video frame image. The CHSD is a chain of rejecters which combines the advantages of Haar-like feature and HoG feature to make the detector more efficient and effective. For human recognition, we introduce an Overlapping Local Phase Feature (OLPF) to describe the face region, which can improve the robustness to pose change and blurring. To well model the variations of faces, an Adaptive Gaussian Mixture Model (AGMM) is presented to describe the distributions of the face images. Since AGMM does not need the facial topology, the proposed method is resistant to face detection error caused by imperfect localization or misalignment. Experimental results demonstrate the effectiveness of the proposed method in public dataset as well as real surveillance video. [ABSTRACT FROM AUTHOR]

The general purpose no reference image quality assessment (NR-IQA) is a challenging task, which faces two hurdles: (1) it is difficult to develop one quality aware feature which works well across different types of distortion and (2) it is hard to learn a regression model to approximate a complex distribution for all training samples in the feature space. In this paper, we propose a novel NR-IQA method that addresses these problems by introducing the multi-domain structural information and piecewise regression. The main motivation of our method is based on two points. Firstly, we develop a new local image representation which extracts the structural image information from both the spatial-frequency and spatial domains. This multi-domain description could better capture human vision property. By combining our local features with a complementary global feature, the discriminative power of each single feature could be further improved. Secondly, we develop an efficient piecewise regression method to capture the local distribution of the feature space. Instead of minimizing the fitting error for all training samples, we train the specific prediction model for each query image by adaptive online learning, which focuses on approximating the distribution of the current test image’s k -nearest neighbor (KNN). Experimental results on three benchmark IQA databases (i.e., LIVE II, TID2008 and CSIQ) show that the proposed method outperforms many representative NR-IQA algorithms. [ABSTRACT FROM AUTHOR]