Search

To defend against manipulation of image content, such as splicing, copy-move, and removal, we develop a Progressive Spatio-Channel Correlation Network (PSCC-Net) to detect and localize image manipulations. PSCC-Net processes the image in a two-path procedure: a top-down path that extracts local and global features and a bottom-up path that detects whether the input image is manipulated, and estimates its manipulation masks at multiple scales, where each mask is conditioned on the previous one. Different from the conventional encoder-decoder and no-pooling structures, PSCC-Net leverages features at different scales with dense cross-connections to produce manipulation masks in a coarse-to-fine fashion. Moreover, a Spatio-Channel Correlation Module (SCCM) captures both spatial and channel-wise correlations in the bottom-up path, which endows features with holistic cues, enabling the network to cope with a wide range of manipulation attacks. Thanks to the light-weight backbone and progressive mechanism, PSCC-Net can process $1,080\text{P}$ images at 50+FPS. Extensive experiments demonstrate the superiority of PSCC-Net over the state-of-the-art methods on both detection and localization. Codes and models are available at https://github.com/proteus1991/PSCC-Net. [ABSTRACT FROM AUTHOR]

The rapid progress in deep learning-based computer vision has opened unprecedented possibilities in computing various high-level analytics for sports. Artificial intelligence techniques such as predictive analysis, automatic highlight generation, and assistant coaching have been applied to improve performance and decision-making for teams and players. To perform any high-level analysis from a game match, collecting the locations (where) and identities (who) of players is crucial and challenging. In this paper, a universal JErsey number DEtector (JEDE) for player identification is presented that predicts players’ bounding boxes and keypoints, along with bounding boxes and classes of jersey digits and numbers in an end-to-end manner. Instead of generating digit proposals from pre-defined anchors, JEDE predicts more robust proposals guided by players’ features and pose estimation. Moreover, a dataset is collected from soccer and basketball matches with annotations on players’ bounding boxes and body keypoints, and jersey digits’ bounding boxes and labels. Extensive experimental results and ablation studies on the collected dataset show that the proposed method outperforms the state-of-the-art methods by a large margin. Both quantitative and qualitative results also demonstrate JEDE’s superior practicality and generalizability over different sports. [ABSTRACT FROM AUTHOR]

In this work we propose to boost video-based person re-identification (Re-ID) by using attribute-enhanced feature presentation. To this end, we not only try to use the ID-relevant attributes more effectively, but also for the first time in literature harness the ID-irrelevant attributes to help model training. The former mainly include gender, age, clothing characteristics, etc., which contain rich and supplementary information about the pedestrian; the latter include viewpoint, action, etc., which are seldom used for identification previously. In particular, we use the attributes to enhance the significant areas of the image with a novel Attribute Salient Region Enhance (ASRE) module that can attend more accurately to the body of the pedestrian, so as to better separate the target from the background. Furthermore, we find that many ID-irrelevant but subject-relevant factors, like the view angle and movement of the target pedestrian, have great impact on the two-dimensional appearance of a pedestrian. We then propose to exploit both the ID-relevant and the ID-irrelevant attributes via a novel triplet loss called the Viewpoint and Action-Invariant (VAI) triplet loss. Based on the above, we design an Attribute Salience Assisted Network (ASA-Net) to perform attribute recognition along with identity recognition, and use the attributes for feature enhancement and hard sample mining. Extensive experiments on MARS and DukeMTMC-VideoReID datasets show that our method outperforms the state-of-the-arts. Also, the visualizations of learning results further prove the effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]

Image quality assessment (IQA) is an important task of image processing and has diverse applications, such as image super-resolution reconstruction, image transmission and monitoring systems. This paper proposes a perceptual hashing algorithm with complementary color wavelet transform (CCWT) and compressed sensing (CS) for reduced-reference (RR) IQA. The CCWT is exploited to decompose input color image into different sub-bands. Since the calculation of CCWT uses all color channels without discarding any information, the distortions introduced by digital operations on color channels are preserved in the CCWT sub-bands. The block-based CS is used to extract features from the CCWT sub-bands. As the Euclidean distance between the block-based CS features is slightly influenced by content-preserving operations, perceptual features constructed by Euclidean distances are robust, discriminative and compact. Hash sequence is finally determined by quantifying the perceptual features. Effectiveness of the proposed hashing is verified by various experiments on four open image databases. Experimental results demonstrate that the proposed hashing is superior to some state-of-the-art algorithms in terms of classification and RR IQA application. [ABSTRACT FROM AUTHOR]

Visual question generation task aims to generate meaningful questions about an image targeting an answer. Existing methods focus on the visual concepts in the image for question generation. However, humans inevitably use their knowledge related to visual objects in images to construct questions. In this paper, we propose a knowledge-based visual question generation model that can integrate visual concepts and non-visual knowledge to generate questions. To obtain visual concepts, we utilize a pre-trained object detection model to obtain object-level features of each object in the image. To obtain useful non-visual knowledge, we first retrieve the knowledge from the knowledge-base related to the visual objects in the image. Considering that not all retrieved knowledge is helpful for this task, we introduce an answer-aware module to capture the candidate knowledge related to the answer from the retrieved knowledge, which ensures that the generated content can be targeted at the answer. Finally, object-level representations containing visual concepts and non-visual knowledge are sent to a decoder module to generate questions. Extensive experiments on the FVQA and KBVQA datasets show that the proposed model outperforms the state-of-the-art models. [ABSTRACT FROM AUTHOR]

Physiological studies have shown that healthy and depressed individuals present different facial changes. Thus, many researchers have attempted to use Convolutional Neural Networks (CNNs) to extract high-level facial dynamic representations for predicting depression severity. However, the max-pooling (or average-pooling) layers in the CNN lead to the loss of subtle depression cues. Without pooling layers, the CNN cannot extract multi-scale information and has difficulties for tensor vectorization. To this end, we propose a Selective Element and Two Orders Vectorization (SE-TOV) network. For the SE-TOV network, an SE block is constructed to adaptively select the effective elements from the tensors obtained by receptive fields of different sizes. Moreover, we propose a TOV block for vectorizing a high-dimensional tensor. On the one hand, TOV block inputs a tensor into the Global Average Pooling layer to obtain the first-order vectorization result. On the other hand, it takes principal components of the correlation matrix of channels in a tensor as the second-order vectorization result. Experimental results on AVEC 2013 (RMSE $=7.42$ , MAE $=6.09$) and AVEC 2014 (RMSE $=7.39$ , MAE $=5.87$) depression databases illustrate the superiority of our approach over previous works. [ABSTRACT FROM AUTHOR]

Deep neural networks have achieved great performance on blind Image Quality Assessment (IQA), but it is still challenging for using one network to accurately predict the quality of images with different distortions. In this paper, a Distortion-Aware Convolutional Neural Network (DACNN) is proposed for blind IQA, which works effectively for not only synthetically distorted images but also authentically distorted images. The proposed DACNN consists of a distortion aware module, a distortion fusion module, and a quality prediction module. In the distortion aware module, a Siamese network-based pretraining strategy is proposed to design a synthetic distortion-aware network for full learning the synthetic distortions, and an authentic distortion-aware network is used for extracting the authentic distortions. To efficiently fuse the learned distortion features, and make the network pay more attention to the essential features, a weight-adaptive fusion network is proposed to adaptively adjust the weight of each distortion. Finally, the quality prediction module is adopted to map the fused features to a quality score. Extensive experiments on four authentic IQA databases and four synthetic IQA databases have proved the effectiveness of the proposed DACNN. [ABSTRACT FROM AUTHOR]

Person search, consisting of jointly or separately trained person detection stage and person Re-ID stage, suffers from significant challenges such as inefficiency and difficulty in acquiring discriminative features. However, certain work has either turned to the end-to-end framework whose performance is limited by task conflicts or has consistently attempted to obtain more accurate bounding boxes (Bboxes). Few studies have focused on the impact of sample-specificity in person search datasets for training a fine-grain Re-ID model, and few have considered obtaining discriminative Re-ID features from Bboxes in a more efficient way. In this paper, a novel sample-enhanced and instance-sensitive (SEIE) framework is designed to boost performance. By analyzing the structure of person search framework, our method refines the two stages separately. For the detection stage, we re-design the usage of Bbox and a sample enhancement combination is proposed to further enhance the quality and quantity of Bboxes. SEC can suppress false positive detection results and randomly generate high-quality positive samples. For the Re-ID stage, we contribute an instance similarity loss to exploit the similarity between classless instances, and an Omni-scale Re-ID backbone is employed to learn more discriminative features. We obtain a more efficient and discriminative person search framework by concatenating the two stages. Extensive experiments demonstrate that our method achieves state-of-the-art performance with a high speed, and significantly outperforms other existing methods. [ABSTRACT FROM AUTHOR]

The finger vein trait has attracted widespread attention for personal authentication in recent years. However, most finger vein verification methods are performed on the single perspective, captured by a monocular near-infrared camera fixed at one side of the finger. Consequently, the contents of a single perspective have few details of the spatial network structure of the finger vein and show noticeable differences even if the posture of the same finger is slightly different. Both of them impact the verification performance. Hence, finger vein images captured from different viewpoints are considered in this work. We first design a low-cost multi-perspective based dorsal finger vein imaging device for data collection. A deep neural network named Hierarchical Content-Aware Network (HCAN) is then proposed to extract the discriminative hierarchical features of the finger vein. Specifically, HCAN is compound of a Global Stem Network (GSN) and a Local Perception Module (LPM). GSN aims to extract the latent global 3D feature from all perspectives through a recurrent neural network. It enables the model to retain the details in previous hidden states by incorporating a memory weighting strategy. LPM is designed to perceive each perspective from the aspect of image entropy. Guided by the entropy loss, LPM captures the prominent local feature and improves the discriminability and robustness of the hierarchical feature. The experimental results on the newly collected THU-MFV database demonstrate the superiority of the proposed method in comparison with other multi-perspective and single-perspective based methods. [ABSTRACT FROM AUTHOR]

2D image-based 3D shape retrieval (2D-to-3D) aims at searching the corresponding 3D shapes (unlabeled) when given a 2D image (labeled), which is a fundamental task in computer vision and has gained a surge of attention in recent years. However, extensive prior works are limited by two settings, 1) reducing domain discrepancy while ignoring the 3D shape style, 2) 3D shapes are simply and brutally pseudo-annotated by the 2D image-supervised classifier, neglecting the structure information underlying the 3D shape domain. To remedy these issues, we propose a feature transformation framework with selective pseudo-labeling (FTSPL) for 2D-to-3D task. Specifically, we first employ CNNs to produce both 2D image and 3D shape (described as multiple views) features, then we force the inter-domain centroid alignment class-wisely to reduce the overall domain discrepancy. In addition to this, we further exploit the intra-category attribute variation (covariance) of 3D shape features to transform the 2D image features. By doing so, we can equip 2D features with 3D shape style. Since the centroid and covariance estimation of 3D shape features require accurate label predictions, we put forward a selective pseudo-labeling module, which can assign reliable pseudo-labels for 3D shapes via nearest category centroid and cluster analysis, respectively, while preserving the structure information of 3D shapes. Comprehensive experiments validate that our model surpasses the state-of-the-arts on standard 2D-to-3D benchmarks (MI3DOR and MI3DOR-2). [ABSTRACT FROM AUTHOR]

We classify the discontinuity of loss in both five-param and eight-param rotated object detection methods as rotation sensitivity error (RSE) which will result in performance degeneration. We introduce a novel modulated rotation loss to alleviate the problem and a rotation sensitivity detection network (RSDet) which consists of an eight-param single-stage rotated object detector and the modulated rotation loss. Our proposed RSDet has several advantages: 1) it reformulates the rotated object detection problem as predicting the corners of objects while most previous methods employ a five-param-based regression method with different measurement units. 2) modulated rotation loss achieves consistent improvement on both five-param and eight-param rotated object detection methods by solving the discontinuity of loss. To further improve the accuracy of our method on objects smaller than 10 pixels, we introduce a novel RSDet++ which consists of a point-based anchor-free rotated object detector and a modulated rotation loss. Extensive experiments demonstrate the effectiveness of both RSDet and RSDet++, which achieve competitive results on rotated object detection in the challenging benchmarks DOTA-v1.0, DOTA-v1.5, and DOTA-v2.0. We hope the proposed method can provide a new perspective for designing algorithms to solve rotated object detection and pay more attention to tiny objects. The codes and models are available at: https://github.com/yangxue0827/RotationDetection. [ABSTRACT FROM AUTHOR]

Image-text retrieval is a fundamental and vital task in multi-media retrieval and has received growing attention since it connects heterogeneous data. Previous methods that perform well on image-text retrieval mainly focus on the interaction between image regions and text words. But these approaches lack joint exploration of characteristics and contexts of regions and words, which will cause semantic confusion of similar objects and loss of contextual understanding. To address these issues, a dual-level representation enhancement network (DREN) is proposed to strength the characteristic and contextual representations by innovative block-level and instance-level representation enhancement modules, respectively. The block-level module focuses on mining the potential relations between multiple blocks within each instance representation, while the instance-level module concentrates on learning the contextual relations between different instances. To facilitate the accurate matching of image-text pairs, we propose the graph correlation inference and weighted adaptive filtering to conduct the local and global matching between image-text pairs. Extensive experiments on two challenging datasets (i.e., Flickr30K and MSCOCO) verify the superiority of our method for image-text retrieval. [ABSTRACT FROM AUTHOR]

Human densepose estimation, aiming at establishing dense correspondences between 2D pixels of human body and 3D human body template, is a key technique in enabling machines to have an understanding of people in images. It still poses several challenges due to practical scenarios where real-world scenes are complex and only partial annotations are available, leading to incompelete or false estimations. In this work, we present a novel framework to detect the densepose of multiple people in an image. The proposed method, which we refer to Knowledge Transfer Network (KTN), tackles two main problems: 1) how to refine image representation for alleviating incomplete estimations, and 2) how to reduce false estimation caused by the low-quality training labels (i.e., limited annotations and class-imbalance labels). Unlike existing works directly propagating the pyramidal features of regions for densepose estimation, the KTN uses a refinement of pyramidal representation, where it simultaneously maintains feature resolution and suppresses background pixels, and this strategy results in a substantial increase in accuracy. Moreover, the KTN enhances the ability of 3D based body parsing with external knowledges, where it casts 2D based body parsers trained from sufficient annotations as a 3D based body parser through a structural body knowledge graph. In this way, it significantly reduces the adverse effects caused by the low-quality annotations. The effectiveness of KTN is demonstrated by its superior performance to the state-of-the-art methods on DensePose-COCO dataset. Extensive ablation studies and experimental results on representative tasks (e.g., human body segmentation, human part segmentation and keypoints detection) and two popular densepose estimation pipelines (i.e., RCNN and fully-convolutional frameworks), further indicate the generalizability of the proposed method. [ABSTRACT FROM AUTHOR]

To improve the viewer’s Quality of Experience (QoE) and optimize computer graphics applications, 3D model quality assessment (3D-QA) has become an important task in the multimedia area. Point cloud and mesh are the two most widely used digital representation formats of 3D models, the visual quality of which is quite sensitive to lossy operations like simplification and compression. Therefore, many related studies such as point cloud quality assessment (PCQA) and mesh quality assessment (MQA) have been carried out to measure the visual quality of distorted 3D models. However, most previous studies utilize full-reference (FR) metrics, which indicates they can not predict the quality level in the absence of the reference 3D model. Furthermore, few 3D-QA metrics consider color information, which significantly restricts their effectiveness and scope of application. In this paper, we propose a no-reference (NR) quality assessment metric for colored 3D models represented by both point cloud and mesh. First, we project the 3D models from 3D space into quality-related geometry and color feature domains. Then, the 3D natural scene statistics (3D-NSS) and entropy are utilized to extract quality-aware features. Finally, a support vector regression (SVR) model is employed to regress the quality-aware features into visual quality scores. Our method is validated on the colored point cloud quality assessment database (SJTU-PCQA), the Waterloo point cloud assessment database (WPC), and the colored mesh quality assessment database (CMDM). The experimental results show that the proposed method outperforms most compared NR 3D-QA metrics with competitive computational resources and greatly reduces the performance gap with the state-of-the-art FR 3D-QA metrics. The code of the proposed model is publicly available now at https://github.com/zzc-1998/NR-3DQA. [ABSTRACT FROM AUTHOR]

Multi-object tracking (MOT), which is crucial for computer vision and video processing, has immense potential for improvement. Traditional tracking-by-detection approaches include feature-based object re-identification methods that use trained features, but these methods suffer from a lack of suitable training data. In training datasets used for MOT, every object in a video sequence must have its own location and ID. However, assigning IDs to each object in every sequence is considerably labor-intensive, and hence current MOT datasets are unsuitable for training re-identification networks. To resolve this issue, this paper proposes a novel self-supervised learning method using several short videos that contain no human-added labels, based on the idea that each video is a set of temporally corresponding image frames. We then describe how to improve tracking performance using a re-identification network trained in a self-supervised manner. In addition, ablation studies were conducted in order to define the optimal parameters, such as number of clips, data augmentation, and appropriate matching algorithms. The proposed approach achieved competitive performance compared with current best-practice methods including supervised methods, achieving MOT accuracy = 62.0% and ID F1-score = 62.7% on the MOT17 benchmark. [ABSTRACT FROM AUTHOR]

Correlation filters with Convolutional Neural Networks (CNNs) features have obtained tremendous attention and success in visual tracking. However, redundant and noisy feature channels existed in CNN features may cause severe over-fitting and greatly limit the discriminative power of the tracking model. To tackle the issue, in this paper, we develop a new and effective channel-aware correlation filters (CACF) method for boosting the tracking performance. Our CACF method aims to dynamically select representative and discriminative feature channels from high-dimensional CNN features to reduce the model complexity and better distinguish the target object from the background. Moreover, the CACF model is solved by the alternating direction method of multipliers (ADMM) to learn correlation filters. By retaining reliable feature channels, our CACF tracking method can reach better generalization ability and discriminative ability to accurately localize the target object. Comprehensive experiments are conducted on challenging tracking datasets, and the experiment results prove that our CACF method obtains favorable tracking accuracy compared to several popular tracking methods. [ABSTRACT FROM AUTHOR]

In this work, we point out that the major dilemma of image aesthetics assessment (IAA) comes from the abstract nature of aesthetic labels. That is, a vast variety of distinct contents can correspond to the same aesthetic label. On the one hand, during inference, the IAA model is required to relate various distinct contents to the same aesthetic label. On the other hand, when training, it would be hard for the IAA model to learn to distinguish different contents merely with the supervision from aesthetic labels, since aesthetic labels are not directly related to any specific content. To deal with this dilemma, we propose to distill knowledge on semantic patterns for a vast variety of image contents from multiple pre-trained object classification (POC) models to an IAA model. Expecting the combination of multiple POC models can provide sufficient knowledge on various image contents, the IAA model can easier learn to relate various distinct contents to a limited number of aesthetic labels. By supervising an end-to-end single-backbone IAA model with the distilled knowledge, the performance of the IAA model is significantly improved by 4.8% in SRCC compared to the version trained only with ground-truth aesthetic labels. On specific categories of images, the SRCC improvement brought by the proposed method can achieve up to 7.2%. Peer comparison also shows that our method outperforms 10 previous IAA methods. [ABSTRACT FROM AUTHOR]

With the prevalence of thermal cameras, RGB-T multi-modal data have become more available for salient object detection (SOD) in complex scenes. Most RGB-T SOD works first individually extract RGB and thermal features from two separate encoders and directly integrate them, which pay less attention to the issue of defective modalities. However, such an indiscriminate feature extraction strategy may produce contaminated features and thus lead to poor SOD performance. To address this issue, we propose a novel CCFENet for a perspective to perform robust and accurate multi-modal expression encoding. First, we propose an essential cross-collaboration enhancement strategy (CCE), which concentrates on facilitating the interactions across the encoders and encouraging different modalities to complement each other during encoding. Such a cross-collaborative-encoder paradigm induces our network to collaboratively suppress the negative feature responses of defective modality data and effectively exploit modality-informative features. Moreover, as the network goes deeper, we embed several CCEs into the encoder, further enabling more representative and robust feature generation. Second, benefiting from the proposed robust encoding paradigm, a simple yet effective cross-scale cross-modal decoder (CCD) is designed to aggregate multi-level complementary multi-modal features, and thus encourages efficient and accurate RGB-T SOD. Extensive experiments reveal that our CCFENet outperforms the state-of-the-art models on three RGB-T datasets with a fast inference speed of 62 FPS. In addition, the advantages of our approach in complex scenarios (e.g., bad weather, motion blur, etc.) and RGB-D SOD further verify its robustness and generality. The source code will be publicly available via our project page: https://git.openi.org.cn/OpenVision/CCFENet. [ABSTRACT FROM AUTHOR]

Image captioning has made significant achievement through developing feature extractor and model architecture. Recently, the image region features extracted by object detector prevail in most existing models. However, region features are criticized for the lacking of background and full contextual information. This problem can be remedied by providing some complementary visual information from patch features. In this paper, we propose a Double-Stream Position Learning Transformer Network (DSPLTN) which exploits the advantages of region features and patch features. Specifically, the region-stream encoder utilizes a Transformer encoder with Relative Position Learning (RPL) module to enhance the representations of region features through modeling the relationships between regions and positions respectively. As for the patch-stream encoder, we introduce convolutional neural network into the vanilla Transformer encoder and propose a novel Convolutional Position Learning (CPL) module to encode the position relationships between patches. CPL improves the ability of relationship modeling by combining the position and visual content of patches. Incorporating CPL into the Transformer encoder can synthesize the benefits of convolution in local relation modeling and self-attention in global feature fusion, thereby compensating for the information loss caused by the flattening operation of 2D feature maps to 1D patches. Furthermore, an Adaptive Fusion Attention (AFA) mechanism is proposed to balance the contribution of enhanced region and patch features. Extensive experiments on MSCOCO demonstrate the effectiveness of the double-stream encoder and CPL, and show the superior performance of DSPLTN. [ABSTRACT FROM AUTHOR]

Video object detection (VOD) focuses on detecting objects for each frame in a video, which is a challenging task due to appearance deterioration in certain video frames. Recent works usually distill crucial information from multiple support frames to improve the reference features, but they only perform at frame level or proposal level that cannot integrate spatial-temporal features sufficiently. To deal with this challenge, we treat VOD as a spatial-temporal hierarchical features interacting process and introduce a Multi-level Spatial-Temporal (MST) feature aggregation framework to fully exploit frame-level, proposal-level, and instance-level information in a unified framework. Specifically, MST first measures context similarity in pixel space to enhance all frame-level features rather than only update reference features. The proposal-level feature aggregation then models object relation to augment reference object proposals. Furthermore, to filter out irrelevant information from other classes and backgrounds, we introduce an instance ID constraint to boost instance-level features by leveraging support object proposal features that belong to the same object. Besides, we propose a Deformable Feature Alignment (DAlign) module before MST to achieve a more accurate pixel-level spatial alignment for better feature aggregation. Extensive experiments are conducted on ImageNet VID and UAVDT datasets that demonstrate the superiority of our method over state-of-the-art (SOTA) methods. Our method achieves 83.3% and 62.1% with ResNet-101 on two datasets, outperforming SOTA MEGA by 0.4% and 2.7%. [ABSTRACT FROM AUTHOR]

During the annotation procedure of salient object detection, researchers usually locate the approximate location of the salient objects first and then process the pixels that need to be finely annotated. Following this idea, we find that the existing methods have limited exploration for solving the problem of positioning salient objects. Furthermore, no effective solution has been proposed for the hard-sample problem related to this task. Therefore, we propose dynamic scale-aware learning to learn dynamic scale weights that vary with different images to solve the first problem. Second, we design a dense sampling strategy for hard samples to construct a graph representation with samples from different classes and different confidence levels. Then, we achieve targeted feature aggregation based on the constructed graph with the help of the graph attention mechanism. We conduct extensive experiments on five benchmark datasets using comprehensive evaluation metrics. The results show that our method outperforms the current state-of-the-art approaches. [ABSTRACT FROM AUTHOR]

This paper introduces deformable convolution in deep learning based joint denoising and demosaicing (JDD), which yields more adaptable representation and larger receptive fields in features extraction for a superior restoration performance. However, the deformable convolution generally leads to considerable computational load and irregular memory access bottleneck, limiting its extensive deployment on edge devices. To address this issue, we develop grouping strategy and assign independent offsets to each kernel group to reduce the computation latency while keeping the accuracy. Motivated by the exploration for aggregate distribution characteristics of deformable offsets, we present the offset sharing methodology to simplify the memory access complexity of deformable convolution. As for hardware acceleration, we specially design a novel deformable matrix multiplication workflow incorporated with a deformable memory mapping unit to boost the computational throughput. The verification experiments on FPGA demonstrate that the proposed deformable convolution based JDD can restore 4K Ultra High Definition (UHD) images at 70FPS and yields significant promotion in visual effect and objective quality assessment. [ABSTRACT FROM AUTHOR]

The goal of video-based person re-identification (Re-ID) is to identify the same person across multiple non-overlapping cameras. The key to accomplishing this challenging task is to sufficiently exploit both spatial and temporal cues in video sequences. However, most current methods are incapable of accurately locating semantic regions or efficiently filtering discriminative spatio-temporal features; so it is difficult to handle issues such as spatial misalignment and occlusion. Thus, we propose a novel feature aggregation framework, multi-task and multi-granularity aggregation with global-guided attention (MMA-GGA), which aims to adaptively generate more representative spatio-temporal aggregation features. Specifically, we develop a multi-task multi-granularity aggregation (MMA) module to extract features at different locations and scales to identify key semantic-aware regions that are robust to spatial misalignment. Then, to determine the importance of the multi-granular semantic information, we propose a global-guided attention (GGA) mechanism to learn weights based on the global features of the video sequence, allowing our framework to identify stable local features while ignoring occlusions. Therefore, the MMA-GGA framework can efficiently and effectively capture more robust and representative features. Extensive experiments on four benchmark datasets demonstrate that our MMA-GGA framework outperforms current state-of-the-art methods. In particular, our method achieves a rank-1 accuracy of 91.0% on the MARS dataset, the most widely used database, significantly outperforming existing methods. [ABSTRACT FROM AUTHOR]

The stable continuation of trajectories among different targets has always been the key to the tracking performance of multi-object tracking (MOT) tasks. If features of the target are aggregated and classified simply, the discriminant features of the target will be ignored. This will affect the robustness of the trajectory generated by the model. Meanwhile, many popular models are keen to execute detection and feature extraction tasks in parallel. But these two tasks will conflict with each other when optimized respectively. Therefore, we propose our tracker MOTFR to solve the above problems. In this paper, we propose a Locally Shared Information Decoupling Module (LSIDM) to reduce task optimization conflicts while ensuring the necessary information sharing. Meanwhile, a feature recoding module for deep extraction of identity discriminative features is proposed, which is called the Feature Purification Module (FPM). By combining LSIDM and FPM modules, the model utilizes the discriminative appearance features to guide the optimization of detection and further improves the performance of our model. To solve the problem of targets disappearing due to various abnormal occlusion, a Short-term Trajectory Online Complement Strategy (STOCS) is proposed to realize the trajectories continuation of these targets in the tracking stage. Through sufficient experiments, we demonstrate the superior performance of our MOTFR, which guarantees high-quality detection while achieving the stability of the target trajectory. [ABSTRACT FROM AUTHOR]

With the boom of deep neural networks, blind image quality assessment (BIQA) has achieved great processes. However, the current BIQA metrics are limited when evaluating low-quality images as compared to medium-quality and high-quality images, which restricts their applications in real world problems. In this paper, we first identify that two challenges caused by distribution shift and long-tailed distribution lead to the compromised performance on low-quality images. Then, we propose an intermediary enhancement-based bilateral network with iterative training strategy for solving these two challenges. Drawing on the experience of transitive transfer learning, the proposed metric adaptively introduces enhanced intermediary images to transfer more information to low-quality images for mitigating the distribution shift. Our metric also adopts an iterative training strategy to deal with the long-tailed distribution. This strategy decouples feature extraction and score regression for better representation learning and regressor training. It not only transfers the knowledge learned from the earlier stage to the latter stage, but also makes the model pay more attention to long-tailed low-quality images. We conduct extensive experiments on five authentically distorted image quality datasets. The results show that our metric significantly improves the evaluating performance on low-quality images and delivers state-of-the-art intra-dataset results. During generalization tests, our metric also achieves the best cross-dataset performance. [ABSTRACT FROM AUTHOR]

Tracking the 6-degree-of-freedom (6D) object pose in video sequences is gaining attention because it has a wide application in multimedia and robotic manipulation. However, current methods often perform poorly in challenging scenes, such as incorrect initial pose, sudden re-orientation, and severe occlusion. In contrast, we present a robust 6D object pose tracking method with a novel hierarchical feature fusion network, refer it as HFF6D, which aims to predict the object’s relative pose between adjacent frames. Instead of extracting features from adjacent frames separately, HFF6D establishes sufficient spatial-temporal information interaction between adjacent frames. In addition, we propose a novel subtraction feature fusion (SFF) module with attention mechanism to leverage feature subtraction during feature fusion. It explicitly highlights the feature differences between adjacent frames, thus improving the robustness of relative pose estimation in challenging scenes. Besides, we leverage data augmentation technology to make HFF6D be used more effectively in the real world by training only with synthetic data, thereby reducing manual effort in data annotation. We evaluate HFF6D on the well-known YCB-Video and YCBInEOAT datasets. Quantitative and qualitative results demonstrate that HFF6D outperforms state-of-the-art (SOTA) methods in both accuracy and efficiency. Moreover, it is also proved to achieve high-robustness tracking under the above-mentioned challenging scenes. [ABSTRACT FROM AUTHOR]

In this paper, we conduct an extensive study on the use of pre-trained convolutional neural networks (CNNs) for omnidirectional image quality assessment (IQA). To cope with the lack of available IQA databases, transfer learning from seven pre-trained CNN models is investigated over retraining on standard 2D databases. In addition, we explore the influence of various image representations and training strategies on the model’s performance. A comparison of the use of projected versus radial content, and multichannel CNN versus patch-wise training is also covered. The experimental results on two publicly available databases are used to draw conclusions about which strategy best fits the visual quality prediction and at which computational cost. The analysis shows that retraining CNN models on 2D IQA databases improves the prediction accuracy. The latter and the required computational time are found to be significantly affected by the training strategy. Cross-database evaluations demonstrate that the nature and variety of the content impact the generalization ability of the models. Finally, we show that conclusions coming from other image processing communities may not hold for IQA. The provided discussion shall provide insights and recommendations when using pre-trained CNNs for omnidirectional IQA. [ABSTRACT FROM AUTHOR]

The intrusion of foreign objects in tracks, one of the sources of injuries and fatalities in metro stations, can be solved as an anomaly detection task. However, the existing anomaly detection methods rarely consider the importance of updating the new knowledge from false alarm data, resulting in repeated mistakes. These methods are also impractical for edge devices that cannot afford the high calculation cost. A sequential updatable anomaly detection (SUAD) framework is proposed to tackle these problems. This framework is based on the Robbin-Monro algorithm and a fast version of Mahalanobis distance. A well-trained model of SUAD can continue to learn new knowledge through the sequential knowledge update module based on the Robbin-Monro algorithm without reviewing the old data. SUAD utilizes a new Mahalanobis distance calculation method based on principal component analysis. This new method exhibits a fast inference speed with a lighter model size than before. SUAD is evaluated on a self-built Metro Anomaly Detection (MAD) dataset and three public datasets. SUAD achieves an average area under the receiver operating characteristic curve score of 99.4% at image level and 99.6% at pixel level on MAD. SUAD also reduces at least 78% model size and 60% memory usage. Competitive results are also achieved in public datasets, including MVTec AD, beanTech Anomaly Detection, and CIFAR-10. [ABSTRACT FROM AUTHOR]

Learning the human depth localization in camera coordinate space plays a crucial role in understanding the behavior and activities of multi-person in 3D scenes. However, existing monocular-based methods rarely combine the global image features and the human body-parts features effectively, resulting in a large gap from the actual location in some cases, e.g., the special body-sized persons and mutual occlusion between humans in the image. This paper presents a novel Robust 3D Human Localization (R3HL) network consisting of two stages: global depth awareness and body-parts depth awareness, to significantly improve the robustness and accuracy of the 3D location. In the first stage, the front-back and far-near relationship estimation module based on multi-person are proposed to make the network extract depth features from the global perspective. In the second stage, the network focuses on the target human. We propose a Pose-guided Multi-person Repulsion (PMR) module to enhance the target human’s features and reduce the interference features produced by the background and other people. In addition, an Adaptive Body-parts Attention (ABA) module is designed to assign different feature weights to each joint. Finally, the human’s absolute depth is obtained through global pooling and fully connected layers. The experimental results show that the attention from the whole image to a single person helps find the absolute location of different body-sized and poses people from diverse scenes. Our method can achieve better performance than other state-of-the-art methods on both indoor and outdoor 3D multi-person datasets. [ABSTRACT FROM AUTHOR]

RGB-D Salient Object Detection (RGB-D SOD) aims at detecting remarkable objects by complementary information from RGB images and depth cues. Although many outstanding prior arts have been proposed for RGB-D SOD, most of them focus on performance enhancement, while lacking concern about practical deployment on mobile devices. In this paper, we propose mobile asymmetric dual-stream networks (MoADNet) for real-time and lightweight RGB-D SOD. First, inspired by the intrinsic discrepancy between RGB and depth modalities, we observe that depth maps can be represented by fewer channels than RGB images. Thus, we design asymmetric dual-stream encoders based on MobileNetV3. Second, we develop an inverted bottleneck cross-modality fusion (IBCMF) module to fuse multimodality features, which adopts an inverted bottleneck structure to compensate for the information loss in the lightweight backbones. Third, we present an adaptive atrous spatial pyramid (A2SP) module to speed up the inference, while maintaining the performance by appropriately selecting multiscale features in the decoder. Extensive experiments are conducted to compare our method with 15 state-of-the-art approaches. Our MoADNet obtains competitive results on five benchmark datasets under four evaluation metrics. For efficiency analysis, the proposed method significantly outperforms other baselines by a large margin. The MoADNet only contains 5.03 M parameters and runs 80 FPS when testing a $256\times 256$ image on a single NVIDIA 2080Ti GPU. [ABSTRACT FROM AUTHOR]

Face Anti-Spoofing (FAS) plays an increasingly important role in face recognition systems for preventing malicious attacks. Existing FAS methods usually show poor generalization performance due to the large difference of domain information (such as collection environment, collection equipment, etc.) between the training set and the testing set. Therefore, we consider training a FAS network that does not pay attention to domain information, in this way to extract relevant liveness feature for the task of FAS. Learned from adversarial learning and disentanglement learning, we design the Domain Adversarial Disentanglement Network with Cross-Domain Synthesis (DADN-CDS) for Face Anti-Spoofing, which achieves disentanglement between domain-irrelevant liveness feature and domain-related feature, so as to minimize the effect of domain information in the needed representation that is used for inference. Specifically, DADN-CDS proposes a dual-branch architecture that can explicitly model and separate out the domain information. To achieve targeted optimization for each component, a novel Task-oriented Three-step Update Strategy (TTUS) is designed to explore a better model update method. Irrelevant loss and 2N-Pair Cross-Domain Loss in TTUS further ensure the disentanglement and task-oriented optimization. Moreover, an Attention-based Cross Synthesis Module (ACSM) is elaborately devised to obtain higher-quality synthetic feature, which performs attention-based feature fusion in a channel-wise way. The design of ACSM can help verify and implicitly facilitate the disentanglement process. Extensive experiments demonstrate that our method achieves the state-of-the-art performance on public datasets and results also suggest the generalization ability of our proposed method. [ABSTRACT FROM AUTHOR]

Unsupervised domain adaptation crowd counting (UDACC) has been studied with practical research utility by getting rid of the labeling burden on large-scale dense crowds in the target domain. Current methods generalize well within the specific domain gap by directly aligning domain distributions or translating synthetic data to realistic images. However, it is difficult to define domain gaps among complex real-world datasets, in which the images vary greatly in style, density level and/or content. To tackle this problem, in this paper, we propose a Cross-Domain Attention Network (CDANet), which can effectively generalize the model to the unlabeled domain on both unsupervised synthetic-to-realistic and realistic-to-realistic crowd counting. Specifically, we propose a Cross-Domain Attention Module (CDAM) to learn domain-related information between the source and target domain, which extracts relations in cross-domain attentive information, thus enhancing crowd-informative features. Moreover, to make our CDAM invariant to domain shifts, we introduce a consistency penalty to ensure that the attention maps are consistent before and after the domain shifting. Thus our CDANet can pay attention to the shared counting information across domains, while remaining its invariant ability during domain adaptation. Extensive experiments on several common benchmarks for UDACC demonstrate that our CDANet gets competitive results on both unsupervised synthetic-to-realistic and realistic-to-realistic UDACC tasks. [ABSTRACT FROM AUTHOR]

With the widespread usage of video capture devices and social media videos, videos are dominating the multimedia landscape. There is an emerging need for video quality assessment (VQA) that forms the backbone of advanced video systems. Night-time videos play an important role in user capturing, hence being able to accurately assess their quality is critical. However, the characteristics of night-time videos differ from those of general in-capture videos; and VQA algorithms that have been developed for general-purpose videos cannot accurately assess the quality of night-time videos. Research is needed to gain a better understanding of how humans perceive the quality of night-time videos, and use this new understanding to develop reliable VQA algorithms. To this end, we construct a large-scale night-time VQA database, namely Mobile In-capture Night-time Database for Video Quality (MIND-VQ), containing 1181 night-time videos, 435 subjects, and over 130000 opinion scores. We perform thorough analyses to reveal subjective quality assessment behaviors of night-time videos. Furthermore, we propose a new VQA model, namely Visibility-based Night-time Video Quality Assessment Network, VINIA. Spatial and temporal visibility-aware components are characterized to reflect properties of human perception of night-time VQA task. A series of experiments are conducted to compare our VINIA with other existing VQA algorithms using our new MIND-VQ database and other public VQA databases. Experimental results show that our subjective VQA database provides new insights and our new VINIA model achieves superior performance in accessing night-time video quality. [ABSTRACT FROM AUTHOR]

Person search aims to localize and identify specific pedestrians from numerous surveillance scene images. In this work, we focus on the noise in person search. We categorize the noise into scene-inherent noise and human-introduced noise. Scene-inherent noise comes from congestion, occlusion, and illumination changes. Human-introduced noise originates from the labeling process. For scene-inherent noise, we propose a novel context contrastive loss to take advantage of the latent contextual information from scene images. Features from context regions are utilized to construct contrastive pairs to constrain the feature discrimination among pedestrians in scene images while maintaining the feature consistency of the same identity. The network can thus learn to distinguish congested and overlapped pedestrians and more robust features can be obtained. For human-introduced noise, we propose a noise-discovery and noise-suppression training process for mislabeling robust person search. After the first training pass, the relation between feature prototypes of different identities is analyzed and the mislabeled pedestrians are discovered. During the second training pass, the label noise is suppressed to reduce the negative influence of mislabeled data. Experiments show that the proposed context-aware noise-robust (CANR) person search can achieve competitive performance. Further ablation studies confirm the effectiveness of CANR. [ABSTRACT FROM AUTHOR]

As a task of cross-modal retrieval between long videos and paragraphs, video-paragraph retrieval is a non-trivial task. Unlike traditional video-text retrieval, the video in video-paragraph retrieval usually contains multiple clips. Each clip corresponds to a descriptive sentence; all the sentences constitute the corresponding paragraph of the video. Previous methods for video-paragraph retrieval usually encode videos and para-graphs from segment-level (clips and sentences) and overall-level (videos and paragraphs). However, there are also contents about actions and objects that exist in the segment. Hence, we propose a Local-Global Graph Pooling Network (LGGP) via Mutual Information Maximization for video-paragraph retrieval. Our model disentangles videos and paragraphs into four levels: overall-level, segment-level, motion-level, and object-level. We construct the Hierarchical Local Graph (segment-level, motion-level, and object-level) and the Hierarchical Global Graph (overall-level, segment-level, motion-level, and object-level), respectively, for semantic interaction among different levels. Meanwhile, to obtain hierarchical pooling features with fine-grained semantic information, we design hierarchical graph pooling methods to maximize the mutual information between pooling features and corresponding graph nodes. We evaluate our model on two video-paragraph retrieval datasets with three different video features. The experimental results show that our model establishes state-of-the-art results for video-paragraph retrieval. Our code will be released at https://github.com/PengchengZhang1997/LGGP. [ABSTRACT FROM AUTHOR]

Camouflaged object detection (COD) aims to identify the objects that conceal themselves in natural scenes. Accurate COD suffers from a number of challenges associated with low boundary contrast and the large variation of object appearances, e.g., object size and shape. To address these challenges, we propose a novel Context-aware Cross-level Fusion Network ($\text{C}^{2}\text{F}$ -Net), which fuses context-aware cross-level features for accurately identifying camouflaged objects. Specifically, we compute informative attention coefficients from multi-level features with our Attention-induced Cross-level Fusion Module (ACFM), which further integrates the features under the guidance of attention coefficients. We then propose a Dual-branch Global Context Module (DGCM) to refine the fused features for informative feature representations by exploiting rich global context information. Multiple ACFMs and DGCMs are integrated in a cascaded manner for generating a coarse prediction from high-level features. The coarse prediction acts as an attention map to refine the low-level features before passing them to our Camouflage Inference Module (CIM) to generate the final prediction. We perform extensive experiments on three widely used benchmark datasets and compare $\text{C}^{2}\text{F}$ -Net with state-of-the-art (SOTA) models. The results show that $\text{C}^{2}\text{F}$ -Net is an effective COD model and outperforms SOTA models remarkably. Further, an evaluation on polyp segmentation datasets demonstrates the promising potentials of our $\text{C}^{2}\text{F}$ -Net in COD downstream applications. Our code is publicly available at: https://github.com/Ben57882/C2FNet-TSCVT [ABSTRACT FROM AUTHOR]

Deep learning-based image inpainting has been widely studied, leading to great success. However, many methods adopt convolution and normalization operations, which will bring up some issues to affect the performance. The vanilla normalization cannot distinguish the pixels in corrupted regions from the other valid pixels, resulting in the mean and variance shifts. In addition, the limited receptive field of convolution makes it unable to capture long-range valid information directly. In order to tackle these challenges, we propose a novel deep generative model for image inpainting with two key modules, namely, the channel and spatially adaptive batch normalization (CSA-BN) module, and the selective latent-space-mapping-based contextual attention (SLSM-CA) layer. We replace the vanilla normalization with the CSA-BN module. By channel and spatially adaptive denormalization, the CSA-BN module can mitigate the spatial mean and variance shifts in each channel in a targeted way. In addition, we also integrate the SLSM-CA layer into our model to capture the long-range correlations explicitly. By introducing dual-branch attention and a feature selection module, the SLSM-CA layer can selectively utilize the multi-scale background information to improve prediction quality. What’s more, it introduces the latent spaces to achieve the low-rank approximations of attention matrices and to reduce computational costs. Extensive quantitative and qualitative evaluations demonstrate the superiority of the proposed method compared with state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Aiming at degrading the capability of the existing forensic methods in discriminating computer generated and natural facial images, a bidirectional conversion between computer-generated and natural facial images based on generative adversarial network (BDC-GAN) is proposed for anti-forensics in this paper. The generator of BDC-GAN is composed of noise encoding and content encoding. In the noise encoding, three high-pass filters are first utilized to extract the sensor pattern noise of the image, and then the stacked convolution layer is combined to continue encoding. In the content encoding, VGG-19 is truncated and fine-tuned to encode the content of the image. Some stacked convolution layers and adaptive instance normalization layer are used in the decoder. The discriminator uses multi-scale image discriminator. Furthermore, content loss and noise loss are well designed, and hyperparameters are reasonably set to accomplish the bidirectional conversion between two domain images meanwhile retaining the original facial contour. Experimental results and analysis demonstrate that the proposed anti-forensic method can achieve better visual quality and stronger deception ability compared with the existing unidirectional CG facial image anti-forensic methods and bidirectional domain adaptive methods, and its effectiveness is verified by the tests on the existing 9 forensic methods. It reveals that the existing forensic techniques can be bypassed by using adversarial learning, and it will eventually push the performance improvement of the discrimination of computer generated and natural facial images. [ABSTRACT FROM AUTHOR]

Few-shot action recognition classifies new actions with only few training samples, of which the mainstream methods adopt class means to obtain prototypes as the representations of each category. However, affected by sample capacity and extreme samples, mean-of-class prototypes can’t well represent the average level of samples. In this paper, we enhance the prototypes from multiple dimensions for better classification. We firstly propose a novel similarity optimization mechanism where Prototype Aggregation Adaptive Loss (PAAL) is designed to deeply mine the similarity between samples and prototypes for enhancing the ability of inter-class differential detail identification. Secondly, for mitigating the impact of the samples on class prototypes, we refactor the prototype calculation formula with Cross-Enhanced Prototype (CEP) to narrow intra-class differences in which Reweighted Similarity Attention (RSA) is designed to update prototypes. Finally, Dynamic Temporal Transformation (DTT) is proposed to alleviate inconsistent distribution of temporal information for obtaining better video-level descriptors. Extensive experiments on standard benchmark datasets demonstrate that our proposed method achieves the state-of-the-art results. [ABSTRACT FROM AUTHOR]

The neural network-based remote sensing image change detection method faces a large amount of imaging interference and severe class imbalance problems under high-resolution conditions, which bring new challenges to the accuracy of the detection network. In this work, to address the imaging interference caused by different imaging angles and times, the siamese strategy and multi-head self-attention mechanism are used to reduce the imaging differences between the dual-temporal images and fully exploit the inter-temporal information. Secondly, a learnable multi-part feature learning module is used to adaptively exploit features from different scales to obtain more comprehensive features. Finally, a mixed loss function strategy is used to ensure that the network converges effectively and excludes the adverse interference of a large number of negative samples to the network. Extensive experiments show that our method outperforms numerous methods on LEVIR-CD, WHU, and DSIFN datasets. [ABSTRACT FROM AUTHOR]

Gait recognition can be used in person identification and re-identification by itself or in conjunction with other biometrics. Although gait has both spatial and temporal attributes, and it has been observed that decoupling spatial feature and temporal feature can better exploit the gait feature on the fine-grained level. However, the spatial-temporal correlations of gait video signals are also lost in the decoupling process. Direct 3D convolution approaches can retain such correlations, but they also introduce unnecessary interferences. Instead of common 3D convolution solutions, this paper proposes an integration of decoupling process into a 3D convolution framework for cross-view gait recognition. In particular, a novel block consisting of a Parallel-insight Convolution layer integrated with a Spatial-Temporal Dual-Attention (STDA) unit is proposed as the basic block for global spatial-temporal information extraction. Under the guidance of the STDA unit, this block can well integrate spatial-temporal information extracted by two decoupled models and at the same time retain the spatial-temporal correlations. In addition, a Multi-Scale Salient Feature Extractor is proposed to further exploit the fine-grained features through context awareness extension of part-based features and adaptively aggregating the spatial features. Extensive experiments on three popular gait datasets, namely CASIA-B, OULP and OUMVLP, demonstrate that the proposed method outperforms state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Learning to generate a task-aware base learner proves a promising direction to deal with few-shot learning (FSL) problem. Existing methods mainly focus on generating an embedding model utilized with a fixed metric (e.g., cosine distance) for nearest neighbour classification or directly generating a linear classifier. However, due to the limited discriminative capacity of such a simple metric or classifier, these methods fail to generalize to challenging cases appropriately. To mitigate this problem, we present a novel deep metric meta-generation method that turns to an orthogonal direction, i.e., learning to adaptively generate a specific metric for a new FSL task based on the task description (e.g., a few labelled samples). In this study, we structure the metric using a three-layers deep attentive network that is flexible enough to produce a discriminative metric for each task. Moreover, different from existing methods that utilize an uni-modal weight distribution conditioned on labelled samples for network generation, the proposed meta-learner establishes a multi-modal weight distribution conditioned on cross-class sample pairs using a tailored variational autoencoder, which can separately capture the specific inter-class discrepancy statistics for each class and jointly embed the statistics for all classes into metric generation. By doing this, the generated metric can be appropriately adapted to a new FSL task with pleasing generalization performance. To demonstrate this, we test the proposed method on three benchmark FSL datasets and gain competitive results with state-of-the-art competitors. [ABSTRACT FROM AUTHOR]

Palmprint recognition is one of the most popular biometric technologies. Recent researches mainly focus on the recognition performance, while pay less attention to the data protection issues. In this paper, we propose an active biometric data protection model for securing palmprint images in transmission or storage scenarios, called Dynamic Random Invisible Watermark Embedding (DRIWE) model. The DRIWE model implicitly embeds a watermark in each original palmprint ROI image, and then separates the embedded watermark from the watermarked image before identification. If the separated watermark is consistent with the original watermark, it indicates that the image is trustworthy and can be used in the subsequent recognition process. Otherwise, it proves that the image has been illegally tampered with. Furthermore, a two-dimensional image information entropy loss is proposed to enhance the generalization of the model to different watermarks. It ensures that the model can always assign enough information to the host image (i.e., original palmprint image) when different watermarks are applied. Thus, it enables the separator to extract the complete watermark from the watermarked image. This greatly enhances the dynamics and randomness of the watermark embedding process and further improves the ability to secure the data. Adequate experiments are conducted on two benchmark palmprint databases. The results show that the proposed DRIWE model has satisfactory attack resistance and strong generalization ability: even if only one watermark is used in training stage, it can be generalized to a dozen of other new watermark images in the testing stage. In addition, the optimal accuracy of the watermarked data is only reduced by 0.07% compared with the original data. [ABSTRACT FROM AUTHOR]

RGB-Infrared cross-modal person re-identification (Re-ID) has drawn increasing attention due to its application value in practice. Most of the current works rely on a supervised training manner. However, in real-world applications, manual collection of pair-wise RGB-Infrared (IR) person data is labor-intensive and time-consuming. Moreover, when a trained model is directly used in another domain, there is usually a significant performance drop. To overcome the above problems, we make the first attempt to transfer the learned model to a new RGB-IR domain which is unlabeled. The practical problem covers two kinds of challenges, i.e., cross-modal (RGB-Infrared) and cross-domain (different dataset) person Re-ID. Previous works have often considered only one of them either cross-modal or cross-domain. In this work, we propose a dual alignment network (DAN) to solve the RGB-Infrared cross-modal cross-domain person Re-ID problem. This network consists of three parts: Domain Adversarial Alignment component (DAA), Pseudo Label Generation module for target domain (PLG), and Cross-Modal Alignment component (CMA). These three modules complement and promote the model to learn domain-invariant and modality-invariant person representations. Further, we propose a protocol of cross-modal cross-domain person Re-ID by synthesizing target domains by adding random noise, adjusting the lighting intensity, and changing the background color, respectively. Experiments on real and synthetic datasets under the same cross-modalities across domains demonstrate the effectiveness of our method. [ABSTRACT FROM AUTHOR]

Feature Pyramid (FP) is typically a fundamental component for detecting multi-scale objects. However, as the network deepens, FP faces two problems: (1) Information loss caused by channel reduction. (2) The insufficient effective receptive field due to convolution with the sliding window mode. We found that the above problems can be alleviated by increasing the semantics extraction weights of the off-centered feature map. In this paper, a new feature pyramid architecture named Off-Centered Semantics Refinement Feature Pyramid Network (OSR-FPN) is proposed. Specifically, OSR-FPN contains two components exploiting the Off-Centered Semantics Refinement (OSR) mechanism: Features Supplement Module (FSM) and Receptive Field Enlargement Module (RFEM). FSM and RFEM are respectively designed to complement the lost context at the highest pyramid level and enrich the semantics by expanding the receptive field. In addition, we propose the Sigmoid-interpolation Padding method to enhance our OSR. Experiments on MS COCO dataset and UAVDT object detection benchmarks demonstrate the effectiveness of our method. As a result, OSR-FPN achieves a better accuracy of complex object detection. [ABSTRACT FROM AUTHOR]

Crowd counting is a task of intelligent applications, and its operation efficiency is very important. However, in order to obtain a better counting performance, most of the existing works often design larger and more complex network structures, which will result in them occupying considerable memory, time and other resources at runtime, seriously limiting their deployment scope and making it difficult to be widely used in practical scenarios. In this paper, in order to overcome the above problems, we propose an effective lightweight encoding-decoding crowd counting network, named Lw-Count. Specifically, in the encoding process, we design an efficient and lightweight convolution module (ELCM), which extracts the crowd features in the network through a refined ghost block to reduce the network parameters and computing cost, and solves the problem of inaccurate counting caused by uneven crowd distribution through spatial group normalization (SGN). In the decoding process, we design a scale regression module (SRM) to reduce the error details and chessboard effect caused by linear interpolation and deconvolution. In addition, we design a new loss function, which enhances the spatial correlation of the density map and the sensitivity of the crowd through a regional normalized cross-correlation loss and counting loss, to ensure the counting accuracy. Extensive experiments on five mainstream datasets demonstrate that Lw-Count achieves an optimal trade-off between counting performance and running speed compared with other methods. [ABSTRACT FROM AUTHOR]

Thanks to the powerful learning capability, deep neural networks (DNNs) have acquired broad applications in single image reflection removal. The DNN-based algorithms relax the constraints of specific priors and learn to generate visually pleasant background layers from massive training data. However, most of them employ a single network structure to recover both the semantic information and local details of the background, which may lead to obvious reflection residue or even failure. To mitigate this deficiency, in this work, we propose a Multi-stage Curvature-guided De-Reflection Network (MCDRNet), which combines multiple network architectures in a unified framework to progressively reconstruct the background layer and refine the fine-grained details. Our framework consists of three stages, where the encoder-decoders are exploited in the first two stages to recover the semantic components of background layers with lower scales and a variant ResNet is applied in the last stage to refine the background details with the original input resolution. In the first two stages, to introduce the structural guidance for the reflection removal, we cascade another decoder branch to restore the curvature map of the background. In addition, at the end of the first two stages, instead of directly passing the intermediate estimates to the next stage, we propose a Non-local Attention Module (NAM) to augment and transmit the features from decoders. Extensive experimental results on several public datasets demonstrate that the proposed MCDRNet outperforms the state-of-the-art methods quantitatively and generates visually better reflection removal results. The source code and pre-trained models are available at https://github.com/NamecantbeNULL/MCDRNet. [ABSTRACT FROM AUTHOR]

Modeling 3D visual saliency has received great attention due to the development of emerging 3D display technologies. Traditional methods relying on low-level features may not be efficient in interpreting 3D visual content from high-level semantic perspective. Despite numerous efforts dedicated to this area, existing 3D visual saliency detection methods do not necessarily excel in exploring the stereoscopic image saliency driven by the intra-view and inter-view dependencies among left and right views. In this paper, we propose a visual saliency detection method for stereoscopic images grounded on adaptive viewpoint feature enhancement via binocular vision. More specifically, the correlation among left and right views is investigated through a delicately designed binocular stereoscopic saliency feature aggregation module, enabling the generation of more representative saliency features towards binocular vision. Subsequently, to further aggregate the saliency features in multiple scales, we design a progressive attention-based saliency feature pyramid extraction module to effectively integrate the features from top-level to down-level based on the network hierarchy mechanism. The saliency maps are ultimately produced for stereoscopic images by evaluating the obtained saliency features. In addition, we create a stereoscopic image saliency dataset (SIS-3D) that includes 1086 stereoscopic image pairs with various content and their corresponding human eye fixation annotations, aiming to further facilitate the research on visual saliency detection for stereoscopic images. Extensive experiments demonstrate that our proposed method improves CC by an average of 4.02% compared to representative counterparts on the newly built saliency dataset and another publicly available dataset. [ABSTRACT FROM AUTHOR]

Fine-grained sketch-based image retrieval (FG-SBIR) is considered an ideal method of image retrieval due to the rich and easily accessible characteristics of sketches. It aims to find the most similar photo from the photo gallery based on the input sketch. Most previous works follow the paradigm that extracting global feature first and then projecting the features of sketch and photo to unified embedding feature space using triplet loss. However, the global feature is not appropriate for extracting the crucial fine-grained information. Based on this principle, we propose a Dual Local Interaction Network (DLI-Net). DLI-Net explores an effective and efficient way to utilize local features for FG-SBIR. Specifically, we first propose a Local Feature Extractor to extract mid-level local features. Then, in response to the problems brought by local features, we propose a Dual Interaction Module, which contains Self Interaction Module and Cross Interaction Module. Self Interaction Module speeds up retrieval by eliminating the redundant local features of background. Cross Interaction Module solves the spatial misalignment by making the sketches interact with photos. Extensive experiments on six commonly used datasets show that our DLI-Net outperforms state-of-the-art competitors by a significant margin with a reasonable retrieval speed. Moreover, to the best of our knowledge, DLI-Net is the first model that beats humans on all six datasets. Besides, DLI-Net also performs best on cross-category fine-grained sketch-based image retrieval task, which further demonstrates local features are more appropriate for FG-SBIR. [ABSTRACT FROM AUTHOR]

Pruning can remove the redundant parameters and structures of Deep Neural Networks (DNNs) to reduce inference time and memory overhead. As one of the important components of DNN, feature maps (FMs) have been widely used in network pruning. However, previous approaches do not fully investigate the discriminative features in FMs, and also do not explicitly utilize all the features associated with each layer in the pruning procedure. In this paper, we explore the discriminative feature of FMs and explicitly investigate the two-adjacent-layer features of each layer to propose a three-phase hierarchical pruning framework, dubbed as FCHP. Firstly, we decompose each FM into several components to extract the discriminative feature. After that, since pruning each layer is related to the FMs of adjacent layers, we explicitly calculate the feature correlation of discriminative features of two adjacent layers, and then use the feature correlation to cluster FMs into several hierarchies to guide subsequent pruning. Finally, we compute the content of discriminative features, and remove channels corresponding to FMs with fewer discriminative features in each hierarchy, respectively. In the experiment, we prune DNNs with the multiple types of architecture on different benchmarks, and the results have achieved the state-of-the-arts in terms of compressed parameters and FLOPs drop. For example, as for ResNet-56 on CIFAR-10, FCHP respectively obtains 50% of parameters and FLOPs reduction with negligible accuracy loss. Besides, as for ResNet-50 on ImageNet, FCHP reduces 40.5% of parameters and 44.1% of FLOPs with 0.43% of Top-1 accuracy drop. [ABSTRACT FROM AUTHOR]