Search

Recognizing actions from still images has been popularly studied recently. In this paper, we model an action class as a flexible number of spatial configurations of body parts by proposing a new spatial sum–product network (SPN). First, we discover a set of parts in image collections via unsupervised learning. Then, our new spatial SPN is applied to model the spatial relationship and also the high-order correlations of parts. To learn robust networks, we further develop a hierarchical spatial SPN method, which models pairwise spatial relationship between parts inside subimages and models the correlation of subimages via extra layers of SPN. Our method is shown to be effective on two benchmark data sets. [ABSTRACT FROM PUBLISHER]

This paper presents new approaches for gait and activity analysis based on data streams of a rotating multibeam (RMB) Lidar sensor. The proposed algorithms are embedded into an integrated 4D vision and visualization system, which is able to analyze and interactively display real scenarios in natural outdoor environments with walking pedestrians. The main focus of the investigations is gait-based person reidentification during tracking and recognition of specific activity patterns, such as bending, waving, making phone calls, and checking the time looking at wristwatches. The descriptors for training and recognition are observed and extracted from realistic outdoor surveillance scenarios, where multiple pedestrians are walking in the field of interest following possibly intersecting trajectories; thus, the observations might often be affected by occlusions or background noise. Since there is no public database available for such scenarios, we created and published a new Lidar-based outdoor gait and activity data set on our website that contains point cloud sequences of 28 different persons extracted and aggregated from 35-min-long measurements. The presented results confirm that both efficient gait-based identification and activity recognition are achievable in the sparse point clouds of a single RMB Lidar sensor. After extracting the people trajectories, we synthesized a free-viewpoint video, in which moving avatar models follow the trajectories of the observed pedestrians in real time, ensuring that the leg movements of the animated avatars are synchronized with the real gait cycles observed in the Lidar stream. [ABSTRACT FROM PUBLISHER]

Prospective authors are requested to submit new, unpublished manuscripts for inclusion in the upcoming event described in this call for papers. [ABSTRACT FROM AUTHOR]

The article offers information on call for papers IEEE Trans. on circuits and systems for video technology special section on video and Language.

Textures contain a wealth of image information and are widely used in various fields such as computer graphics and computer vision. With the development of machine learning, the texture synthesis and generation have been greatly improved. As a very common element in everyday life, wallpapers contain a wealth of texture information, making it difficult to annotate with a simple single label. Moreover, wallpaper designers spend significant time to create different styles of wallpaper. For this purpose, this paper proposes to describe wallpaper texture images by using multi-label semantics. Based on these labels and generative adversarial networks, we present a framework for perception driven wallpaper texture generation and style transfer. In this framework, a perceptual model is trained to recognize whether the wallpapers produced by the generator network are sufficiently realistic and have the attribute designated by given perceptual description; these multi-label semantic attributes are treated as condition variables to generate wallpaper images. The generated wallpaper images can be converted to those with well-known artist styles using CycleGAN. Finally, using the aesthetic evaluation method, the generated wallpaper images are quantitatively measured. The experimental results demonstrate that the proposed method can generate wallpaper textures conforming to human aesthetics and have artistic characteristics. [ABSTRACT FROM AUTHOR]

The paper proposes a method to effectively fuse multi-exposure inputs and generate high-quality high dynamic range (HDR) images with unpaired datasets. Deep learning-based HDR image generation methods rely heavily on paired datasets. The ground truth images play a leading role in generating reasonable HDR images. Datasets without ground truth are hard to be applied to train deep neural networks. Recently, Generative Adversarial Networks (GAN) have demonstrated their potentials of translating images from source domain $X$ to target domain $Y$ in the absence of paired examples. In this paper, we propose a GAN-based network for solving such problems while generating enjoyable HDR results, named UPHDR-GAN. The proposed method relaxes the constraint of the paired dataset and learns the mapping from the LDR domain to the HDR domain. Although the pair data are missing, UPHDR-GAN can properly handle the ghosting artifacts caused by moving objects or misalignments with the help of the modified GAN loss, the improved discriminator network and the useful initialization phase. The proposed method preserves the details of important regions and improves the total image perceptual quality. Qualitative and quantitative comparisons against the representative methods demonstrate the superiority of the proposed UPHDR-GAN. [ABSTRACT FROM AUTHOR]

Digital watermarking technology and watermark attack methods are mutually reinforcing and complementary. Currently, traditional watermark attack methods are relatively mature, but these traditional attack methods will inevitably damage the visual quality of original images (OIs). Therefore, this paper proposes a covert attack method called residual dense based imperceptible watermark attack network (RD-IWAN). First, this paper designs a watermark attack residual dense network (WARDN) based on the residual dense network (RDN), which can effectively remove the watermark information in the middle and low frequency features of the watermarked image (WMI). Second, to improve the attack ability of the network, this paper innovatively proposes a progressive preprocessing method based on the information enhancement preprocessing method. Concurrently, to ensure the imperceptibility of this watermark attack method, a comprehensive loss function that combines the perceptual loss and mean square error loss (MSE) of OI and attacked watermarked image (AWMI) is designed in this study. Finally, attack experiments are designed and performed on watermarks with different embedding strengths and sizes. Experimental results show that, compared to traditional attack methods, the watermark attack method proposed in this paper exhibits stronger attack ability and higher imperceptibility. [ABSTRACT FROM AUTHOR]

Human beings tend to learn unknown knowledge in a gradual process, from the basic to the complex. Based on this point, we propose a progressive learning method for producing service strategies according to requests, with a hierarchical priori knowledge and reinforcement learning. Service strategy aims to guide how to perform home services and takes into consideration the relationship between actions and objects in home environment. In this paper, strategy generation is regarded as a text generation problem in question answering (QA). Firstly, a hierarchical priori knowledge with service-object correlation at the bottom and action-object correlation at the top is constructed to assist the understanding on the relationship of objects and actions in service strategies. Service-object correlation guides how to select proper objects with the correct order, while action-object correlation associates actions in strategies according to selected objects. Based on the hierarchical priori knowledge, a progressive learning method is proposed to make the model produce effective strategies with a sequential cognition, from service-object correlation (objects) to action-object correlation (actions). After that, reinforcement learning is employed to enhance the progressive guidance, by designing rewards in terms of the hierarchical priori knowledge. Finally, the proposed method is tested with both comparative experiments and ablation studies, and the experimental results demonstrate the superiority in producing comprehensive and logical strategies, indicating that the progressive learning method in our paper can further improve the QA performance. [ABSTRACT FROM AUTHOR]

This paper presents a new objective quality metric that was adapted to the complex characteristics of immersive video (IV) which is prone to errors caused by processing and compression of multiple input views and virtual view synthesis. The proposed metric, IV-PSNR, contains two techniques that allow for the evaluation of quality loss for typical immersive video distortions: corresponding pixel shift and global component difference. The performed experiments compared the proposal with 31 state-of-the-art quality metrics, showing their performance in the assessment of quality in immersive video coding and processing, and in other applications, using commonly used image quality assessment databases– TID2013 and CVIQ. As presented, IV-PSNR outperforms other metrics in immersive video applications and still can be efficiently used in the evaluation of different images and videos. Moreover, basing the metric on the calculation of PSNR allowed the computational complexity to remain low. Publicly available, efficient implementation of IV-PSNR software was provided by the authors of this paper and is used by ISO/IEC MPEG for evaluation and research on the upcoming MPEG Immersive video (MIV) coding standard. [ABSTRACT FROM AUTHOR]

In Versatile Video Coding (VVC), reference picture resampling (RPR) is supported to change resolution without inserting an instantaneous decoder refresh (IDR) picture or intra random access picture (IRAP). The introduction of RPR enables VVC to adapt to unstable network conditions, which benefits real-time application scenarios a lot. However, the existing bitrate leap and the low quality of the upsampled frames are the bottlenecks of promoting its application. In this paper, we propose an efficient framework to alleviate bitrate surges and further improve the coding performance. The contributions of this paper mainly include the following three aspects. Firstly, it is proved theoretically and statistically that the temporal long-distance high-resolution (HR) reference pictures can provide better prediction than temporal short-distance low-resolution (LR) reference pictures when the temporal distance between them is less than a threshold. Secondly, motivated by the theoretical analysis, existing HR reference pictures in the reference pictures list (RPL) are used to replace LR reference pictures and an adaptive decision on the number of HR reference pictures kept in RPL is applied. Thirdly, an adaptive upsampling filter (AUF) is proposed to restore pictures with the original resolution more accurately. Simulation results show 15.09% BD-rate savings on average can be achieved for the 2:1 scaling ratio under RPR configurations. For the 1.5:1 scaling ratio, the BD-rate saving is 8.14%. Besides, the bitrate surge in RPR is reduced greatly. About 25% bits reductions on average at the leaping point can be observed and the bits decrease rate is up to 50% for sequences with little and slow motion. [ABSTRACT FROM AUTHOR]

In this paper, a simple yet effective network pruning framework is proposed to simultaneously address the problems of pruning indicator, pruning ratio, and efficiency constraint. This paper argues that the pruning decision should depend on the convolutional weights, and thus proposes novel weight-dependent gates (W-Gates) to learn the information from filter weights and obtain binary gates to prune or keep the filters automatically. To prune the network under efficiency constraints, a switchable Efficiency Module is constructed to predict the hardware latency or FLOPs of candidate pruned networks. Combined with the proposed Efficiency Module, W-Gates can perform filter pruning in an efficiency-aware manner and achieve a compact network with a better accuracy-efficiency trade-off. We have demonstrated the effectiveness of the proposed method on ResNet34, ResNet50, and MobileNet V2, respectively achieving up to 1.33/1.28/1.1 higher Top-1 accuracy with lower hardware latency on ImageNet. Compared with state-of-the-art methods, W-Gates also achieves superior performance. [ABSTRACT FROM AUTHOR]

Contrastive self-supervised representation learning methods maximize the similarity between the positive pairs, and at the same time tend to minimize the similarity between the negative pairs. However, in general the interplay between the negative pairs is ignored as they do not put in place special mechanisms to treat negative pairs differently according to their specific differences and similarities. In this paper, we present Extended Momentum Contrast (XMoCo), a self-supervised representation learning method founded upon the legacy of the momentum-encoder unit proposed in the MoCo family configurations. To this end, we introduce a cross consistency regularization loss, with which we extend the transformation consistency to dissimilar images (negative pairs). Under the cross consistency regularization rule, we argue that semantic representations associated with any pair of images (positive or negative) should preserve their cross-similarity under pretext transformations. Moreover, we further regularize the training loss by enforcing a uniform distribution of similarity over the negative pairs across a batch. The proposed regularization can easily be added to existing self-supervised learning algorithms in a plug-and-play fashion. Empirically, we report a competitive performance on the standard Imagenet-1K linear head classification benchmark. In addition, by transferring the learned representations to common downstream tasks, we show that using XMoCo with the prevalently utilized augmentations can lead to improvements in the performance of such tasks. We hope the findings of this paper serve as a motivation for researchers to take into consideration the important interplay among the negative examples in self-supervised learning. [ABSTRACT FROM AUTHOR]

This paper presents the overview and rationale behind the Decoder-Side Depth Estimation (DSDE) mode of the MPEG Immersive Video (MIV) standard, using the Geometry Absent profile, for efficient compression of immersive multiview video. A MIV bitstream generated by an encoder operating in the DSDE mode does not include depth maps. It only contains the information required to reconstruct them in the client or in the cloud: decoded views and metadata. The paper explains the technical details and techniques supported by this novel MIV DSDE mode. The description additionally includes the specification on Geometry Assistance Supplemental Enhancement Information which helps to reduce the complexity of depth estimation, when performed in the cloud or at the decoder side. The depth estimation in MIV is a non-normative part of the decoding process, therefore, any method can be used to compute the depth maps. This paper lists a set of requirements for depth estimation, induced by the specific characteristics of the DSDE. The depth estimation reference software, continuously and collaboratively developed with MIV to meet these requirements, is presented in this paper. Several original experimental results are presented. The efficiency of the DSDE is compared to two MIV profiles. The combined non-transmission of depth maps and efficient coding of textures enabled by the DSDE leads to efficient compression and rendering quality improvement compared to the usual encoder-side depth estimation. Moreover, results of the first evaluation of state-of-the-art multiview depth estimators in the DSDE context, including machine learning techniques, are presented. [ABSTRACT FROM AUTHOR]

In this paper, a data assimilation network is proposed to tackle the challenges of domain generalization for person re-identification (ReID). Most of the existing research efforts only focus on single-dataset issues, and the trained models are difficult to generalize to unseen scenarios. This paper presents a distinctive idea to improve the generality of the model by assimilating three types of images: style-variant images, misaligned images and unlabeled images. The latter two are often ignored in the previous domain generalization ReID studies. In this paper, a non-local convolutional block attention module is designed for assimilating the misaligned images, and an attention adversary network is introduced to correct it. A progressive augmented memory is designed for assimilating the unlabeled images by progressive learning. Moreover, we propose an attention adversary difference loss for attention correction, and a labeling-guide discriminative embedding loss for progressive learning. Rather than designing a specific feature extractor that is robust to style shift as in most previous domain generalization work, we propose a data assimilation meta-learning procedure to train the proposed network, so that it learns to assimilate style-variant images. It is worth mentioning that we add an unlabeled augmented dataset to the source domain to tackle the domain generalization ReID tasks. Extensive experiments demonstrate that our approach significantly outperforms the state-of-the-art domain generalization methods. [ABSTRACT FROM AUTHOR]

Co-salient object detection (CoSOD) aims to detect common salient objects sharing the same attributes in an image group. The key issue of CoSOD is how to model the inter-saliency relations within an image group. The major limitation of previous methods is that they pre-define the group-to-one relations within an image group. In this paper, we propose a new concept of structural inter-saliency relations and solve the CoSOD with deep reinforcement learning framework. Firstly, we design a semantic relation graph (SRG) to model the structural inter-saliency relations. Then the feature selecting agent (FS-agent) aims to select the informative features, which can help the SRG effectively model structural inter-saliency relations. Finally, relation updating agent (RU-agent) progressively updates the SRG to focus on the co-salient relations like human decision-making process. Extensive experiments on co-saliency datasets show that because of well modeling inter-saliency relations in image group, our proposed method achieves superior performance compared to the state-of-the-art methods. We hope that this paper can motivate future research for visual co-analysis tasks. [ABSTRACT FROM AUTHOR]

Recently, research on six-degree-of-freedom virtual reality (VR) systems based on a light field (LF) has been actively conducted. The LF-based approach is photorealistic and less prone to errors than the existing three-dimensional (3D) modeling-based approach. On the other hand, because the amount of data is very large for LF-based immersive virtual reality, it is important to manage the data efficiently. This paper makes two major contributions to data management for rendering acceleration in this context. First, the GPUs commonly used for high-speed rendering require optimization of the data transfer process between the host memory and the device memory. This paper proposes a simple but effective way to maximize data reuse by finding the proper size of a transfer unit depending on the target applications and the system capability constraints. Also, view synthesis and data transfer are performed in a pipelined manner to hide the data movement time. Second, for the LF rendering of a wide space, data management in the storage-host-GPU hierarchy is attempted for the first time. As the size of the virtual space increases, the required amount of LF data grows rapidly and exceeds the allocatable host memory capacity. Because the loading of data stored in storage is a very slow process, a system-level LF rendering structure that carefully considers the characteristics of the storage-host-GPU memory hierarchy should be designed. In this paper, progressive LF updating is proposed, and it prevents rendering stalls caused by slow storage data loading. Experimental results show that a 360-degree view with a $9000 \times 2048$ resolution can be rendered in an average of 2.57 ms from LF data with a $4096 \times 2048$ resolution, achieving performance close to 400 frames per second. [ABSTRACT FROM AUTHOR]

Since gait can be utilized to identify individuals from a far distance without their interaction and coordination, recently many gait recognition methods have been proposed. However, due to a real-world scenario of clothing changes, a degradation occurs for most of these methods. Thus in this paper, a more efficient gait recognition method is proposed to address the problem of clothing variances. First, part-based gait features are formulated from two different perspectives, i.e., the separated body parts that are more robust to clothing changes and the estimated human skeleton key-point regions. It is reasonable to formulate such features for cloth-changing gait recognition, because these two perspectives are both less vulnerable to clothing changes. Given that each feature has its own advantages and disadvantages, a more efficient gait feature is generated in this paper by assembling these two features together. Moreover, since local features are more discriminative than global features, in this paper more attention is focused on the local short-range features. Also, unlike most methods, in our method we treat the estimated key-point features as a set of word embeddings, and a transformer encoder is specifically used to learn the dependence of each correlative key-points. The robustness and effectiveness of our proposed method are certified by experiments on CASIA Gait Dataset B, and it has achieved the state-of-the-art performance on this dataset. [ABSTRACT FROM AUTHOR]

At present, many studies have shown that partitioning the gait sequence and its feature map can improve the accuracy of gait recognition. However, most models just cut the feature map at a fixed single scale, which loses the dependence between various parts. So, our paper proposes a structure called Part Feature Relationship Extractor (PFRE) to discover all of the relationships between each parts for gait recognition. The paper uses PFRE and a Convolutional Neural Network (CNN) to form the RPNet. PFRE is divided into two parts. One part that we call the Total-Partial Feature Extractor (TPFE) is used to extract the features of different scale blocks, and the other part, called the Adjacent Feature Relation Extractor (AFRE), is used to find the relationships between each block. At the same time, the paper adjusts the number of input frames during training to perform quantitative experiments and finds the rule between the number of input frames and the performance of the model. Our model is tested on three public gait datasets, CASIA-B, OU-LP and OU-MVLP. It exhibits a significant level of robustness to occlusion situations, and achieves accuracies of 92.82% and 80.26% on CASIA-B under BG # and CL # conditions, respectively. The results show that our method reaches the top level among state-of-the-art methods. [ABSTRACT FROM AUTHOR]

The content based image retrieval aims to find the similar images from a large scale dataset against a query image. Generally, the similarity between the representative features of the query image and dataset images is used to rank the images for retrieval. In early days, various hand designed feature descriptors have been investigated based on the visual cues such as color, texture, shape, etc. that represent the images. However, the deep learning has emerged as a dominating alternative of hand-designed feature engineering from a decade. It learns the features automatically from the data. This paper presents a comprehensive survey of deep learning based developments in the past decade for content based image retrieval. The categorization of existing state-of-the-art methods from different perspectives is also performed for greater understanding of the progress. The taxonomy used in this survey covers different supervision, different networks, different descriptor type and different retrieval type. A performance analysis is also performed using the state-of-the-art methods. The insights are also presented for the benefit of the researchers to observe the progress and to make the best choices. The survey presented in this paper will help in further research progress in image retrieval using deep learning. [ABSTRACT FROM AUTHOR]

With the rapid development of intelligent transportation, automated traffic surveillance is considered as an important component. In the field of traffic surveillance, it is particularly important to achieve robust and real-time tracking of vehicles in complex scenes. In this paper, a robust real-time vehicle tracking model named NoisyOTNet is proposed, which formulates tracking as reinforcement learning with parameter space noise. In this formulation, the exploration ability of the model is enhanced to improve the robustness of tracking. Specifically, we develop a new implementation for noisy network based on deep deterministic policy gradients (DDPGs) with parameter noise, which can better cope with the tracking task and directly predict the tracking result. To improve the tracking accuracy in complex conditions, e.g. fast motion and large deformation, this paper presents an adaptive update strategy that can exploit the vehicle spatial-temporal information based on Upper Confidence Bound (UCB) algorithm by exploiting. Moreover, as for the recovery of the lost target, a relocation algorithm based on incremental learning is developed. The results of extensive experiments demonstrate that the proposed NoisyOTNet can effectively track vehicles in complex scenes and achieve competitive performance compared to the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Smoke detection in video is a challenging task because of the irregular shape of smoke, its complex motion state, which is affected by temperature, wind and other external factors, and background disturbances. Pixel-based foreground modeling method is a crucial step in many smoke detection systems and can be applied to efficiently focus on a certain object or a specific region to detect movements or anomalies. In video analysis, it is a natural idea to move the focus from the pixel-level foreground to the feature-level foreground. In this paper, the feature foreground is generated by the middle layer of a convolutional neural network (CNN) to guide the temporal modeling process for smoke objects. A novel temporal module called the Feature Foreground Module (FFM) is proposed to boost learning of a smoke temporal representation. Consider the problem of smoke analysis in video, we present a novel unifying approach, named an enhanced feature foreground network (EFFNet), that performs both smoke source prediction and detection. Efficient branch networks are designed in EFFNet, to predict the source mask and bounding boxes of smoke plumes in video. To the best of our knowledge, this is the first paper to study the source of smoke using deep learning methods. Finally, experiments on a realistic smoke dataset and a public dataset show that EFFNet method performs much better than do previous state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Gaming video streaming services have grown tremendously in the past few years, with higher resolutions, higher frame rates and HDR gaming videos getting increasingly adopted among the gaming community. Since gaming content as such is different from non-gaming content, it is imperative to evaluate the performance of the existing encoders to help understand the bandwidth requirements of such services, as well as further improve the compression efficiency of such encoders. Towards this end, we present in this paper GamingHDRVideoSET, a dataset consisting of eighteen 10-bit UHD-HDR gaming videos and encoded video sequences using four different codecs, together with their objective evaluation results. Additionally, the paper discusses the codec compression efficiency of most widely used practical encoders, i.e., x264 (H.264/AVC), x265 (H.265/HEVC) and libvpx (VP9), as well the recently proposed encoder libaom (AV1), on 10-bit, UHD-HDR content gaming content. Our results show that the latest compression standard AV1 results in the best compression efficiency, followed by HEVC, H.264, and VP9. [ABSTRACT FROM AUTHOR]

Moving cast shadow identification and extraction are the commonly encountered problems in visual surveillance applications. In the video object detection problem, a shadow is a category of the dynamic entity which needs to be removed for accurate detection of moving objects. In this paper, we have proposed a new scheme to detect the moving object in video frames while removing the shadow. The proposed scheme is based on the notion of background modeling and model learning. The background is modeled by the proposed Spatio-Temporal Kernel Density Estimation (ST-KDE) based model, but the model learning takes place in online mode in the fused feature space. In the fusion process, the Local Binary Pattern (LBP) features of the ST-KDE model of a frame are determined and are fused probabilistically with the Gabor features of the corresponding original frame. To achieve accurate background modeling, weights in the probabilistic fusion are adaptively determined from the scene dynamics. Background model learning is pixel-based where model histograms learn the background with every temporal frame. Learning and classification of the fused feature frames happen simultaneously with frame by frame. The classified frame thus obtained contains the object with some residual shadow which is eliminated by entropy map and thresholding. The main novelties of our paper are: (i) ST-KDE based background modeling, (ii) Determination of the weights of the fusion process in an adaptive framework and, (iii) online background model learning in feature space. The proposed modeling approach has been tested successfully with different video sequences considered from CDnet, Lasiesta, ATON-CVRR, SRD, and ISTD databases. The performance metrics of the proposed scheme have been compared with many existing methods and it has been found that the proposed method exhibits improved performance. [ABSTRACT FROM AUTHOR]

In this paper, we propose a deep neural network for single image reflection removal. More specifically, we design a convolutional-grid module and take it as the building block of a feature fusion pyramid. The module leverages the combination effect of the grid topology to create a rich ensemble of receptive fields. Embedding the modules into a pyramidal architecture further expands the coverage of receptive fields. Another benefit of the pyramid is to fuse the multi-scale features learned by the modules locating at the ascending and descending pathways. The rich diversity of features helps the neural network analyze the contexts around overlapping objects at various spatial ranges and harvest the cues for layer separation. The proposed work also exploits useful semantic cues from the hyper-column descriptors generated by a pre-trained VGG-19 model to reduce the ambiguity of layer separation. In light of the low correlation between background and reflection layers, we design a channel-correlation based conditional discriminator to penalize residual reflection. The discriminator uses channel-wise attention to screen the features that can distinguish real background images from estimated ones. This paper also presents a task-driven regularization strategy. The high sensitivity of semantic segmentation to reflection is exploited for assessing the completeness of reflection removal. Training with this regularization strategy can boost the performance of both reflection removal and high-level task. The comparison against state-of-the-art algorithms on four public benchmark datasets demonstrates that this work exhibits superior performance in handling the complex reflections in wild scenarios. The proposed network architecture is also applicable to haze removal, which is another ill-posed layer separation problem, and has shown encouraging performance. [ABSTRACT FROM AUTHOR]

3D object recognition is a hot research topic. Particularly, view-based methods, which represent a 3D object with a collection of its rendered views on the 2D domain, play an important role in this field. Currently, view-based researches tend to aggregate information from multiple views via pooling based strategies to endow the models with the characteristic of view permutation invariance, at the cost of inevitable loss of useful features. In this paper, we introduce a new method that learns a more comprehensive descriptor for a 3D object from its views while successfully keeping its robustness to the variation of view permutation. Our method disentangles the information in the set of multi-view images into a global category-related feature and a set of view-permutation related features. To unbind these two parts, an encode-decoder based disentangling architecture is proposed, which barely bring extra computations compared to the baseline model. Systematic experiments are conducted for this new method to demonstrates the effectiveness and the competitive performance based on ModelNet40, ModelNet10, and ShapeNetCore55 datasets. Codes for our paper will be released soon on “ https://github.com/hjjpku/multi%5fview%5fsort ”. [ABSTRACT FROM AUTHOR]

This paper addresses the issue of detecting the use of information hiding at anti-copying 2D barcodes. Prior hidden information detection schemes have their roots in either heuristic-based or Machine Learning (ML). However, prior heuristics-based schemes lack a rigorous theoretical analysis. Prior ML-based information schemes lack robustness because a printed 2D barcode is very much environmentally dependent. Thus, an information hiding detection scheme trained in one environment often does not work well in another environment. In this paper, we propose two hidden information detection schemes for existing anti-copying 2D barcodes. The first scheme directly uses the pixel distance to detect the use of an information hiding scheme in a 2D barcode, referred to as the Pixel Distance Based Detection (PDBD) scheme. The second scheme first calculates the variance of raw signal and the covariance between the recovered and raw signals, and then based on the variance results, detects the use of information hiding scheme in a 2D barcode, referred to as the Pixel Variance Based Detection (PVBD) scheme. Moreover, we design advanced Illegitimately-Copying (IC) attacks to evaluate the security of two existing anti-copying 2D barcodes. We conduct extensive performance comparisons among the proposed schemes and prior schemes under different capturing devices, e.g., different scanners or camera phones. Our experimental results show that the PVBD scheme can correctly detect the existence of hidden information at both the 2LQR code and the LCAC 2D barcode. Moreover, the successful attacking probability of the proposed IC attacks achieves 0.6538 for the 2LQR code and 1 for the LCAC 2D barcode. [ABSTRACT FROM AUTHOR]

In this paper, we focus on subjective and objective Point Cloud Quality Assessment (PCQA) in an immersive environment and study the effect of geometry and texture attributes in compression distortion. Using a Head-Mounted Display (HMD) with six degrees of freedom, we establish a subjective PCQA database, named SIAT Point Cloud Quality Database (SIAT-PCQD). Our database consists of 340 distorted point clouds compressed by the MPEG point cloud encoder with the combination of 20 sequences and 17 pairs of geometry and texture quantization parameters. The impact of distorted geometry and texture attributes is further discussed in this paper. Then, we propose two projection-based objective quality evaluation methods, i.e., a weighted view projection based model and a patch projection based model. Our subjective database and findings can be used in point cloud processing, transmission, and coding, especially for virtual reality applications. The subjective dataset https://dx.doi.org/10.21227/ad8d-7r28 http://codec.siat.ac.cn/video%5fdownload%5fsiat-pcqd.htmlhas been released in the public repository. [ABSTRACT FROM AUTHOR]

Most of the existing single object trackers track the target in a unitary local search window, making them particularly vulnerable to challenging factors such as heavy occlusions and out-of-view movements. Despite the attempts to further incorporate global search, prevailing mechanisms that cooperate local and global search are relatively static, thus are still sub-optimal for improving tracking performance. By further studying the local and global search results, we raise a question: can we allow more dynamics for cooperating both results? In this paper, we propose to introduce more dynamics by devising a dynamic attention-guided multi-trajectory tracking strategy. In particular, we construct dynamic appearance model that contains multiple target templates, each of which provides its own attention for locating the target in the new frame. Guided by different attention, we maintain diversified tracking results for the target to build multi-trajectory tracking history, allowing more candidates to represent the true target trajectory. After spanning the whole sequence, we introduce a multi-trajectory selection network to find the best trajectory that deliver improved tracking performance. Extensive experimental results show that our proposed tracking strategy achieves compelling performance on various large-scale tracking benchmarks. The project page of this paper can be found at https://sites.google.com/view/mt-track/. [ABSTRACT FROM AUTHOR]

Due to the lack of distinctiveness to the background, robust monocular 6DOF pose tracking of less-distinct objects remains an important open problem. In this paper, we firstly analyze the object distinctiveness in the tracking process. Then, we propose a novel contour-part model based robust monocular pose tracking method for less-distinct objects. This paper uses the traditional contour feature for pose tracking in a novel strategy called contour part model. First, the contour part model is built by segmenting the projected contour rendered from the 3D model into contour segments of a certain length adaptively according to the Shi-Tomasi cornerness scores. Then, the correspondence is detected in the input image for each contour part by gradient orientation based template matching. Finally, pose tracking is achieved by solving the PnP problem. Experiment results on semi-synthetic and real images show that the proposed method performs better than the existing methods when pose tracking less-distinct objects and shows great robustness toward interference. Additionally, we combine the edge feature and the regional feature in a simple strategy. The overall performance of the fused method is boosted according to the experimental results on the RBOT dataset. The fact shows that the fusion of the edge and regional features has great potential for improving tracking performance. [ABSTRACT FROM AUTHOR]

Visual question generation task aims to generate meaningful questions about an image targeting an answer. Existing methods focus on the visual concepts in the image for question generation. However, humans inevitably use their knowledge related to visual objects in images to construct questions. In this paper, we propose a knowledge-based visual question generation model that can integrate visual concepts and non-visual knowledge to generate questions. To obtain visual concepts, we utilize a pre-trained object detection model to obtain object-level features of each object in the image. To obtain useful non-visual knowledge, we first retrieve the knowledge from the knowledge-base related to the visual objects in the image. Considering that not all retrieved knowledge is helpful for this task, we introduce an answer-aware module to capture the candidate knowledge related to the answer from the retrieved knowledge, which ensures that the generated content can be targeted at the answer. Finally, object-level representations containing visual concepts and non-visual knowledge are sent to a decoder module to generate questions. Extensive experiments on the FVQA and KBVQA datasets show that the proposed model outperforms the state-of-the-art models. [ABSTRACT FROM AUTHOR]

Image quality assessment (IQA) is an important task of image processing and has diverse applications, such as image super-resolution reconstruction, image transmission and monitoring systems. This paper proposes a perceptual hashing algorithm with complementary color wavelet transform (CCWT) and compressed sensing (CS) for reduced-reference (RR) IQA. The CCWT is exploited to decompose input color image into different sub-bands. Since the calculation of CCWT uses all color channels without discarding any information, the distortions introduced by digital operations on color channels are preserved in the CCWT sub-bands. The block-based CS is used to extract features from the CCWT sub-bands. As the Euclidean distance between the block-based CS features is slightly influenced by content-preserving operations, perceptual features constructed by Euclidean distances are robust, discriminative and compact. Hash sequence is finally determined by quantifying the perceptual features. Effectiveness of the proposed hashing is verified by various experiments on four open image databases. Experimental results demonstrate that the proposed hashing is superior to some state-of-the-art algorithms in terms of classification and RR IQA application. [ABSTRACT FROM AUTHOR]

The focus of few shot learning research has been on the development of meta-learning recently, where a meta-learner is trained on a variety of tasks in hopes of being generalizable to new tasks. Tasks in meta training and meta test are usually assumed to be from the same domain, which would not necessarily hold in real world scenarios. In this paper, we propose variational hyperparameter inference for few-shot learning across domains. Based on an especially successful algorithm named model agnostic meta learning, the proposed variational hyperparameter inference integrates meta learning and variational inference into the optimization of hyperparameters, which enables the meta-learner with adaptivity for generalization across domains. In particular, we choose to learn adaptive hyperparameters including the learning rate and weight decay to avoid the failure in the face of few labeled examples across domain. Moreover, we model hyperparameters as distributions instead of fixed values, which will further enhance the generalization ability by capturing the uncertainty. Extensive experiments are conducted on two benchmark datasets including few shot learning dataset within-domain and across-domain. The results demonstrate that our methods outperforms previous approaches consistently, and comprehensive ablation studies further validate its effectiveness on few shot learning both within domains and across domains. [ABSTRACT FROM AUTHOR]

Person search, consisting of jointly or separately trained person detection stage and person Re-ID stage, suffers from significant challenges such as inefficiency and difficulty in acquiring discriminative features. However, certain work has either turned to the end-to-end framework whose performance is limited by task conflicts or has consistently attempted to obtain more accurate bounding boxes (Bboxes). Few studies have focused on the impact of sample-specificity in person search datasets for training a fine-grain Re-ID model, and few have considered obtaining discriminative Re-ID features from Bboxes in a more efficient way. In this paper, a novel sample-enhanced and instance-sensitive (SEIE) framework is designed to boost performance. By analyzing the structure of person search framework, our method refines the two stages separately. For the detection stage, we re-design the usage of Bbox and a sample enhancement combination is proposed to further enhance the quality and quantity of Bboxes. SEC can suppress false positive detection results and randomly generate high-quality positive samples. For the Re-ID stage, we contribute an instance similarity loss to exploit the similarity between classless instances, and an Omni-scale Re-ID backbone is employed to learn more discriminative features. We obtain a more efficient and discriminative person search framework by concatenating the two stages. Extensive experiments demonstrate that our method achieves state-of-the-art performance with a high speed, and significantly outperforms other existing methods. [ABSTRACT FROM AUTHOR]

Deep neural networks have achieved great performance on blind Image Quality Assessment (IQA), but it is still challenging for using one network to accurately predict the quality of images with different distortions. In this paper, a Distortion-Aware Convolutional Neural Network (DACNN) is proposed for blind IQA, which works effectively for not only synthetically distorted images but also authentically distorted images. The proposed DACNN consists of a distortion aware module, a distortion fusion module, and a quality prediction module. In the distortion aware module, a Siamese network-based pretraining strategy is proposed to design a synthetic distortion-aware network for full learning the synthetic distortions, and an authentic distortion-aware network is used for extracting the authentic distortions. To efficiently fuse the learned distortion features, and make the network pay more attention to the essential features, a weight-adaptive fusion network is proposed to adaptively adjust the weight of each distortion. Finally, the quality prediction module is adopted to map the fused features to a quality score. Extensive experiments on four authentic IQA databases and four synthetic IQA databases have proved the effectiveness of the proposed DACNN. [ABSTRACT FROM AUTHOR]

Personalized Fixation-based Object Segmentation (PFOS) aims at segmenting the gazed objects in images conditioned on personalized fixations. However, the performances of existing PFOS methods are degraded when facing anomalous fixation maps (some fixations fall in the background) or enormous objects because of their poor localization ability. In this paper, we propose a novel Selective Intra-image Similarity Network (SISNet) that achieves significant performance by precisely localizing the gazed objects. First, we propose a Response Purifying Module (RPM) to eliminate the false response regions caused by anomalous fixations in the background. By suppressing these false responses, we can significantly reduce the negative impacts caused by anomalous fixations. Second, we propose an intra-image similarity module (ISM) to better localize large objects by integrating more long-range information. In addition, we propose a new Discriminative Intersection-over-Union metric that evaluates whether PFOS methods can produce distinctive predictions for varying fixations. Experiments on the PFOS and our proposed OSIE-CFPS-UN datasets prove that our network achieves remarkable improvements and outperforms existing state-of-the-art methods. Code has been published at https://www.github.com/moothes/SISNet. [ABSTRACT FROM AUTHOR]

The critical challenge of single image inpainting stems from accurate semantic inference via limited information while maintaining image quality. Typical methods for semantic image inpainting train an encoder-decoder network by learning a one-to-one mapping from the corrupted image to the inpainted version. While such methods perform well on images with small corrupted regions, it is challenging for these methods to deal with images with large corrupted area due to two potential limitations. 1) Such one-to-one mapping paradigm tends to overfit each single training pair of images; 2) The inter-image prior knowledge about the general distribution patterns of visual semantics, which can be transferred across images sharing similar semantics, is not explicitly exploited. In this paper, we propose the Generative Memory-guided Semantic Reasoning Model (GM-SRM), which infers the content of corrupted regions based on not only the known regions of the corrupted image, but also the learned inter-image reasoning priors characterizing the generalizable semantic distribution patterns between similar images. In particular, the proposed GM-SRM first pre-learns a generative memory from the whole training data to explicitly learn the distribution of different semantic patterns. Then the learned memory are leveraged to retrieve the matching semantics for the current corrupted image to perform semantic reasoning during image inpainting. While the encoder-decoder network is used for guaranteeing the pixel-level content consistency, our generative priors are favorable for performing high-level semantic reasoning, which is particularly effective for inferring semantic content for large corrupted area. Extensive experiments on Paris Street View, CelebA-HQ, and Places2 benchmarks demonstrate that our GM-SRM outperforms the state-of-the-art methods for image inpainting in terms of both visual quality and quantitative metrics. [ABSTRACT FROM AUTHOR]

The rapid progress in deep learning-based computer vision has opened unprecedented possibilities in computing various high-level analytics for sports. Artificial intelligence techniques such as predictive analysis, automatic highlight generation, and assistant coaching have been applied to improve performance and decision-making for teams and players. To perform any high-level analysis from a game match, collecting the locations (where) and identities (who) of players is crucial and challenging. In this paper, a universal JErsey number DEtector (JEDE) for player identification is presented that predicts players’ bounding boxes and keypoints, along with bounding boxes and classes of jersey digits and numbers in an end-to-end manner. Instead of generating digit proposals from pre-defined anchors, JEDE predicts more robust proposals guided by players’ features and pose estimation. Moreover, a dataset is collected from soccer and basketball matches with annotations on players’ bounding boxes and body keypoints, and jersey digits’ bounding boxes and labels. Extensive experimental results and ablation studies on the collected dataset show that the proposed method outperforms the state-of-the-art methods by a large margin. Both quantitative and qualitative results also demonstrate JEDE’s superior practicality and generalizability over different sports. [ABSTRACT FROM AUTHOR]

As one of the fundamental technologies for scene understanding, semantic segmentation has been widely explored in the last few years. Light field cameras encode the geometric information by simultaneously recording the spatial information and angular information of light rays, which provides us with a new way to solve this issue. In this paper, we propose a high-quality and challenging urban scene dataset, containing 1074 samples composed of real-world and synthetic light field images as well as pixel-wise annotations for 14 semantic classes. To the best of our knowledge, it is the largest and the most diverse light field dataset for semantic segmentation. We further design two new semantic segmentation baselines tailored for light field and compare them with state-of-the-art RGB, video and RGB-D-based methods using the proposed dataset. The outperforming results of our baselines demonstrate the advantages of the geometric information in light field for this task. We also provide evaluations of super-resolution and depth estimation methods, showing that the proposed dataset presents new challenges and supports detailed comparisons among different methods. We expect this work inspires new research direction and stimulates scientific progress in related fields. The complete dataset is available at https://github.com/HAWKEYE-Group/UrbanLF. [ABSTRACT FROM AUTHOR]

To improve the viewer’s Quality of Experience (QoE) and optimize computer graphics applications, 3D model quality assessment (3D-QA) has become an important task in the multimedia area. Point cloud and mesh are the two most widely used digital representation formats of 3D models, the visual quality of which is quite sensitive to lossy operations like simplification and compression. Therefore, many related studies such as point cloud quality assessment (PCQA) and mesh quality assessment (MQA) have been carried out to measure the visual quality of distorted 3D models. However, most previous studies utilize full-reference (FR) metrics, which indicates they can not predict the quality level in the absence of the reference 3D model. Furthermore, few 3D-QA metrics consider color information, which significantly restricts their effectiveness and scope of application. In this paper, we propose a no-reference (NR) quality assessment metric for colored 3D models represented by both point cloud and mesh. First, we project the 3D models from 3D space into quality-related geometry and color feature domains. Then, the 3D natural scene statistics (3D-NSS) and entropy are utilized to extract quality-aware features. Finally, a support vector regression (SVR) model is employed to regress the quality-aware features into visual quality scores. Our method is validated on the colored point cloud quality assessment database (SJTU-PCQA), the Waterloo point cloud assessment database (WPC), and the colored mesh quality assessment database (CMDM). The experimental results show that the proposed method outperforms most compared NR 3D-QA metrics with competitive computational resources and greatly reduces the performance gap with the state-of-the-art FR 3D-QA metrics. The code of the proposed model is publicly available now at https://github.com/zzc-1998/NR-3DQA. [ABSTRACT FROM AUTHOR]

Multi-object tracking (MOT), which is crucial for computer vision and video processing, has immense potential for improvement. Traditional tracking-by-detection approaches include feature-based object re-identification methods that use trained features, but these methods suffer from a lack of suitable training data. In training datasets used for MOT, every object in a video sequence must have its own location and ID. However, assigning IDs to each object in every sequence is considerably labor-intensive, and hence current MOT datasets are unsuitable for training re-identification networks. To resolve this issue, this paper proposes a novel self-supervised learning method using several short videos that contain no human-added labels, based on the idea that each video is a set of temporally corresponding image frames. We then describe how to improve tracking performance using a re-identification network trained in a self-supervised manner. In addition, ablation studies were conducted in order to define the optimal parameters, such as number of clips, data augmentation, and appropriate matching algorithms. The proposed approach achieved competitive performance compared with current best-practice methods including supervised methods, achieving MOT accuracy = 62.0% and ID F1-score = 62.7% on the MOT17 benchmark. [ABSTRACT FROM AUTHOR]

Correlation filters with Convolutional Neural Networks (CNNs) features have obtained tremendous attention and success in visual tracking. However, redundant and noisy feature channels existed in CNN features may cause severe over-fitting and greatly limit the discriminative power of the tracking model. To tackle the issue, in this paper, we develop a new and effective channel-aware correlation filters (CACF) method for boosting the tracking performance. Our CACF method aims to dynamically select representative and discriminative feature channels from high-dimensional CNN features to reduce the model complexity and better distinguish the target object from the background. Moreover, the CACF model is solved by the alternating direction method of multipliers (ADMM) to learn correlation filters. By retaining reliable feature channels, our CACF tracking method can reach better generalization ability and discriminative ability to accurately localize the target object. Comprehensive experiments are conducted on challenging tracking datasets, and the experiment results prove that our CACF method obtains favorable tracking accuracy compared to several popular tracking methods. [ABSTRACT FROM AUTHOR]

The existing state-of-the-art (SOTA) video salient object detection (VSOD) models have widely followed short-term methodology, which dynamically determines the balance between spatial and temporal saliency fusion by solely considering the current consecutive limited frames. However, the short-term methodology has one critical limitation, which conflicts with the real mechanism of our visual system — a typical long-term methodology. As a result, failure cases keep showing up in the results of the current SOTA models, and the short-term methodology becomes the major technical bottleneck. To solve this problem, this paper proposes a novel VSOD approach, which performs VSOD in a complete long-term way. Our approach converts the sequential VSOD, a sequential task, to a data mining problem, i.e., decomposing the input video sequence to object proposals in advance and then mining salient object proposals as much as possible in an easy-to-hard way. Since all object proposals are simultaneously available, the proposed approach is a complete long-term approach, which can alleviate some difficulties rooted in conventional short-term approaches. In addition, we devised an online updating scheme that can grasp the most representative and trustworthy pattern profile of the salient objects, outputting framewise saliency maps with rich details and smoothing both spatially and temporally. The proposed approach outperforms almost all SOTA models on five widely used benchmark datasets. [ABSTRACT FROM AUTHOR]

Invertible color-to-grayscale conversion is a research issue in grayscale image colorization, which is a complex problem resulting from information loss. This paper presents an innovative invertible color-to-grayscale conversion idea by independently compressing the chromaticity plane and hiding it into the corresponding luminance plane as a watermark. Since the chromaticity and luminance planes are orthogonal in the proposed method, they can be processed efficiently without mutual influence. By using an efficient lossy compression operation, we can save more chromatic information. By using two high-capacity data hiding techniques (reversible watermarking (RW) and least significant bit (LSB) substitution), we can embed the compressed chromatic information into the luminance plane perfectly. For the purpose of integrality authentication, the luminance plane is hashed as part of the embedded information before the embedding. Experimental results have shown that higher quality of reconstructed color images can be achieved by using RW, but the quality of the synthesized grayscale drops sharply. By using LSB substitution, we can obtain high-quality synthesized grayscale and reconstructed color images simultaneously. Furthermore, we have compared the proposed LSB-based scheme with several recently reported state-of-the-art methods to validate the superiority of the proposed approach. [ABSTRACT FROM AUTHOR]

This paper presents a novel method to determine rate-distortion optimized transform coefficients for efficient compression of videos generated from point clouds. The method exploits a generalized frequency selective extrapolation approach that iteratively determines rate-distortion-optimized coefficients for all basis functions of two-dimensional discrete cosine and sine transforms. The method is applied to blocks containing both occupied and unoccupied pixels in video based point cloud compression for HEVC encoding. In the proposed algorithm, only the values of the transform coefficients are changed such that resulting bit streams are compliant to the V-PCC standard. For all-intra coded point clouds, bitrate savings of more than 4% for geometry and more than 6% for texture error metrics with respect to standard encoding can be observed. These savings are more than twice as high as savings obtained using competing methods from literature. In the randomaccess case, our proposed method outperforms competing V-PCC methods by more than 0.5%. [ABSTRACT FROM AUTHOR]

Translation averaging is known to be more difficult than rotation averaging due to scale ambiguity, estimation sensitivity, and solution uncertainty. Existing approaches have exposed their limitations in terms of accuracy, robustness, simplicity, or efficiency. To tackle this tough problem, a simple yet effective translation averaging pipeline, termed as Incremental Translation Averaging (ITA), is proposed in this paper. It combines the advantages of high accuracy and robustness in incremental parameter estimation pipeline and the advantages of high simplicity and efficiency in global motion averaging approach. Unlike the traditional translation averaging methods which estimate all the absolute camera locations simultaneously and suffer from inaccuracy in parameter estimation and incompleteness in scene reconstruction, our ITA computes them novelly in an incremental way with higher accuracy and robustness. Thanks to the introduction of incremental parameter estimation thought into the translation averaging pipeline, 1) our ITA is robust to measurement outliers and accurate in parameter estimation; and 2) our ITA is simple and efficient because of its less dependency on complicated optimization, carefully-designed preprocessing, or additional information. Comprehensive evaluations on the 1DSfM dataset demonstrate the effectiveness of our ITA and its advantages over several state-of-the-art translation averaging approaches. [ABSTRACT FROM AUTHOR]

Deep Neural Networks (DNN) are gaining higher commercial values in computer vision applications, e.g., image classification, video analytics, etc. This calls for urgent demands of the intellectual property (IP) protection of DNN models. In this paper, we present a novel watermarking scheme to achieve the ownership verification of DNN architectures. Existing works all embedded watermarks into the model parameters while treating the architecture as public property. These solutions were proven to be vulnerable by an adversary to detect or remove the watermarks. In contrast, we claim the model architectures as an important IP for model owners, and propose to implant watermarks into the architectures. We design new algorithms based on Neural Architecture Search (NAS) to generate watermarked architectures, which are unique enough to represent the ownership, while maintaining high model usability. Such watermarks can be extracted via side-channel-based model extraction techniques with high fidelity. We conduct comprehensive experiments on watermarked CNN models for image classification tasks and the experimental results show our scheme has negligible impact on the model performance, and exhibits strong robustness against various model transformations and adaptive attacks. [ABSTRACT FROM AUTHOR]

Recently, angular margin-based methods have become the mainstream approach for unconstrained face recognition with remarkable success. However, robust face recognition still remains a challenge, as the face is subject to variations in pose, age, expression, occlusion, and illumination, especially in unconstrained scenarios. Since the training dataset are always collected in unconstrained scenarios, it is inevitable that there’re significant number of hard examples in the training process. In this paper, we design a hard example selection function to effectively identify hard examples in the training procedure with the supervision of angular margin-based losses. Furthermore, a novel Intra-class Hard Example Mining (IHEM) loss function is proposed, which penalizes the cosine distance between the hard examples and their class centers to enhance the discriminative power of face representations. To ensure high performance for face recognition, we combine the supervision of angular margin-based loss and IHEM loss for model training. Specifically, during the training procedure, the angular margin-based loss guarantees the power of feature discrimination for face recognition, while the IHEM loss further encourages the intra-class compactness of hard example. Extensive results demonstrate the superiority of our approach. [ABSTRACT FROM AUTHOR]

Deep learning has greatly promoted the development of multi-view stereo in recent years. However, how to measure the reliability of the estimated depth map for practical applications and make reasonable depth hypothesis sampling for the cost volume building in the coarse-to-fine architecture are still unresolved crucial problems. To this end, an Uncertainty Guided multi-view Network (UGNet) is proposed in this paper. In order to enable the network to perceive the uncertainty, an uncertainty-aware loss function is introduced, which not only can infer uncertainty implicitly in an unsupervised manner but also can reduce the bad impact of high uncertainty regions and the erroneous labels in the training set during training. Moreover, an uncertainty-based depth hypothesis sampling strategy is further proposed to adaptively determine the depth search range of each pixel for finer stages, which helps to generate more rational depth intervals compared with other methods and build more compact cost volumes without redundancy. Experimental results on DTU dataset, BlendedMVS dataset, Tanks and Temples dataset and ETH3D high-res benchmark show that our method achieves promising reconstruction results compared with other state-of-the-art methods. [ABSTRACT FROM AUTHOR]

This paper introduces deformable convolution in deep learning based joint denoising and demosaicing (JDD), which yields more adaptable representation and larger receptive fields in features extraction for a superior restoration performance. However, the deformable convolution generally leads to considerable computational load and irregular memory access bottleneck, limiting its extensive deployment on edge devices. To address this issue, we develop grouping strategy and assign independent offsets to each kernel group to reduce the computation latency while keeping the accuracy. Motivated by the exploration for aggregate distribution characteristics of deformable offsets, we present the offset sharing methodology to simplify the memory access complexity of deformable convolution. As for hardware acceleration, we specially design a novel deformable matrix multiplication workflow incorporated with a deformable memory mapping unit to boost the computational throughput. The verification experiments on FPGA demonstrate that the proposed deformable convolution based JDD can restore 4K Ultra High Definition (UHD) images at 70FPS and yields significant promotion in visual effect and objective quality assessment. [ABSTRACT FROM AUTHOR]

The stable continuation of trajectories among different targets has always been the key to the tracking performance of multi-object tracking (MOT) tasks. If features of the target are aggregated and classified simply, the discriminant features of the target will be ignored. This will affect the robustness of the trajectory generated by the model. Meanwhile, many popular models are keen to execute detection and feature extraction tasks in parallel. But these two tasks will conflict with each other when optimized respectively. Therefore, we propose our tracker MOTFR to solve the above problems. In this paper, we propose a Locally Shared Information Decoupling Module (LSIDM) to reduce task optimization conflicts while ensuring the necessary information sharing. Meanwhile, a feature recoding module for deep extraction of identity discriminative features is proposed, which is called the Feature Purification Module (FPM). By combining LSIDM and FPM modules, the model utilizes the discriminative appearance features to guide the optimization of detection and further improves the performance of our model. To solve the problem of targets disappearing due to various abnormal occlusion, a Short-term Trajectory Online Complement Strategy (STOCS) is proposed to realize the trajectories continuation of these targets in the tracking stage. Through sufficient experiments, we demonstrate the superior performance of our MOTFR, which guarantees high-quality detection while achieving the stability of the target trajectory. [ABSTRACT FROM AUTHOR]