Search

Previous work has shown that CNN-based local distribution learning can efficiently reconstruct high-resolution images, but with limited performance improvement against complex degraded images. In this paper, we propose an enhanced local distribution learning framework, called ELDRN, which successfully generalizes local distribution learning to realistic images whose degradation process is complex and unknowable. The cores of our ELDRN are the parallel attention block and dilated neighborhood sampling. The former mines discriminative features at both spatial and channel levels, that is, parameters for constructing local distributions, thus improving the robustness of distributions to real degradation patterns. To deal with the fact that the reference range of the target sub-pixel is not exactly equal to its neighborhood, we explicitly increase the sampling density, i.e. , fusing more sampled pixels to produce the target sub-pixel. Experiments conducted on RealSR dataset illustrate that our ELDRN outperforms recent learning-based SISR methods and reconstructs visually-pleasant high-quality images. • CNN-based enhanced local distribution learning method is proposed. • Parallel attention module is proposed to extract effective feature. • Dilated neighborhood sampling strategy is proposed. [ABSTRACT FROM AUTHOR]

Large scale image understanding is a challenging but significant task to comprehend image contents on the internet. The de-facto standard methods based on machine learning or computer vision still suffer from a phenomenon of visual polysemia and concept polymorphism (VPCP). To resolve the VPCP, semantic dictionary has been proposed to characterize the membership distribution between visual appearances and semantic concepts. In this paper, we propose an online semantic dictionary learning algorithm on the base of both locality sensitive hashing (LSH) and stochastic approximations, which can scale up to large scale datasets with millions of training samples and speedup the efficiency of follow-up processing. With the help of the LSH-based semantic dictionary, we develop an extension of the spatial pyramid matching (SPM) kernel method by generalizing the dictionary as a basic semantic description. The efficiency of our approach is validated in the experiments of web-scale semantic image search and image classification on the ImageNet dataset and Caltech-256 dataset. [ABSTRACT FROM AUTHOR]

Clothing style analysis is a critical step for understanding images of people. To automatically identify the style of clothing that people wear is a challenging task due to various poses of person and large variations for even the same clothing category. Suit as one of the clothing style is a key element in many important activities. In this paper, we propose a novel suits detection method for images of people in unconstrained environments. In order to cope with various human poses, human pose estimation is incorporated. By analyzing the style of clothing, we propose the color features, shape features and statistical features for suits detection. Experiments with four popular classifiers have been conducted to demonstrate that the proposed features are effective and robust. Comparative experiments with Bag of Words (BoW) method demonstrate that the proposed features are superior to BoW which is a popular method for object detection. The proposed method has achieved promising performance over our dataset, which is a challenging web image set with various human poses and diverse styles of clothing. [ABSTRACT FROM AUTHOR]

In traditional image retrieval, images are commonly represented using Bag-of-visual-Words (BoW) built from image local features. However, the lack of spatial and structural information suppresses its performance in applications. In this paper, we introduce a multi-cues description by fusing structural, content and spatial information for partial-duplicate image retrieval. Firstly, we propose a rotation-invariant Local Self-Similarity Descriptor (LSSD), which captures the internal structural layouts in the local textural self-similar regions around interest points. Then, based on the spatial pyramid model, we make use of both LSSD and SIFT to construct an image representation with multi-cues. Finally, we formulate the Semi-Relative Entropy as the distance metric. Comparison experiments with state-of-the-art methods on four popular databases show the efficiency and effectiveness of our approach. [ABSTRACT FROM AUTHOR]

Highlights: [•] A novel approach for ROI detection in pornographic images is put forward. [•] A ROI-based codebook algorithm is proposed. [•] A hybrid approach of pornographic image detection is explored. [Copyright &y& Elsevier]

Thanks to the powerful modeling capabilities of multi-head self attention (MSA), transformers have shown significant performance gains in vision tasks. However, as transformers require heavy computation, more efficient designs are required. In this paper, we present an efficient transformer architecture named GoLDFormer for image restoration. GoLDFormer extends the capability of window-based self-attention through two core designs. First, We propose a globally-enhanced window-based transformer block (G-WTB), which applies transposed attention to a compressed window representation rather than the spatial features, thus establishing connections between all windows with less computational complexity. Second, since the interactions between image content and window attention weights can be interpreted as spatially varying convolution, we introduce an adaptive filter structure into transformer models and propose a deformable filtering block (DFB) to enable cross-window connections. By adjusting the shape of the generated filters in the DFB, we can balance the computational costs and the degree of adjacent window interaction. Extensive experiments on several image restoration tasks demonstrate that GoLDFormer achieves competitive results against recent methods but with optimal computational costs. • We propose GoLDFormer, including two blocks, to enhance W-MSA globally and locally. • Globally-enhanced attention block captures global dependencies between all windows. • Deformable filtering block captures local dependencies by modifying filter size. • Our GoLDFormer achieves competitive results against recent lightweight methods. [ABSTRACT FROM AUTHOR]

As the latest generation of video coding standards, versatile video coding (VVC) introduces several new coding tools in transform coding to concentrate the energy of residual blocks. In this work, we propose a regular multiplier (RM) based hardware architecture for 2D transform that can process different transform sizes and types using a unified architecture. This architecture consists of a 1D row transform, a transpose memory, and a 1D column transform, allowing for processing 16 coefficients per cycle. For the 1D row/column transform, the unified calculation structure consists of 512 regular multipliers supporting various transform sizes and types. For the transpose memory, we design an SRAM-based diagonal storage approach, along with a FIFO for storing the block information. Compared with the state-of-the-art MCM-based design, the proposed RM-based design shows a 30.9% reduction in area while can operate at a higher frequency. [ABSTRACT FROM AUTHOR]

The goal of No-Reference Image Quality Assessment (NR-IQA) is to endow computers with a human-like ability to evaluate an image's quality without comparison to a reference. Current deep learning-based methods mainly work in the spatial domain to measure the quality, heavily rely on semantic information, and less on the degradation of the image itself, struggling to accurately judge the quality of an image in similar scenes. In this paper, we propose a novel degradation priors learning architecture to address the NR-IQA task by leveraging learnable degradation priors, along with semantic features. The multi-task learning strategy is introduced to ensure our model could obtain accurate degradation priors for the NR-IQA task. Extensive experiments on public benchmarks demonstrate that our approach outperforms state-of-the-art solutions. Besides we also collect an additional dataset namely ReD-1K to illustrate the superiority of our approach to judge the image quality in similar scenes. • We propose a novel architecture involving degradation priors and semantic features for NR-IQA. • A multi-task learning framework is proposed for NR-IQA, intergating semantic features and frequency domain degradation features. • We collect a new dataset namely ReD-1K, which consists of 537 pairs of degraded and non-degraded real images. [ABSTRACT FROM AUTHOR]

The assessment of image is a critical task in computer vision, particularly in a no-referenced context, which presents numerous challenges. Though Deep neural networks have exhibited excellent performance on no-reference quality assessment (NR-IQA), the distortion-aware NR-IQA methods are limited to specific distortion type and struggle to handle realistic distortion scenarios. In this work, we propose a feature rectification and enhancement convolutional neural network namely FREIQA for NR-IQA. Our approach targets to fully utilize the multi-stage semantic features to produce a well fused and rectified score vector for effective image quality regression. Specifically, the multi-stage semantic features extracted by a pre-trained backbone would be rectified with a multi-level channel attention module before integrating to a global score vector. The fused score vector then will be gradually optimized through the perception feature provided by the multi-stage semantic feature, and finally sent to quality regression network to obtain the image quality score. Experimental results on public benchmarks including LIVE, TID2013, CSIQ, LIVEC, KADID-10k, KonIQ-10k and Waterloo exploration database show that our FREIQA achieve the state-of-the-art performance. [ABSTRACT FROM AUTHOR]

Affine motion estimation (AME) which is newly introduced in versatile video coding (VVC) plays a significant role in the bit-rate reduction for rotation and zooming scenes. However, it brings the complexity extremely increased. In this paper, a fast AME algorithm is designed to cope with the high-complexity problem. Three crucial stages are involved in the fast algorithm, which is the optimal inter-mode decision based on the coding unit (CU) partition, the fast algorithm in affine motion search (AMS), and the fast inter-mode decision based on a decision tree. Specifically, the optimal inter-mode will be determined straightforwardly when the parent CU is checked whether it is translational motion estimation (TME). Then, three fast algorithms are made in AMS, which are early termination based on control point motion vector (C P M V), early termination of the iteration process, and fast fine granularity C P M V search. Finally, the optimal inter-mode is predicted using a decision tree based on eight essential features. Experimental results show that the proposed algorithm achieves 10.20% and 10.33% encoding time-saving on average under Low Delay B (LDB) and Random Access (RA) configuration, while the BD-Rate loss is only 0.12% and 0.14%, respectively. • Propose a three-stage scheme to decrease the complexity of the affine ME in VVC. • Utilize the CU partition including parent-CU and child-CU to reduce complexity. • Propose a three-step fast algorithm for the fast algorithm in affine motion search. • Find eight key features to predict the optimal inter-mode based on the decision tree model. [ABSTRACT FROM AUTHOR]

This work optimizes tensor-times-dense matrix multiply (Ttm) for general sparse and semi-sparse tensors on CPU and NVIDIA GPU platforms. Ttm is a computational kernel in tensor methods-based data analytics and data mining applications, such as the popular Tucker decomposition. We first design an in-place sequential SpTtm to avoid explicit data reorganizing between a tensor and a matrix in its conventional approach. We further optimize SpTtm on NVIDIA GPU platforms. Five approaches including employing fine thread granularity, arranging coalesced memory access, rank blocking, and using fast GPU shared memory are developed for GPU-SpTtm. We also optimize semi-sparse tensor-times-dense matrix multiply (SspTtm) to take advantage of the inside dense sub-structures. The optimized SpTtm and SspTtm are applied to Tucker decomposition to improve its overall performance. Our sequential SpTtm is 3 – 120 × faster than the SpTtm from Tensor Toolbox library. GPU-SpTtm obtains 6 – 19 × speedup on NVIDIA K40c and 23 – 67 × speedup on NVIDIA P100 over CPU-SpTtm respectively. Our GPU-SpTtm is 3. 9 × faster than the state-of-the-art GPU implementation. Our SspTtm implementations outperform SpTtms by up to 4. 5 × , which handles the input semi-sparse tensor in a general way. Tucker decomposition achieves up to 3. 2 × speedup after applying the optimized Ttms. The code will be publicly released in ParTI! library: https://github.com/hpcgarage/ParTI. • Designed an in-place SpTTM algorithm to avoid tensor-matrix data transformation. • Optimized SpTTM and SspTTM on GPUs. • Implement the Tucker decomposition on a single node GPU system. • Get better performance than state of the arts. [ABSTRACT FROM AUTHOR]

In the age of information explosion, image classification is the key technology of dealing with and organizing a large number of image data. Currently, the classical image classification algorithms are mostly based on RGB images or grayscale images, and fail to make good use of the depth information about objects or scenes. The depth information in the images has a strong complementary effect, which can enhance the classification accuracy significantly. In this paper, we propose an image classification technology using principal component analysis based on multi-view depth characters. In detail, firstly, the depth image of the original image is estimated; secondly, depth characters are extracted from the RGB views and the depth view separately, and then the reducing dimension operation through the PCA is implemented. Eventually, the SVM is applied to image classification. The experimental results show that the method has good performance. [ABSTRACT FROM AUTHOR]

This paper introduces a novel video saliency model for salient object detection in videos. Firstly, we generate multi-level deep features via a symmetrical convolutional neural network, in which the inputs are the current frame and the optical flow image. Then, the multi-level deep features are integrated in a hierarchical manner using a fusion network, which deploys attention module to make a selection for deep features. Lastly, the integrated deep feature is combined with the boundary information originated from shallow layer of the feature extraction networks, and the saliency map is generated by the saliency prediction step. The key advantages of our model lie on the attention module, the hierarchical integration and the boundary information, in which the former one acts as weight filter and is used to select the most salient regions in deep features, the middle one gives an effective integration manner for features from different layers and the last one provides well-defined boundaries for saliency map. Extensive experiments are performed on two challenging video dataset, and the experimental results show that our model consistently outperforms the state-of-the-art saliency models in a large margin. [ABSTRACT FROM AUTHOR]

Surface defect detection has become more and more important in the industrial manufacture and engineering application in recent years. However, due to the lack of overall perception and interaction among features layers, lots of computer vision-based detection methods cannot grab the complete details of defects when dealing with complex scenes, such as low contrast and irregular shape. Therefore, in this paper, we propose a Context-aware Aggregation Network (CANet) to accurately pop-out the defects, where we focus on the mining of context cues and the fusion of multiple context features. To be specific, embarking on the multi-level deep features extracted by encoder, we first deploy a sufficient exploration to dig the context information by deploying the weighted convolution pyramid (WCP) module, which extracts multi-scale context features, transfers the information flow between different resolution features, and fuses the features with same resolution. By this way, we can obtain the effective context pyramid features. Then, the decoder deploys the weighted context attention (WCA) module to filter the irrelevant information from context features and employs the cascaded fusion structure (CFS) to aggregate the multiple context cues in a hierarchical way. Following this way, the generated high-quality saliency maps can highlight the defects accurately and completely. Extensive experiments are performed on four public datasets, and the results firmly prove the effectiveness and superiority of the proposed CANet under different evaluation metrics. [ABSTRACT FROM AUTHOR]

The cutting-edge RGB saliency models are prone to fail for some complex scenes, while RGB-D saliency models are often affected by inaccurate depth maps. Fortunately, light field images can provide a sufficient spatial layout depiction of 3D scenes. Therefore, this paper focuses on salient object detection of light field images, where a Similarity Retrieval-based Inference Network (SRI-Net) is proposed. Due to various focus points, not all focal slices extracted from light field images are beneficial for salient object detection, thus, the key point of our model lies in that we attempt to select the most valuable focal slice, which can contribute more complementary information for the RGB image. Specifically, firstly, we design a focal slice retrieval module (FSRM) to choose an appropriate focal slice by measuring the foreground similarity between the focal slice and RGB image. Secondly, in order to combine the original RGB image and the selected focal slice, we design a U-shaped saliency inference module (SIM), where the two-stream encoder is used to extract multi-level features, and the decoder is employed to aggregate multi-level deep features. Extensive experiments are conducted on two widely used light field datasets, and the results firmly demonstrate the superiority and effectiveness of the proposed SRI-Net. • We deploy the FSRM to explicitly dig complementary information from all focal slices. • The similarity computation is utilized to retrieve the most valuable focal slice. • To aggregate the retrieved focal slice and the RGB image, we deploy the two-stream SIM to extract and fuse the deep features of them. [ABSTRACT FROM AUTHOR]

Human skin detection in images is desirable in many practical applications, e.g., human–computer interaction and adult-content filtering. However, existing methods are mainly suffer from confusing backgrounds in real-world images. In this paper, we try to address this issue by exploring and combining several human skin properties, i.e. color property, texture property and region property. First, images are divided into superpixels, and robust skin seeds and background seeds are acquired through color property and texture property of skin. Then we combining color, region and texture properties of skin by proposing a novel skin color and texture based graph cuts (SCTGC) to acquire the final skin detection results. Comprehensive and comparative experiments show that the proposed method achieves promising performance and outperforms many state-of-the-art methods over publicly available challenging datasets with a great part of hard images. [ABSTRACT FROM AUTHOR]

Compared with deadzone hard-decision quantization (HDQ), rate-distortion optimized quantization (RDOQ) in HEVC brings non-negligible coding gain, however consumes considerable computations caused by exhaustive search over multiple candidates to determine optimal output level. Benefiting from efficient prediction in HEVC, transform blocks are frequently quantized to all zero, especially in small-size blocks. It is worthwhile to detect all zero block (AZB) for transform blocks to bypass subsequent computation-intensive RDOQ. Traditional thresholding based AZB detection algorithms are well-suited for deadzone quantized blocks, however miss partial optimal results in RDOQ and suffer from more or less accuracy degradation in RDOQ. This paper proposes a novel multi-stage AZB detection algorithm for RDOQ blocks with good tradeoff between complexity and accuracy. At the first stage, genuine all zero blocks (G_AZB) which are quantized to all zero both in HDQ and RDOQ are prejudged by comparison with conservative threshold determined by mathematical derivation for deadzone HDQ. At the second stage, an adaptive threshold model is built using adaptive deadzone offset by simulating the behavior patterns existing in RDOQ, aiming to further detect the pseudo AZB (P_AZB) which are quantized to all zero in RDOQ however not all zero in HDQ. At the final stage, machine learning based detection is proposed to classify the remaining "cunning" all zero blocks using eight distinguished RDO-related features, by which subtle working mechanism in RDOQ is leveraged. The experimental results demonstrate that the proposed algorithm achieves up to 7.471% total coding computation saving with 0.064% BD-RATE increment compared with RDOQ on average. Moreover, the average FNR and FPR detection accuracies are 6.3% and 6.5% respectively. [ABSTRACT FROM AUTHOR]