Search

Image demoireing is a multi-faceted image restoration task involving both moire pattern removal and color restoration. In this paper, we raise a general degradation model to describe an image contaminated by moire patterns, and propose a novel multi-scale bandpass convolutional neural network (MBCNN) for single image demoireing. For moire pattern removal, we propose a multi-block-size learnable bandpass filters (M-LBFs), based on a block-wise frequency domain transform, to learn the frequency domain priors of moire patterns. We also introduce a new loss function named Dilated Advanced Sobel loss (D-ASL) to better sense the frequency information. For color restoration, we propose a two-step tone mapping strategy, which first applies a global tone mapping to correct for a global color shift, and then performs local fine tuning of the color per pixel. To determine the most appropriate frequency domain transform, we investigate several transforms including DCT, DFT, DWT, learnable non-linear transform and learnable orthogonal transform. We finally adopt the DCT. Our basic model won the AIM2019 demoireing challenge. Experimental results on three public datasets show that our method outperforms state-of-the-art methods by a large margin. [ABSTRACT FROM AUTHOR]

Constant bit rate (CBR) videos are widely used in streaming playback applications. However, the image quality of the CBR video is often unstable, especially for scenes with large motion. To this end, we design a new model to represent the distortion of High Efficiency Video Coding (HEVC) constant bit rate video, and propose a neural network for a constant bit rate video quality enhancement (CBREN). We propose a dual-domain restoration module (DRM) to jointly learn the prior knowledge in the pixel domain and the frequency domain. To address the degradation resulting from compression, we propose a two-step quantization degradation estimation strategy. The Inverse DCT (IDCT) Translation Unit (ITU) is used to constrain the quantization table of the constant bit rate video to a suitable range, and the Dynamic Alpha Unit (DAU) is used to fine-tune the quantization table according to the content of each frame. In order to effectively reduce the block distortion of different sizes produced in the compression process, we adopt a multi-scale network. Extensive experiments show that our approach can greatly enhance the quality of CBR compressed video. Moreover, our method can also be applied to constant quantization parameter (CQP) video enhancement tasks, and is certainly superior to existing methods. [ABSTRACT FROM AUTHOR]

Achieving balance between efficiency and performance is a key problem for convolution neural network (CNN)-based single-image super-resolution (SISR) algorithms. Existing methods tend to directly output high-resolution (HR) pixels or residuals to reconstruct the HR image and focus a lot of attention on designing powerful CNN backbones. However, this reconstruction way requires the CNN backbone to have good ability to fit the mapping function from LR pixels to HR pixels, which certainly held these methods back from achieving extreme efficiency and from working in embedded environments. In this work, we propose a novel distribution learning architecture to estimate the local distribution and reconstruct HR pixels by sampling the local distribution with the corresponding 2D coordinates. We also improve the backbone structure to better support the proposed distribution learning architecture. The experimental results demonstrate that the proposed method achieves state-of-the-art performance for extremely efficient SISR and exhibits a good balance between efficiency and performance. [ABSTRACT FROM AUTHOR]

The brain is an energy-consuming organ that heavily relies on the heart for energy supply. Heart abnormalities detected by electrocardiogram (ECG) might provide diagnostic indicators for brain dysfunctions such as stroke. Diagnosis of brain diseases by ECG requires proficient domain knowledge, which is both time and labor consuming. Deep learning is capable of constructing a nonlinear correlation between ECG and stroke without prior expert knowledge. Here, we propose a data-driven classifier-Dense convolutional neural Network (DenseNet) for stroke prediction based on 12-leads ECG data. With our finely-tuned model, we obtain the training accuracy of 99.99% and the prediction accuracy of 85.82%. To our knowledge, this is the first report studying the correlation between stroke and ECG with the aid of deep learning. The results indicate that ECG is a valuable complementary technique for stroke diagnostics. [ABSTRACT FROM AUTHOR]

3D layout is crucial for scene understanding and reconstruction, and very useful in applications like real estate and furniture design. In this paper, we propose a fully automatic solution to estimate 3D layout of an indoor scene from a single 2D image. Our technique contains two key components. Firstly, we train a neural network that directly estimates room structure lines from the input image. Secondly, we propose a novel technique to automatically identify the layout topology of an input image, followed by a nonlinear optimization with equality constraints to estimate the final 3D layout of a scene. Based on our knowledge, this is the first fully automatic technique to achieve single image-based 3D layout estimation of an indoor scene. We evaluate our method on the public datasets $LSUN$ , $Hedau$ and $3DGP$ and the results show that the proposed method achieves accurate 3D layout reconstruction on various images with different layout topologies. [ABSTRACT FROM AUTHOR]

Most person re-identification (re-ID) approaches are based on supervised learning, which requires manually annotated data. However, it is not only resource-intensive to acquire identity annotation but also impractical for large-scale data. To relieve this problem, we propose a cross-camera unsupervised approach that makes use of unsupervised style-transferred images to jointly optimize a convolutional neural network (CNN) and the relationship among the individual samples for person re-ID. Our algorithm considers two fundamental facts in the re-ID task, i.e., variance across diverse cameras and similarity within the same identity. In this paper, we propose an iterative framework which overcomes the camera variance and achieves across-camera similarity exploration. Specifically, we apply an unsupervised style transfer model to generate style-transferred training images with different camera styles. Then we iteratively exploit the similarity within the same identity from both the original and the style-transferred data. We start with considering each training image as a different class to initialize the Convolutional Neural Network (CNN) model. Then we measure the similarity and gradually group similar samples into one class, which increases similarity within each identity. We also introduce a diversity regularization term in the clustering to balance the cluster distribution. The experimental results demonstrate that our algorithm is not only superior to state-of-the-art unsupervised re-ID approaches, but also performs favorably compared with other competing unsupervised domain adaptation methods (UDA) and semi-supervised learning methods. [ABSTRACT FROM AUTHOR]

Frame rate up-conversion (FRUC) usually suffers from unreliable motion vectors due to the absence of the current frame to be interpolated. In addition, since the majority of video sequences are usually compressed by various coding standards to reduce the data volume, the quality of the generated frames in the FRUC will be further impaired. To address this problem, we proposed two FRUC algorithms based on deep residual network. We first present a deep residual network for the FRUC (DRNFRUC), which consists of feature extraction, feature recursive analysis, and image restoration parts with a skip connection between the input and the output of the network. The proposed DRNFRUC takes the result of an arbitrary existing FRUC method as the input and is able to significantly reduce the edge blurring and blocking artifacts when the motion of the block is violent. In addition, we proposed a deep residual network with weighted convolutional motion compensation (DRNWCMC) for the FRUC, where the convolution operations can be embedded into the motion compensation interpolation (MCI) in any existing MCI-based FRUC method. In DRNWCMC, we first devise two convolutional neural networks corresponding to the forward and backward motion compensated frames, respectively. And then, the adaptive interpolation coefficients for motion compensation are designed as two $1\times1$ convolutional kernels. Finally, the interpolation result of WCMC is fed into another convolutional neural network to further improve the performance. All the parameters involved in the DRNWCMC are trained simultaneously under the same cost function. The experimental results show that the two proposed algorithms can remarkably improve both the objective and subjective quality of the interpolated frames. [ABSTRACT FROM AUTHOR]

The understanding of web images has been a hot research topic in both artificial intelligence and multimedia content analysis domains. The web images are composed of various complex foregrounds and backgrounds, which makes the design of an accurate and robust learning algorithm a challenging task. To solve the above significant problem, first, we learn a cross-modality bridging dictionary for the deep and complete understanding of a vast quantity of web images. The proposed algorithm leverages the visual features into the semantic concept probability distribution, which can construct a global semantic description for images while preserving the local geometric structure. To discover and model the occurrence patterns between intra- and inter-categories, multi-task learning is introduced for formulating the objective formulation with Capped- $\ell _{1}$ penalty, which can obtain the optimal solution with a higher probability and outperform the traditional convex function-based methods. Second, we propose a knowledge-based concept transferring algorithm to discover the underlying relations of different categories. This distribution probability transferring among categories can bring the more robust global feature representation, and enable the image semantic representation to generalize better as the scenario becomes larger. Experimental comparisons and performance discussion with classical methods on the ImageNet, Caltech-256, SUN397, and Scene15 datasets show the effectiveness of our proposed method at three traditional image understanding tasks. [ABSTRACT FROM AUTHOR]

Image restoration methods based on convolutional neural networks have shown great success in the literature. However, since most of networks are not deep enough, there is still some room for the performance improvement. On the other hand, though some models are deep and introduce shortcuts for easy training, they ignore the importance of location and scaling of different inputs within the shortcuts. As a result, existing networks can only handle one specific image restoration application. To address such problems, we propose a novel adaptive residual network (ARN) for high-quality image restoration in this paper. Our ARN is a deep residual network, which is composed of convolutional layers, parametric rectified linear unit layers, and some adaptive shortcuts. We assign different scaling parameters to different inputs of the shortcuts, where the scaling is considered as part parameters of the ARN and trained adaptively according to different applications. Due to the special construction of ARN, it can solve many image restoration problems and have superior performance. We demonstrate its capabilities with three representative applications, including Gaussian image denoising, single image super resolution, and JPEG image deblocking. Experimental results prove that our model greatly outperforms numerous state-of-the-art restoration methods in terms of both peak signal-to-noise ratio and structure similarity index metrics, e.g., it achieves 0.2–0.3 dB gain in average compared with the second best method at a wide range of situations. [ABSTRACT FROM AUTHOR]

The dense connection is a powerful technique to build wider and deeper convolution neural networks (CNNs) for handling several computer vision tasks. Despite the excellent performance, it consumes numerous parameters and produces a large weight model file. We studied the distribution of convolution layers and proposed a hard parameter sharing approach known as convolution pool (CP) for compressing dense-connection-based image restoration CNN models. CP is used to reallocate the parameters to specific convolution layers to ensure that some can be shared in different layers. We design a set of dense-connection-based baselines for three typical image restoration tasks, including image denoising, super-resolution, and JPEG deblocking, to validate the performance of the proposed method. Moreover, we comprehensively analyze the potential problems by introducing CP, including group convolution, dilated convolution, and modeling efficiency. Experimental results demonstrate that the proposed method can efficiently achieve an impressive compression rate with negligible performance reduction. [ABSTRACT FROM AUTHOR]

Re-ranking is becoming a critical part of retrieved based Person re-identification algorithms. Existing re-ranking methods always require lots of queries and memory to go through the k -nearest neighbors. To solve this problem, we introduce a Feature Relation Map (FRM) to mine the latent relation between the k -neighbors through convolution neural network, and propose a metric learning based Similarity Evaluation (SE) model to obtain the re-ranking distance from the FRM. The dilated convolution is then introduced by concatenating the dilated convolution kernels and normal convolution kernels along the channel dimension in the improvement of the SE model (named SE-d model), to allow the SE model to efficiently compare more samples pairs for obtaining the final re-ranking distance. Further, we embedding out FRM-SE model to the existing re-ranking methods to prove the effectiveness of our re-ranking model. The experiments on Market1501, CUHK03, DukeMTMC and MSMT17 benchmarks illustrates the superiority of proposed method to the state-of-the-art re-ranking methods. Although the SE model make a great improve performance, the SE-d model gains an increase of 0.31%, 0.63% in top-1 respectably compared with the SE model in the DukeMTMC and MSMT17 datasets. Meanwhile, the SE-d model speed up the model convergence and shorten the training time. Furthermore, in the transfer learning setting, the model trained on either Market1501, CUHK03 or DukeMTMC can achieve a comparable accuracy improvement on the MSMT17 dataset, which validates the generalization of our SE model. [ABSTRACT FROM AUTHOR]

Salient object detection (SOD) in optical remote sensing images (RSIs) plays a significant role in many areas such as agriculture, environmental protection, and the military. However, since the difference in imaging mode and image complexity between RSIs and natural scene images (NSIs), it is difficult to achieve remarkable results by directly extending the saliency method targeting NSIs to RSIs. Besides, we note that the convolutional neural networks (CNNs) based U-Net cannot effectively acquire the global long-range dependency, and the Transformer doesn't adequately characterize the spatial local details of each patch. Therefore, to conduct salient object detection in RSIs, we propose a novel two-branch architecture based network for Aggregating the Transformers and CNNs, namely ATC-Net, where the local spatial details and the global semantic information are fused into the final high-quality saliency map. Specifically, our saliency model adopts an encoder-decoder architecture including two parallel encoder branches and a decoder. Firstly, the two parallel encoder branches extract global and local features by using Transformer and CNNs, respectively. Then, the decoder employs a series of feature-enhanced fusion (FF) modules to aggregate multi-level global and local features by interactive guidance and enhance the fused feature via attention mechanism. Finally, the decoder deploys the read out (RO) module to fuse the aggregated feature of FF module and the low-level CNN feature, steering the feature to focus more on spatial local details. Extensive experiments are performed on two public optical RSIs datasets, and the results show that our saliency model consistently outperforms 30 state-of-the-art methods. [ABSTRACT FROM AUTHOR]